When git gets slow — structural prescriptions for a large repository

Introduction — twelve seconds for one status

The clone takes 40 minutes, switching branches gives you time to make coffee, and one status check burns 12 seconds.

$ time git status
On branch main
nothing to commit, working tree clean

real	0m12.417s
user	0m1.902s
sys	0m9.884s

In this state, the first thing people try is whatever the top search result prescribes. Clone shallow, run the cleanup command hard, move big files to LFS. All three are correct prescriptions in the right situation, but applied without telling the causes apart they usually make nothing faster and only make things harder to undo.

There are broadly three reasons a repository gets slow: a long commit history, a large number of tracked files, and big binaries inside it. The three create different bottlenecks and their prescriptions do not overlap.

Measure first — how to tell the three apart

The fastest diagnosis is object statistics.

$ git count-objects -vH
count: 1042
size: 8.31 MiB
in-pack: 2841903
packs: 14
size-pack: 6.42 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

If the pack size is wildly out of proportion to the commit count, suspect binaries. If the pack count is in double digits, housekeeping is overdue. Now count the other two axes.

$ git rev-list --count HEAD
84213

$ git ls-files | wc -l
  212847

Eighty thousand commits is not much of a problem on modern hardware. Two hundred and twelve thousand files in the working tree, on the other hand, is more than enough to explain a 12-second status. A status check by default stats every tracked path.

You can also count directly where the large objects are.

$ git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
  | awk '$1 == "blob"' | sort -k3 -n -r | head -5
blob 4d7a2146 104857600 design/hero.psd
blob 91c0ffee  87031808 assets/renders/intro-01.mov
blob 2b7d13aa  73400320 assets/renders/intro-02.mov
blob a04e1f39  52428800 vendor/sdk/ios-framework.zip
blob 77bc09d1  41943040 design/board-v3.psd

If you want to see which command spends its time where, turn on the built-in tracing.

$ GIT_TRACE2_PERF=1 git status 2>&1 | grep -E 'region|data' | tail -20

A prescription chosen without measuring usually turns out to be the one with the worst side effects. Counting always comes first.

Reduce what you download — shallow clone and partial clone are not substitutes

The most common advice is a shallow clone.

$ git clone --depth=1 https://github.com/example/monorepo.git

Transfer volume certainly drops, but this clone has no history. As a result every command that depends on history is crippled. The log ends at one commit, you cannot trace who changed a given line when or why, and merge and rebase fail because there is no way to find the common ancestor of two branches. Tag-based version string generation stops working too. And deepening the clone later because you turned out to need history is often heavier on the server than fetching from scratch.

A partial clone is a different animal. It fetches every commit and tree and only pulls file contents when they are needed.

$ git clone --filter=blob:none https://github.com/example/monorepo.git
Cloning into 'monorepo'...
remote: Enumerating objects: 2841903, done.
remote: Counting objects: 100% (2841903/2841903), done.
Receiving objects: 100% (1204418/1204418), 412.55 MiB | 22.31 MiB/s, done.
Resolving deltas: 100% (812004/812004), done.
Updating files: 100% (48211/48211), done.

Here the log, bisect and merge-base computation are all normal. The price is the network latency at the moment you need old file contents. Tracing the history of an old line gets noticeably slower.

There is also a middle form that filters by size. Deferring only large files mitigates much of the binary problem without a separate server.

$ git clone --filter=blob:limit=1m https://github.com/example/monorepo.git

To sum up. A shallow clone is a tool for a checkout you are going to throw away, and the thing worth recommending as the default on a developer machine is a partial clone. Even in an automation pipeline, if merge-base computation is needed, a shallow clone was already the wrong choice.

Reduce the working tree — sparse checkout and a filesystem monitor

If the file count is the bottleneck, no clone strategy will fix it. You have to reduce the paths actually laid out in the working tree.

$ git clone --filter=blob:none --no-checkout https://github.com/example/monorepo.git
$ cd monorepo
$ git sparse-checkout init --cone --sparse-index
$ git sparse-checkout set services/payments libs/common
$ git checkout main

$ git ls-files | wc -l
    2143

Two options matter here. Cone mode decides by directory prefix alone, which removes almost all of the matching cost. The free-form ignore-rule-style patterns from older documents have to run the entire pattern set against every path, which gets slower as the file count grows. The sparse index option folds out-of-scope directories into a single entry inside the index file itself. If you shrink only the working tree and leave the index alone, a status check still walks all 212 thousand entries.

Next comes the filesystem monitor. The root cause of a slow status is that it reads information about every path to find out what changed, and if the operating system tells you about changes there is no need for that. Since Git 2.37 a built-in monitor ships without any separate tool.

$ git config core.fsmonitor true
$ git config core.untrackedCache true

$ git status    # the first run is slow because it fills the cache
$ time git status
real	0m0.512s

The reason to enable the untracked cache alongside it is that the monitor only tells you about changed paths, while discovering new files is a separate directory traversal. You have to turn both on for the effect to land.

Make reads faster — the commit-graph and housekeeping

There is a separate prescription for slowness caused by a long history. Log and merge-base computation both mean opening commit objects one by one and following parents, and if you precompute that information into a separate file you can skip decompression entirely.

$ git commit-graph write --reachable --changed-paths
Expanding reachable commits in commit graph: 84213, done.
Writing out commit graph in 4 passes: 100% (336852/336852), done.

$ time git log --oneline -- services/payments | wc -l
    1841
real	0m0.63s

Path-limited logs change the most. The changed-paths option also stores a probabilistic filter over which paths changed in each commit, so most commits get skipped without ever being opened.

You do not have to manage these auxiliary files by hand. Since Git 2.31, scheduling maintenance tasks is a standard feature. It replaces the old habit of putting the cleanup command in cron.

$ git maintenance start
$ git config --get-all maintenance.repo
/Users/dev/work/monorepo

One piece of common bad advice has to be called out here.

# Widely repeated, but usually a net loss
$ git gc --aggressive

This option throws away every existing delta chain and recomputes compression from scratch with a much larger search window. On a multi-gigabyte repository it burns CPU for hours, and the space it wins back is usually not large, because the deltas Git makes routinely are already good enough. If your problem is that lookups are slow because there are many packs, what you need is not recomputation but rearrangement.

$ git repack --geometric=2 -d --write-midx
$ git multi-pack-index write

Geometric repacking picks only the small packs and merges them, leaving the big ones alone. The multi-pack index binds several packs into one lookup index, removing the effect the pack count has on lookup cost. The goal of housekeeping is not to end up with one pack, it is to finish a lookup in one step.

Big binaries — what LFS does and does not do

Once design files or video live in the repository, every other prescription is crippled. Delta compression barely works on binaries, so editing a 100-megabyte file ten times grows the repository by a gigabyte.

What Git LFS does is simpler than it sounds. For the paths you designate it swaps the file content for a short pointer text at commit time, and downloads the real content from a separate server at checkout time. What Git stores is always just the pointer.

$ git lfs track "*.psd"
$ cat .gitattributes
*.psd filter=lfs diff=lfs merge=lfs -text

$ git cat-file -p HEAD:design/hero.psd
version https://git-lfs.github.com/spec/v1
oid sha256:4d7a214614ab2935c943f9e0ff69d22eafe21c1b8e4a6d0f0a2d0b1e6dbf3a9c
size 104857600

Here is the most misunderstood point. Adopting LFS today does not make the binary you committed yesterday disappear from the repository. The trees of commits that already exist still point at the original blobs, and those blobs are still sitting in the packs. Clone size does not shrink by a single byte. Cleaning up the past requires a separate history rewrite, and that carries the same price as the next section.

$ git lfs migrate import --include="*.psd,*.mov" --everything
migrate: Sorting commits: ..., done.
migrate: Rewriting commits: 100% (84213/84213), done.
migrate: Updating refs: ..., done.

The limits are worth knowing clearly too. The server has to support LFS, and storage and bandwidth usually cost extra. The local cache has to be managed separately. And LFS only moves binaries somewhere else, it does not make them mergeable. If two people edit the same design file, someone still has to pick a side by hand. To really reduce that problem you want file locking, or better, to keep build artifacts out of the repository in the first place.

$ git lfs prune                       # clean up the old local cache
$ GIT_LFS_SKIP_SMUDGE=1 git clone ... # fetch pointers only, download selectively later
$ git lfs pull --include="design/hero.psd"

One thing to add: the size-filtered partial clone above delivers part of the same effect without any separate server. For a repository you are starting fresh, that option is worth evaluating before adopting LFS.

A repository that already grew — filter-repo and the price of every hash changing

To actually remove old binaries or a large file that got in by mistake, you have to rewrite history. The filter-branch that appears in older documents is discouraged by the official Git documentation itself, and the recommended tool today is filter-repo.

$ pip install git-filter-repo

$ git filter-repo --path assets/renders --invert-paths
Parsed 84213 commits
New history written in 132.71 seconds; now repacking/cleaning...
Completely finished after 218.04 seconds.

$ git count-objects -vH
in-pack: 1932450
size-pack: 1.14 GiB

6.42 GiB became 1.14 GiB. That is the good news, and the price starts now.

Rewriting history means the parent of every commit changes, and when the parent changes every commit hash changes. The consequences do not stop at one repository.

• Every open PR ends up pointing at commits that do not exist.
• Tags and releases have to be pushed again, and every deployment config and submodule reference pinned to a specific commit hash breaks.
• Everyone on the team has to clone fresh. Fetching into an existing clone brings in both the old and the new history and doubles the repository. That is exactly why filter-repo deliberately removes the remote configuration after it runs.
• It does not vanish on the server side either. Hosting services keep old commits alive through PR refs and forks, so actual deletion often requires a separate cleanup request.

So this job needs a plan, not a command. Lock merges, announce the time, rewrite, force-push, have everyone re-clone, and clean up old branches, in that order. This is the only irreversible prescription in this post, which is why it is placed last.

Closing — classification comes before prescription

You do not need to memorize the commands in this post. What you need to remember is the order. Count commits, files and pack size first, then apply the prescription that matches the result, one at a time.

Slow because the history is long means partial clone and commit-graph. Slow because there are many files means sparse checkout and a filesystem monitor, and if you do not shrink the index along with the tree the benefit is cut in half. If binaries are the problem it is LFS or a size-filtered partial clone, and erasing the past means accepting the irreversible decision of a history rewrite.

The two lines that give most teams the biggest win are turning on scheduled maintenance and the filesystem monitor. No side effects, easy to undo, and the difference is immediate. Rewriting history can wait until you have confirmed that those two lines were not enough.

What objects and packfiles actually look like is picked up in Reading Git commands through internals.