Expand description
Pack entries to oids: the indexer’s half of the split, off the ack path.
Pack entries → oids, which is the one thing crate::pack_walk cannot
tell you and the objects table is keyed by.
This is the indexer’s work, not the ack path’s (§13.9: “the index is built
after the ack, over a channel”). It inflates every entry, applies every delta
chain and hashes the result, which is what git index-pack does and what
makes it the expensive half of a push. Nothing here is called before a client
is told its push landed.
§What it keeps and what it throws away
Peak memory is deliberately not “the whole history inflated”. An object is held only while an unresolved entry still has to read it, and for nothing else:
| object | held | until |
|---|---|---|
| a delta base | yes | its last dependent has consumed it |
| everything else — blob, commit, tree, tag | no | it leaves through the [PayloadSink] and is dropped |
The base set and, more to the point, how many entries read each base are known before a byte is inflated: the walk already collected every delta base, so the resolver starts with an exact reference count per base and drops a payload the moment that count reaches zero. The last dependent gets the base moved into it rather than cloned, so a linear delta chain — one base, one dependent, the ordinary shape of real history — copies no payload at all.
§What this table said before 2026-08-11, and what it cost
It said commits, trees and tags were kept “because the graph and reach
tables are built from them”. They are not, and had not been since §14’s
exploded table became the fold’s input — absorb_one reads
Resolved::index_entry() and nothing else. Two copies of the object set were
being held for a reader that no longer existed:
contentkept every base and every typed object for the whole call. Nothing ever removed a row, so “held only while something still needs them” described an intention, not the code.Resolved::payloadkept a second copy of the same bytes, cloned in.
MEASURED on oden 2026-08-11, tests/resolve_peak_memory.rs, peak live
heap over resolve_walked against the pack’s own inflated payload:
| pack | before | after |
|---|---|---|
| forge-year-8a-240b-12c, 28 893 objects, 111.0 MB | 2.03x | 0.03x |
| stage-red-nobitmap, 12 328 objects, 343.5 MB | 1.34x (461.2 MB) | 0.02x (5.9 MB) |
| facett, 7 232 objects, 72.8 MB | 1.20x | 0.42x |
| znippy’s own, 5 388 objects, 84.4 MB | 0.75x | 0.14x |
The worst pack in the corpus went 2.03x → 0.42x. It is facett, whose
delta chains fan out widely enough that a real base working set is genuinely
live at once — which is the bound behaving as intended rather than a residue.
End to end, one gunnar-server, one push --mirror, anonymous RSS held
afterwards (not peak — this memory was never given back):
| fixture | before | after |
|---|---|---|
| 36 172 objects, 290 MB inflated, 5.9 MB pack | 575.7 MB | 185.1 MB |
| 352 396 objects, 2.89 GB inflated, 61.8 MB pack | 4480.1 MB | 439.9 MB |
The second row is the one that mattered: 10x the repository cost 10x the
memory before and 2.4x after, so the resolver’s footprint stopped tracking
the history. What is left is bounded and named — redb’s page cache
(ZNIPPY_GIT_REDB_CACHE_BYTES, 64 MiB by default, measured at 129.3 MB held
with an 8 MiB ceiling and 303.9 MB with 256 MiB), the exploded table’s
FLUSH_BYTES write buffer, and
Derived’s in-memory graph, which is genuinely O(repository).
§⚠ The clone that got slower, and why it is not this code
Read this before “fixing” the reference counting. vs_forge_clone on the
352 396-object fixture reported this change as a +78 % CPU regression on
every znippy arm, and the natural reading — a base dropped at its “last
use” has to be re-resolved when something later wants it again — is wrong. It
is not even mechanically possible: an undercount cannot cause a re-resolve
here, because there is no re-resolve to fall into. resolve_walked fails the
whole pack by name.
What it actually is, MEASURED on oden 2026-08-11, one gunnar-server, seed the fixture then clone it, steady state (clones 2-3), quiet box, only the allocator and this file varying:
| resolver | allocator | clone CPU | anon after seed |
|---|---|---|---|
| before | glibc | 1.47 s | 4481 MB |
| after | glibc | 2.62 s | 438 MB |
| before | mimalloc | 1.18 s | 1767 MB |
| after | mimalloc | 1.17 s | 701 MB |
With a competent allocator the two resolvers are identical (1.18 against
1.17). The regression is entirely glibc malloc: the old resolver’s retained
4.5 GB was accidentally acting as a pre-warmed pool, so the clone’s own
~2 GB of allocations came off a free list instead of out of the kernel. Take
the leak away and glibc charges for the memory it should have been charging
for all along.
Three independent measurements say the same thing, and none of them involve this file being slower:
- Bisect. A build with only
Resolved::payloadremoved — the reference counting NOT applied,contentstill never pruned — reproduces the whole regression (2.56–2.62 s). Adding the reference counting on top then costs nothing (2.55–2.65 s) and takes anon-after-seed from 2350 MB to 438 MB. The counting is free; the not leaking is what glibc punishes. - Restart. Restart the server after seeding, so the derived tables are folded onto a fresh heap: before 2.20–2.22 s, after 2.20–2.24 s — identical to within 1 %. The old resolver’s 1.47 s is the outlier, not the new one’s 2.62 s.
- The resolver itself. Over the same 6 real packs both revisions accept, 1.058 s before against 0.975 s after: this code is ~8 % faster.
So there is no cheaper trade to find inside this module, and an LRU over
bases would buy nothing. The lever, if the CPU is wanted back, is the
allocator — and LD_PRELOAD=libmimalloc.so.3 is enough to get it with no
code change at all.
§Where the payloads it throws away now go
§14’s exploded table is eager (decided 2026-08-08), so every object this
module resolves has to reach [crate::exploded::ExplodedTable] — including
the blobs the table above says are dropped. They still are: the payload
leaves through a [PayloadSink] as it is produced, rather than being
accumulated into the returned Vec. That is the difference between a peak
footprint of “the bases plus the typed objects” and one of “the whole pack
inflated at once”, and eager resolution does not get to change it.
A thin pack’s REF_DELTA base is an object the client knows the server
already has, and it is usually a blob. Resolving it needs that blob’s
content, which is exactly what the exploded table now holds — so the
[BaseSource] can answer from it in one point lookup instead of re-resolving
a whole pack. A thin pack whose external base is still not re-derivable
is refused by name ([ResolveError::MissingBase]) rather than resolved by
guesswork, unchanged.
Structs§
- NoBases
- A source that has nothing. The right one for a self-contained pack, and it makes “this pack was thin” an error rather than a silent success.
- Resolve
Error - The one failure that is a decision rather than a corrupt pack.
- Resolved
- One resolved object: exactly what the
objectstable stores, and no payload.
Traits§
- Base
Source - Where an entry that deltas against something outside the pack gets its base.
Functions§
- resolve
- Resolve every entry of
packto an oid, keeping no payloads. - resolve_
walked - Same, for a caller that already walked the pack (
putdoes, for the closure check) and must not walk it twice — and for the one that wants the payloads.