Skip to main content

Module delta

Module delta 

Source
Expand description

Computing a git delta, for the one entry shape this engine cannot copy: a stored delta whose base falls outside the request and whose receiver holds nothing. The only place in this crate that computes one. Computing a git delta, for the one entry shape this engine cannot copy.

§Why an engine whose whole claim is “it copies” computes anything at all

crate::git_ops::GitStore::emit_set copies stored pack entries byte for byte, and that is the product. One shape defeats it: a stored delta whose base falls outside the request. The base cannot be added — a base pulled into a narrowed clone drags an object the client never asked for, and if it is a tree it arrives owing children the pack does not carry, which is the did not receive expected object defect §2 of the standing report records. Naming the base by oid in a REF_DELTA works for a fetch, where the receiver already holds it, and cannot work for a clone, where the receiver holds nothing.

That left exactly one answer for a clone: send the object whole. Measured on h2h-linear-sha1-2048c-1024f-16k, that is 961 of 31 805 entries — 3.0 % — and it doubled the wire: 11 613 666 bytes where stock git sends 5 820 000 for the identical object set. Three per cent of the entries were essentially all of a 2× gap, because the 961 are 16 KiB blobs whose stored delta is 284 bytes.

So this module exists to answer the third option: re-delta the boundary entry against something the pack DOES carry. It is the only place in this crate that computes a delta, crate::git_ops::GitStore::whole_entry is the only other place that deflates on a serving path, and both run on the same boundary fraction and nowhere else.

§The receipt stays a fact

An entry this produces is crate::pack_walk::EmitEntry::recompressed, same as the whole rebuild it replaces — its payload really was inflated and re-deflated, and copied + recompressed == objects is unchanged. What is new is crate::pack_walk::EmitEntry::deltified, a strict subset of recompressed: of the entries that had to be rebuilt, this many went out as a computed delta rather than whole. A full clone measures all three as zero, because a whole-repository request contains every base.

§The format

git’s delta stream, which crate::resolve already applies: the base’s size and the target’s size as LEB128, then instructions. A byte with the top bit set is a copy — the low nibble selects which of four offset bytes follow, bits 4-6 which of three size bytes, all little-endian and all absent when zero. A byte with the top bit clear is an insert of that many literal bytes, 1..=127.

Two encoding hazards, both avoided by construction rather than by comment:

  • a copy whose three size bytes are all zero means 65536, not zero, so a zero-length copy must never be emitted — [Delta::copy] is only ever called with len >= MIN_MATCH;
  • a copy offset is four bytes, so a base at or past 4 GiB cannot be named. delta refuses such a base up front rather than truncating one.

§The search, and what it deliberately does not do

git pack-objects sorts every object by type, path hash and size and then tries a sliding window of 10 candidates per object, at pack.depth 50. That is a global search and it is the expensive half of packing.

This does none of it. The caller arrives with one candidate base — the nearest ancestor of the entry’s own stored delta chain that is inside the request — and this module only has to encode against it. The candidate is free: it is read out of the archive’s own back-references, which are the packer’s original similarity judgement, already made and already stored.

§What it produced, end to end

One server process, one seeded store, oden 2026-08-11, git fsck --full --strict exit 0 on every arm. The only difference between the columns is [enabled]:

arm                      objects   whole (off)   re-delta (on)
full clone                36 172     6 200 252     6 200 252    unchanged
narrowed clone `base`     31 805    11 613 666     6 424 726    1.81x
--filter=blob:none        16 491     1 566 022     1 566 022    unchanged
incremental fetch          4 367     1 339 879     1 339 879    unchanged

The narrowed pack’s own entry shapes say where it went: 3 153 whole entries become 2 294, so exactly 859 of the 961 found an in-request ancestor — the number this module’s search predicted before it was written — and 5 188 940 bytes came off, 6 041 per entry. Stock git sends 5 820 000 for the same 31 805 objects, so what was 2.00× git is now 1.10×.

§Cost

One pass over the base to index it and one over the target to encode, both linear, with a 16-byte block hash and a bounded collision chain, plus one inflate of the base — which the single-slot cache spares for a run of entries sharing one ancestor.

Against that, the deflate it feeds is strictly cheaper than the one it replaces: a ~1 KB delta instead of a 16 KiB object. The two roughly cancel. Measured on the server’s own utime + stime over 10 narrowed clones, three interleaved rounds, CPU pressure gated below 2.00: 139 ms per clone with it on, 137 ms with it off, against a within-arm spread of 133-145. So the 1.81× on the wire costs nothing measurable in CPU, which is the trade this engine had to be able to make and could not be assumed.

Functions§

delta
A git delta that rebuilds target out of base, or None when there is no point sending one.
enabled
Is the boundary re-delta on? true unless ZNIPPY_GIT_BOUNDARY_DELTA is set to 0, off or false.