znippy handler for git object stores — the git package format.
One archive holds one git repository (D12: one .znippy per repo). The
consumer is gunnar, a pure-Rust git server whose repack() writes the cold
tier.
The two tiers a repository can be stored as
| tier | data entries | what it preserves |
|---|---|---|
| objects | one per object, named by its oid hex, holding its canonical bytes "<type> <size>\0<content>" |
the oid is checkable by re-hashing the entry |
| packs | pack-<id>.pack + pack-<id>.idx |
the client's own deflate and every delta chain |
The object tier is the one this format was designed around and is what the
__gunnar_* sub-indexes describe. The pack tier is what gunnar actually
seals, and the reason is measured: a git object in a pack is already
deflated and often a delta, so storing it inflated per oid costs an OpenZL
decode plus a fresh zlib deflate on every fetch, for ever, and forecloses
pack-copy permanently. gunnar measured an archive of 60000 small git objects
at 3.4× larger than its input. See [sections::GitIndexBuilder::pack_tier].
What this format adds on top of a plain archive
| column / module | carries |
|---|---|
object_type |
blob / tree / commit / tag for the object tier; packfile / pack-index for the pack tier — typed listing with no reads |
object_size |
content length, so quota gates and size analytics are index-only |
__gunnar_oid__ |
an stree over the first 8 bytes of every oid → its lookup row |
__gunnar_graph__ |
the commit graph as Arrow: oid, parents[], tree, committer_time, generation |
__gunnar_reach__ |
per-commit reachability bitmaps (roaring) over object ordinals |
__gunnar_refs__ / __gunnar_secrets__ |
the server's push logs, on either tier |
The last three object modules are emitted for the object tier only — a pack
carries its own .idx, which is a better oid index than __gunnar_oid__
because it addresses pack offsets.
All of these modules are reserved (znippy_common::is_reserved_module),
so znippy list, decompress, the iceberg sink and the manifest readers skip
them exactly as they skip the lookup and the trie.
Laws it honours
P-1 one archive = one ecosystem (--format git); P-2 it writes only
the columns it declared; P-3 it is discovered through [meta]; P-4 a
malformed or hostile entry never panics — object_type degrades to
unknown and the archive still writes.
Who writes the reserved sections
[GitIndexBuilder] does, handed to ArrowIpcSink::with_reserved_builder by
the writer that knows the object set — gunnar's repack(). It is deliberately
not wired into znippy compress --format git: the CLI's small-file batch
pass never sees objects that take the big-file path, so a CLI-built index
would be silently incomplete, and a silently incomplete oid index is worse
than none. znippy compress --format git therefore writes the two columns and
no reserved sections; znippy run git lookup|graph reads archives that carry
them.
Native builtin, registered in znippy-cli/src/handlers.rs::builtin_handlers.
Deliberately not a WASM plugin: the oid index's hot path is stree, which
needs AVX2, an mmap and prefetch, none of which wasm offers.