Crate znippy_plugin_git

Expand description

znippy handler for git object stores — the git package format.

One archive holds one git repository (D12: one .znippy per repo). The consumer is gunnar, a pure-Rust git server whose repack() writes the cold tier.

§The two tiers a repository can be stored as

tier	data entries	what it preserves
objects	one per object, named by its oid hex, holding its canonical bytes `"<type> <size>\0<content>"`	the oid is checkable by re-hashing the entry
packs	`pack-<id>.pack` + `pack-<id>.idx`	the client’s own deflate and every delta chain

The object tier is the one this format was designed around and is what the __gunnar_* sub-indexes describe. The pack tier is what gunnar actually seals, and the reason is measured: a git object in a pack is already deflated and often a delta, so storing it inflated per oid costs an OpenZL decode plus a fresh zlib deflate on every fetch, for ever, and forecloses pack-copy permanently. gunnar measured an archive of 60000 small git objects at 3.4× larger than its input. See sections::GitIndexBuilder::pack_tier.

§What this format adds on top of a plain archive

column / module	carries
`object_type`	`blob` / `tree` / `commit` / `tag` for the object tier; `packfile` / `pack-index` for the pack tier — typed listing with no reads
`object_size`	content length, so quota gates and size analytics are index-only
`__gunnar_oid__`	an `stree` over the first 8 bytes of every oid → its lookup row
`__gunnar_graph__`	the commit graph as Arrow: `oid, parents[], tree, committer_time, generation`
`__gunnar_reach__`	per-commit reachability bitmaps (roaring) over object ordinals
`__gunnar_refs__` / `__gunnar_secrets__`	the server’s push logs, on either tier

The last three object modules are emitted for the object tier only — a pack carries its own .idx, which is a better oid index than __gunnar_oid__ because it addresses pack offsets.

All of these modules are reserved (znippy_common::is_reserved_module), so znippy list, decompress, the iceberg sink and the manifest readers skip them exactly as they skip the lookup and the trie.

§Laws it honours

P-1 one archive = one ecosystem (--format git); P-2 it writes only the columns it declared; P-3 it is discovered through [meta]; P-4 a malformed or hostile entry never panics — object_type degrades to unknown and the archive still writes.

§Who writes the reserved sections

GitIndexBuilder does, handed to ArrowIpcSink::with_reserved_builder by the writer that knows the object set — gunnar’s repack(). It is deliberately not wired into znippy compress --format git: the CLI’s small-file batch pass never sees objects that take the big-file path, so a CLI-built index would be silently incomplete, and a silently incomplete oid index is worse than none. znippy compress --format git therefore writes the two columns and no reserved sections; znippy run git lookup|graph reads archives that carry them.

Native builtin, registered in znippy-cli/src/handlers.rs::builtin_handlers. Deliberately not a WASM plugin: the oid index’s hot path is stree, which needs AVX2, an mmap and prefetch, none of which wasm offers.

Re-exports§

pub use archive_write::ArchiveWrite;
pub use archive_write::Faults;
pub use archive_write::FastWriter;
pub use archive_write::SafeWriter;
pub use arms::ALL_ENV;
pub use arms::DEFAULT_REDB_CACHE_BYTES;
pub use arms::ENV_BOUNDARY_DELTA;
pub use arms::ENV_EMIT_WORKERS;
pub use arms::ENV_EXPLODE;
pub use arms::ENV_GC;
pub use arms::ENV_INDEX;
pub use arms::ENV_REACH_COMMITS;
pub use arms::ENV_REDB_CACHE;
pub use arms::ENV_WRITER;
pub use arms::GcArm;
pub use arms::IndexArm;
pub use arms::StoreConfig;
pub use arms::WriterArm;
pub use arms::env_reads;
pub use arms::env_reads_here;
pub use arms::redb_cache_bytes;
pub use indexer::AccountIndexer;
pub use indexer::IndexJob;
pub use indexer::IndexRow;
pub use indexer::IndexerPool;
pub use indexer::Lookup;
pub use indexer::PushPath;
pub use uring_write::UringWriter;
pub use graph::CommitNode;
pub use graph::decode_graph;
pub use graph::graph_schema;
pub use graph::read_graph;
pub use object::GitHashKind;
pub use object::GitObject;
pub use object::GitObjectKind;
pub use object::PACKFILE_TYPE;
pub use object::PACK_INDEX_TYPE;
pub use object::PackFileKind;
pub use object::canonical;
pub use object::is_oid_path;
pub use object::pack_path_kind;
pub use object::parse_canonical;
pub use object::parse_commit;
pub use object::tree_entries;
pub use oid_index::GitOidIndex;
pub use oid_index::OidEntry;
pub use oid_index::OidHit;
pub use oid_index::OidLayout;
pub use oid_index::key_for_oid;
pub use pushlog::CompactionPolicy;
pub use pushlog::CompactionReport;
pub use pushlog::Finish;
pub use pushlog::PushLog;
pub use pushlog::PushLogScan;
pub use pushlog::scan_frames;
pub use reach::ReachEntry;
pub use reach::ReachPolicy;
pub use reach::decode_reach;
pub use reach::read_reach;
pub use reach::reach_schema;
pub use refs::RefLog;
pub use refs::RefState;
pub use refs::read_refs;
pub use refs::refs_schema;
pub use secrets::SecretState;
pub use secrets::SecretUpdate;
pub use secrets::SecretsLog;
pub use secrets::read_secrets;
pub use secrets::secrets_schema;
pub use sections::GitIndexBuilder;
pub use git_ops::GitStore;
pub use git_ops::LookupPath;
pub use git_ops::SelectedStore;
pub use git_ops::lookup_path;
pub use git_ops::open_from_env;
pub use git_ops::open_selected;
pub use serve::HEAD;

Modules§

archive_write: The push path’s archive writer: one trait, three durability contracts (FastWriter, SafeWriter, UringWriter). The push path’s archive writer: one trait, three durability contracts.
arms: Which implementation of each trait a store is built on. One selector, read once at construction, never per operation — arms::StoreConfig. Which implementation of each trait a store is built on — one selector, read once, never per operation.
delta: Computing a git delta, for the one entry shape this engine cannot copy: a stored delta whose base falls outside the request and whose receiver holds nothing. The only place in this crate that computes one. Computing a git delta, for the one entry shape this engine cannot copy.
exploded: §14’s exploded objects table: resolved content, derived from the verbatim truth, built eagerly by the same indexer, droppable at any time. §14’s exploded objects: the CONTRACT — who hands a resolved object over, and what the counters mean.
exploded_arrow: The exploded objects table: ONE Arrow IPC file, one row per object.
gc: Garbage collection: one trait, two implementations (gc::CompactInPlace in place, gc::NewGeneration into a new generation — the default). Garbage collection, as a trait with two implementations.
git_ops: GitOps — the typed API gunnar calls, and the store that implements it. Base znippy never sees this trait. GitOps — ours, typed, and base znippy never sees it.
git_oracle: Stock git as the arbiter, inside a repository. Test support only — nothing on a serving, storing or indexing path may call it, because it forks git. It is pub because tests/concurrent_push.rs compiles against the library, and it exists because git index-pack --strict segfaults outside a repository, which is how an oracle came to be incapable of failing. Stock git as the arbiter for a pack this crate emitted — inside a repository, and never as a segfault mistaken for a verdict.
graph: __gunnar_graph__ — the commit graph as Arrow.
index_layout: One git object index, three Arrow IPC layouts, measured against each other.
indexer: The shared, off-the-ack-path index builder — one indexer per account, gatling fan-out, index tables built last. The shared index path: one indexer per account, off the ack path.
object: Canonical git object bytes — the unit a git-format znippy archive stores.
oid_index: __gunnar_oid__ — the reserved oid index.
pack_walk: One pass over a pushed packfile’s entries — the split the closure check falls out of. Ours, gated against gix’s grammar entry for entry. The split: one pass over a pushed packfile’s entries, and nothing else.
pushlog: The append-only push log: the one mechanism __gunnar_refs__ and __gunnar_secrets__ are both built from.
reach: __gunnar_reach__ — reachability bitmaps.
read_stack: The read path as one stack: Ragnar stree in front, the Arrow index tables it points into, and the redb un-sealed tail it falls through to. One read stack, not three components.
refs: __gunnar_refs__ — the ref namespace as an append-only Arrow push log.
replicate: Store-side replication to another gunnar. Deliberately empty — signature and contract only, no transport. Replication — sending a repository’s stored bytes out to another gunnar.
resolve: Pack entries to oids: the indexer’s half of the split, off the ack path. Pack entries → oids, which is the one thing crate::pack_walk cannot tell you and the objects table is keyed by.
secrets: __gunnar_secrets__ — per-repository secret material, on the same one-RecordBatch-per-push log as crate::refs.
sections: GitIndexBuilder — turns a set of git objects into the three reserved sections a git-format archive carries.
serve: GitServe — the reading contract: decoded reads, the HEAD accessor pair, negotiation and pack emission. Layered above git_ops, answered out of Arrow IPC, and containing no gix — see the module header for why that is a rule rather than a coincidence. The reading contract, answered out of Arrow IPC. No gix, anywhere.
store: The storage functions. Nothing else is permitted here.
uring_write: io_uring arm of archive_write. Linux only — the module is empty elsewhere rather than substituting a pwrite path under an io_uring name. Impl 3 — the io_uring arm: one submission, four ops, kernel-enforced ordering.

Structs§

Caps: What the client said it can parse, as far as pack emission is concerned.
NativeGitPlugin: Native git-object handler.
PackStats: The receipt for one GitServe::emit_pack.
ReachSet: One GitServe::select answer.
RefCas: The typed ref rejection, re-exported so a consumer of this crate reaches it without also naming the contract crate. RefRejection::of(&err) is the only sanctioned way to ask whether a ref write lost a race — never a message. One ref edit with the value the caller expects to find.
RefRow: One row of the ref namespace: name, oid, peeled.
RefUpdate: One ref update inside a push. Owned by the git-storage-trait contract; re-exported here so crate::refs::RefUpdate stays a valid path. One ref update in a push.
Stored: What get hands back: the stored bytes and enough type information that they cannot be mistaken for something else.
TxId: The receipt for one durable transaction.

Enums§

Observed: The typed ref rejection, re-exported so a consumer of this crate reaches it without also naming the contract crate. RefRejection::of(&err) is the only sanctioned way to ask whether a ref write lost a race — never a message. What was actually there — including the case where something was there and could not be decoded.
RefRejection: The typed ref rejection, re-exported so a consumer of this crate reaches it without also naming the contract crate. RefRejection::of(&err) is the only sanctioned way to ask whether a ref write lost a race — never a message. Why a ref write was refused, typed.
RefTarget: The typed ref rejection, re-exported so a consumer of this crate reaches it without also naming the contract crate. RefRejection::of(&err) is the only sanctioned way to ask whether a ref write lost a race — never a message. One ref target, backend-neutrally: an object, or another ref.

Constants§

GIT_TYPE_ID: DenseUnion / pkg_type discriminant. Clear of the built-ins (1–25), media (25), skidbladnir (40) and rust-toolchain (41).
UNKNOWN_OBJECT_TYPE: object_type written when the entry is not parseable as a canonical git object. A distinct, queryable state — never a silent blob.

Traits§

GitOps: The eleven. Typed, backend-neutral, and the only entry point a git server needs into storage.
GitServe: The READING contract.

Type Aliases§

Oid: An object id, borrowed.