Skip to main content

Module ingest

Module ingest 

Source
Expand description

The substrate’s GitPackIngest implement (PackIngest) — ADR-020 §2’s ingestion composition, for real: the received pack indexes into a budgeted host working area (thin bases resolving through the substrate Find), the indexed objects unpack into alkblobs as loose-form objects under the git-object family (the wave-7 object front’s family put — the store’s verification-on-put is the oid re-derivation, corruption cannot enter undetected), the guards ride the w8-pin-registry’s custody seam, connectivity closes the graph from the pushed tips, and the working area is discarded. The reference shape is POC-O’s ingest.rs; the reference semantics are the wave-3 doubles’ MemPackIngest (tests/common/mod.rs in the core crate).

§The recorded design decisions (the wave-8 gate’s reading list)

  • The decompressed-size cap’s enforcement, layered (the task’s named design point — how the Limits::max_object_size bound holds through gix’s own index/inflate machinery):
    1. the counting reader is the defense-in-depth layer the wave-3 gate confirmed: the pack stream’s byte total is bounded by Limits::max_pack_size; a breach is a pack-level failure mapping into the unpack ng leg, never Err (the wire side’s relay breaches separately fail the session closed);
    2. the indexing leg rides gix’s indexing machinery (gix_pack::Bundle::write_to_directory) with its alloc_limit_bytes knob set to the cap: gix’s decode paths vet every allocation-relevant pack-declared quantity against the limit before reserving, so a hostile declaration beyond the cap errors before any allocation, and the machinery’s exact-decompression discipline (a stream that stops early or overproduces against its declared buffer is corruption) makes a lying-small declaration fail too — the hostile-declared-size class maps to the unpack ng leg over the real machinery;
    3. the unpack leg is this module’s own incremental machinery — the wave-5 doubles’ guard verbatim: each entry’s declared size is vetted against the cap before anything is read, the inflate reads in bounded chunks accumulating only produced bytes (the running total aborts at the cap), and each delta’s materialized result is checked against its declaration and produced incrementally — nothing on the unpack path ever sizes an allocation from the pack’s own declaration.
  • The working area’s shape: a fresh per-push directory under the constructor’s root knob (no env — conventions 4/5); the disk shadow is the pack the counting reader bounds (pack + index ≈ the pack budget — no new Limits field, ADR-009’s no-new-kinds rule). The authoritative object home is the store; the area (temporary pack + index) is discarded after unpack — on every path, the explicit return paths’ removal riding a best-effort residue removal that also covers a panic’s unwind (Drop’s bounded directory rm — the one blocking best-effort on the async side).
  • Unpack order and the store’s role: the indexing machinery normalizes the pack (thin-base entries inserted physically and their ref-deltas rewritten, in-pack ref-deltas resolved through the index), so the unpack iterates the written pack in pack order over the index’s entries; every materialized object re-derives (the pack-side oid answer), cross-checks against the index’s own oid, and goes through the store’s batch form — one alkblobs::Batch group over the working set (the store’s pin-before-publish ordering, one commit discipline). Existing content dedups in the store; re-putting every indexed object (rather than skipping known ones) keeps the sweep race closed: an object present-but-sweep-eligible could otherwise vanish mid-window under a bare presence check. Delta bases read back through the Find front (verification-on-get is the same re-derivation guard), one base body at a time — the unpack stage’s memory is the cap’s working set, never O(pack).
  • Connectivity: the full-graph walk over the substrate Find from the pushed tips — commits walk parents and trees, trees walk entries, tags walk targets, blobs terminate (gix-fsck’s Connectivity::check_commit does not walk parents; this is the owning walk, the same two-stage discipline as generation). A missing closure member fails into the unpack ng leg naming the missing object.
  • Status shapes: pack-level validation failures (parse, index, thin-base miss, the cap, connectivity) land in PreparedPush::unpack’s ng leg with a server-chosen reason; updates returns the input list verbatim on every path (ADR-018 §5); Err stays StorageError — infrastructure only (working-area construction, task joins, backing-store legs). The empty pack (zero objects) is valid and reports as such; the push’s tips must then exist in the store (the same connectivity walk, vacuous over deletes-only updates). The ng legs retain nothing — their guards drop with the prepare call and the just-put objects are the same sweep-eligible recovery the crash case rides; a fully ingested push retains its alkblobs::Batch into the composition-internal PendingPinRegistry keyed by repo_id, correlated content-wise with the verbatim prepared update list (the custody seam the w8-pin-registry task owns).

§Threading

The whole composition runs on tokio::task::spawn_blocking with owned handles moved in (backend.md §concurrency model — the permit stays wire-side, the impl’s threading invisible); every sync→async hop goes through ObjectFront::block_on from the blocking contexts — no consumer constructs its own bridge.

Structs§

PackIngest
The substrate’s GitPackIngest implement.