Skip to main content

Module exploded

Module exploded 

Source
Expand description

§14’s exploded objects table: resolved content, derived from the verbatim truth, built eagerly by the same indexer, droppable at any time. §14’s exploded objects: the CONTRACT — who hands a resolved object over, and what the counters mean.

The store itself is crate::exploded_arrow: one Arrow IPC table, one row per object, payload in the row. This module is what both ends agree on — [PayloadSink] and the sinks that are not a store ([NoSink], [CaptureOne]), plus [ExplodedStats], so no caller can tell the medium apart by its instruments.

§What §14 decided, and what it cost

§14 fixed the shape — the verbatim pack bytes are the truth and the resolved objects are a derived side table — and left one sub-question open, “resolve eagerly or lazily?”. Decided 2026-08-08: EAGER, built by the background indexer, with disk explicitly not a consideration (“we don’t care if disk is tripled — we want wall speed minimum”). So there is no size cap, no threshold and no lazy fallback that skips an object to save space; what an operator can choose is crate::exploded_arrow::ExplodePolicy, and that is a setting rather than a heuristic.

  push ─► verbatim bytes fsynced ─► ack        ← the truth, §14
                   │
                   └─ channel ─► indexer ─► resolve ─► ┬─► objects table  (oid → extent)
                                                       └─► THIS SINK ─► the Arrow table

The decision was right and the first medium was wrong. MEASURED on linux.git, oden 2026-08-13, 11 697 976 objects: 6.4 GB of verbatim pack resolves to 16.8 GB of content — 2.6×, inside the 3× that “tripled” budgeted — and redb held those 16.8 GB in a 204 GB file, 12× page churn, while serialising the gatling_for_each fan-out behind one write transaction at 96.8% of a single core. The eager decision stands; redb is gone from this path.

§Three readers, and the third is why it is not optional

readerhad to donow does
a content readresolve a delta chain, inflate its baseone point lookup
a thin pack’s external base (crate::resolve::BaseSource)read and re-resolve a whole packone point lookup
Derived::graph / reachable() after a clean reopennothing — the payloads were stored nowherea scan of the commits

The third row is a correctness hole, not a speed one. Commit and tree payloads were held only in RAM, so a store that shut down cleanly came back with every pack’s indexed bit legitimately set, nothing re-queued, and an empty commit graph — MEASURED on oden 2026-08-08, 2687 rows and 0 of 551 commits, with reachable() quietly returning a commit alone instead of its closure and gc seeing an empty live set.

§Droppable, and that is not a caveat

Every row is re-derivable from the verbatim pack bytes, so the table can be deleted at any time without consulting a client: [crate::git_ops::Absorber::adopt_journal] compares its row count against the objects table’s and, if it is short, declares no pack absorbed — so every pack is re-queued and re-exploded. Absent means fall back and rebuild; it never means wrong. That is §13.12’s indexed-bit logic applied unchanged to a derived table, and [crate::git_ops::tests::dropping_the_exploded_table_and_reopening_still_answers] asserts it by deleting the file.

§What it costs, measured

MEASURED on oden 2026-08-08, release, one real 2687-object / 5.4 MiB pack, four runs per column, 1-minute loadavg 1.87–2.04. The two columns are the same binary with the sink swapped to [NoSink] and the fold stubbed out, so nothing but the table differs between them.

without the tablewith it
ack (put_pack returns)31.0–34.2 ms31.0–31.3 msunchanged
drain (one pack absorbed)166.7–168.6 ms316.2–329.8 ms1.9x
indexer throughput15 940–16 120 rows/s8 146–8 499 rows/s0.51x

The ack path is unchanged by construction, not by measurement. put_pack walks, checks the closure, appends verbatim, fsyncs twice and queues 24 bytes, and not one of those lines touches this. The measured ack ranges overlap and the wider one is the left column, which is how a null result looks; no ack figure here is evidence of anything.

Those figures are redb’s. They are kept because the ack column is a construction argument that still holds, and the drain column is the honest record of what the eager decision cost when it was taken — not a claim about what the Arrow table costs, which has not been measured on that pack.

Structs§

CaptureOne
A sink that keeps one object: the fallback path’s, for when the table cannot answer and the content has to come back out of the verbatim truth.
ExplodedStats
Counters, all of them applied output: rows that exist, lookups that happened. Nothing here is configuration echoed back.
NoSink
A sink that keeps nothing. What every caller that only wants oids passes, and what makes “this resolve built no exploded rows” a visible choice rather than an omission.

Traits§

PayloadSink
Where every object the resolver produces is handed over.