Module exploded_arrow 

The exploded objects table: ONE Arrow IPC file, one row per object.

What this replaces, and what it measured

crate::exploded holds oid → [kind][payload…] in redb. It works, and it is the wrong medium — MEASURED on linux.git, oden 2026-08-13, 11 697 976 objects:

pack as pushed (verbatim, deflated + delta-encoded)	6.4 GB
its objects fully resolved (git verify-pack -v, summed)	16.8 GB
objects.exploded on disk	204 GB

So the 2026-08-08 eager decision was RIGHT — resolved content is 2.6× the pack, comfortably inside the 3× it budgeted — and the medium was wrong by 12×. redb is a copy-on-write B-tree: 11.7 million small keys committed in FLUSH_BYTES batches rewrite interior pages over and over, and 187 of those 204 GB are page churn rather than anybody’s data.

It cost throughput for the same reason. crate::indexer fans a drained batch across N workers with gatling_for_each (LAW 3, never rayon) and every one of them then queued behind redb’s single write transaction — MEASURED: 96.8% of ONE core with 31 idle while the exploder ran.

The shape

One table, one file, the payload IN the row — which is what this engine already does everywhere else (crate::reach stores a bitmap as DataType::Binary, crate::secrets a ciphertext):

  oid | object_type | mode | path | payload

An Arrow IPC stream is [schema message][batch message]… — append-only, no pages to rewrite, so the file is the payload total plus framing. Each flush encodes one batch with IpcDataGenerator and appends it with write_message; nothing earlier in the file is ever touched again.

payload is LargeBinary and not Binary deliberately: Binary offsets are i32, so one batch would cap at 2 GiB of payload and overflow silently on a corpus nobody tested. The wider offset costs four bytes a row.

mode and path — the row IS a file

A git object on its own is content without a name. The two nullable columns carry the name when the caller knows it (a tree walk does; a bare pack resolve does not), which is what lets this table be read as a filesystem rather than as a content-addressed bag. They are nullable and best-effort for a reason that is inherent and not a shortcut: one blob is reachable at many paths in many commits, so a path column can only ever record a path — the one it was first exploded at. Null means “not known here”, never “at the root”.

Reopening is cheap, and the index is lazy

A batch’s metadata carries its row count and its body length, so ExplodedArchive::open walks the message framing — seeking over the bodies, reading kilobytes — and knows how many rows the table holds and where every batch begins without touching a gigabyte. That is what adopt_journal needs on startup.

The oid → row index costs one sequential pass and is therefore built on the first lookup, not on open. A push-heavy process never pays for it; a process that reads pays once. After that a point read is two preads — the schema message and the one batch — decoded through StreamDecoder, so it never scans the batches in front of the one it wants.

The lookup is ragnar’s static tree — see [OidTree]

That index used to be a Vec<(Vec<u8>, Located)> walked with binary_search_by: one heap allocation per oid — 11.7 million of them on linux.git — and a log₂ n chain of dependent loads over scattered Vec headers. It is now the same stree (znippy-zoomies) this crate already puts over oids in crate::oid_index, over flat parallel arrays. nornir-workspace.toml’s performance law names that structure by name; this is the second place in this crate that obeys it, and it obeys it by reusing crate::oid_index::key_for_oid rather than deriving a second key.

Droppable, and it is one rm

Every row is re-derivable from the verbatim pack bytes, so the file can be deleted without consulting a client: [crate::git_ops::Absorber::adopt_journal] compares this table’s row count against the objects table’s and re-queues every pack if it is short. Absent means fall back and rebuild; it never means wrong. One file, one delete — which is the property the engine’s own tests assert.

Structs

ExplodedArchive

§14’s derived table for one repository.

Stats

Counters, all of them applied output.

Enums

ExplodePolicy

How much of a pack is resolved into the exploded table.

Constants

COL_ENC

oid | object_type | mode | path | payload.

COL_MODE

COL_OID

COL_PATH

COL_PAYLOAD

COL_TYPE

ENC_RAW

ENC_ZLIB

FLUSH_BYTES

Buffered payload bytes that force a flush.

MIN_ROW_COMPRESS

Rows shorter than this are never compressed: a zlib header and trailer are 8 bytes, and a tiny tree does not earn them back. Set from examples/explode_compress_ratio.rs over the kernel mirror’s pool.

TOMBSTONE

A row that retires an oid.

Functions

explode_default

The current process-wide default — what an unset environment means right now.

exploded_schema

Schema v2 (2026-09-04): enc sits BEFORE payload, so the payload’s data buffer stays the LAST buffer of the batch — which is what [payload_data_offset] reads the extent off. A v1 file has no enc column; every reader treats a missing column as ENC_RAW, so pools written before this open unchanged and are never rewritten for it.

exploded_schema_v1

Schema v1 — what every pool written before 2026-09-04 carries, and what this build keeps appending to such a pool (see ExplodedArchive::legacy_v1).

kind_code

The pack type codes, the same mapping crate::exploded uses, so the two media cannot disagree about what a 2 means.

kind_of

set_explode_default

Set the process-wide default explode policy for every store opened AFTER this call (an explicit crate::arms::ENV_EXPLODE still wins). The twin of crate::arms::set_graph_cache_default, for an embedder that learns its role at runtime and cannot set the environment — edition 2024 makes set_var unsafe, and gunnar forbids unsafe — such as gunnar’s twin, which holds packs, serves no reads, and must store no exploded pool at all (ExplodePolicy::Off).