Module indexer 

The shared, off-the-ack-path index builder — one indexer per account, gatling fan-out, index tables built last. The shared index path: one indexer per account, off the ack path.

Every ArchiveWrite arm uses this. It is not per-impl and it is not part of the durability contract — it is what happens after append has already returned to the pushing client.

  push ──► ArchiveWrite::append ──► ack to client
                   │
                   └─ (pack_id, offset, len) ──► account's channel
                                                       │
                                      ┌────────────────┘  worker SLEEPS on
                                      ▼                   recv() when empty
                                 drain a batch
                                      │
                                 gatling fork-join  (NEVER rayon — LAW 3)
                                      │           one pack row per job
                                      ▼
                                 ObjectAbsorb   ── the pack's OBJECT rows,
                                      │            oids and all
                                      ▼
                                 Arrow index tables + "indexed" bit

One indexer per account

Each account gets its own AccountIndexer: its own channel, its own worker. An account pushing a monorepo cannot delay another account’s small push, because there is no queue and no worker between them to contend for. The target box is 192 cores / 8 TB; an idle account costs a parked thread and nothing else — the worker blocks in recv(), it does not poll and it does not spin. idle_indexers_sleep_they_do_not_spin measures that as CPU time rather than trusting the sentence.

(This is deliberately not gunnar’s P-004 situation. There the fan-out is thousands of concurrent short-lived requests, which is why the serve path pins thread_limit: Some(1). Here it is long-lived, mostly-idle accounts, and a parked thread per account is the cheap answer.)

Zero-copy handoff

IndexJob is { pack_id, offset, len } — 24 bytes, Copy, no pointer into the pack. Extents cross the channel, never buffers. The worker preads the extent back when it gets to it, so a 2 GiB push costs the channel 24 bytes and the ack path zero copies.

Fan-out inside a drain is gatling

A drained batch is handed to gatling_forkjoin::gatling_for_each: N workers self-dispatch off one atomic cursor, no barrier, std::thread::scope. rayon is banned across this constellation (LAW 3) and rayon_free_law.rs enforces it.

The object-level half

A pack row is derivable from the extent alone — version, object count, a hash of the stored bytes. Object rows are not. They need the pack walked, its delta chains applied and every oid hashed, which is git-shaped work this file must not know how to do. So the drain calls out, through [ObjectAbsorb], and GitStore implements it.

The absorber it calls is the same object a falling-back read calls, not a background copy of it (LAW 5): one absorb, one gate, one place the objects table is written. The two paths cannot drift because there is only one of them.

Ordering inside one drained job is fixed and it is the whole safety argument:

  pack row built  ─►  objects absorbed  ─►  indexed bit set

The bit goes up last, so it cannot be set over an index that does not yet hold the pack’s objects. An absorb that fails leaves the bit clear and the error counted (AccountIndexer::absorb_failures) — the pack stays on the fallback path, which is slow and right, rather than being declared indexed, which would be fast and wrong.

Why an early read cannot be wrong

The index tables are built last. A read arriving between the ack and the index is therefore reading an archive whose index does not mention the pack. Rather than let it conclude “absent”, every pack carries an indexed bit, which is membership in the published-row map — one structure, so the bit and the row cannot drift apart. AccountIndexer::lookup returns Lookup::Indexed only when the bit is set, and otherwise hands back the journal’s extent list so the caller falls back to scanning — slower, never wrong. For FastWriter there is no journal to fall back to and the answer is Lookup::Unknowable: one more thing that arm does not promise.

Structs

AccountIndexer

One account’s indexer: its own channel, its own worker, its own tables.

IndexJob

One unit of index work. Offsets and extents only — never a buffer.

IndexRow

One row of the built index — everything derivable from the pack bytes, which is exactly the work that was kept off the ack path.

IndexerPool

One indexer per account, created on first push from that account.

PushPath

A git server’s push path: one ArchiveWrite arm plus the shared indexer.

Enums

Lookup

What a read gets when it asks for a pack.

Traits

ObjectAbsorb

The object-level ingress, called by the drain and by nothing else here.

Functions

index_schema

The index table’s Arrow schema.