Module archive_write 

The push path’s archive writer: one trait, three durability contracts (FastWriter, SafeWriter, UringWriter). The push path’s archive writer: one trait, three durability contracts.

A git client pushes a packfile. Its bytes are already zlib-deflated and often delta-encoded, so they go into the znippy Apache Arrow IPC archive verbatim — never re-compressed, never re-encoded (the reason is measured in this crate’s own module docs: re-inflating per oid cost 3.4× the input on 60 000 objects). What does vary is how much durability the server buys before it sends the client its ack, and that is the only axis this module exposes.

The index tables are not built here. append returns a byte extent; the extent goes over a channel to the per-account indexer in crate::indexer, which builds the Arrow index tables after the bytes are down. See that module for why a read arriving before the index is ready still cannot be wrong.

The three arms, and what each one actually promises

impl	on return, the bytes are…	a crash right after return
FastWriter	in the page cache	loses them
SafeWriter	on the platter, and a journal row points at them	keeps them
UringWriter	on the platter, and a journal row points at them	keeps them

FastWriter bounds the other two: it is the ceiling that the cost of durability is measured against. It is also a selectable arm — see crate::arms::WriterArm — because there are workloads whose contract is not git’s (a rebuildable mirror, a bulk import that is re-run on failure, a benchmark), and the operator who picks it is choosing the row above. Nothing here refuses it; the table is what it promises.

The ordering the two durable arms both obey

Taken from znippy-common/src/hot.rs (:92, :310-327) and not re-invented:

  blob bytes  ->  fsync(blobs)  ->  journal row  ->  fsync(journal)

The blob bytes are durable before any row references them. A crash between the two leaves orphan bytes nobody points at — dead payload the seal drops. The reverse order would leave an index row pointing into a hole, which is a corrupt archive rather than a lost append.

The BufWriter trap

The journal is written through a std::io::BufWriter. flush() moves userspace → kernel; sync_all() moves kernel → platter. sync_all() without a prior flush() syncs nothing you just wrote and looks perfectly durable — the call succeeds, the fsync is real, and the bytes are still sitting in a userspace Vec. SafeWriter deliberately keeps the userspace buffer so that this ordering is load-bearing and a test can see it fail; the guard journal_flush_before_sync_is_load_bearing in tests/archive_write.rs is the one that watched it.

The journal is a LOG, and a reopen appends to it

One Arrow IPC schema message at the head of the file, then one batch message per acked pack, for the life of the archive. A writer opened over an archive that already has a journal appends — it writes no second schema and it truncates nothing.

There is exactly one other kind of row and it is appended the same way: a tombstone ([retire_packs]), written by a gc that found every object of a pack dead. It retires that pack’s extent without removing its row, because removing a row is the one thing this file’s contract forbids. See [JOURNAL_TOMBSTONE] for the encoding and [JournalRow] for why a pack’s ordinal counts pack rows rather than raw rows.

That is not a style choice, it is the crash-recovery contract. The extents in this file are one half of §13.12’s indexed bit: a pack is unabsorbed iff its extent is here and its rows are not in the index, and GitStore::open_with diffs the two on every open. A journal truncated by the reopen would erase the evidence that a durable pack was ever acked, so the pack’s bytes would stay on disk with nothing pointing at them and nothing able to re-queue them — the exact failure the ordering above exists to prevent, arriving one restart later. Emitting a second schema message mid-file would be no better: arrow’s StreamReader stops at it, so [read_journal()] would silently return only the rows written before this process started.

Structs

FastWriter

The cheating arm. Returns before the bytes are on disk.

Faults

Injected faults, so the crash-ordering guard exercises the real append rather than a hand-rolled twin of it (LAW 5: one writer, not two copies that a guard then watches agree).

SafeWriter

The ordering znippy’s own hot path already uses, followed rather than re-invented.

SealReport

What one seal_generation_zero put on disk. Every count is taken off the journal and the sealed file, never echoed back from an argument.

Enums

JournalRow

One journal row, read back and interpreted.

Constants

JOURNAL_ALIGNMENT

Alignment of an IPC message in the journal. 8 is what znippy_common::hot’s journal segments use.

JOURNAL_TOMBSTONE

The blob_size of a journal row that is not a pack: a tombstone, whose blob_offset names a pack this archive has retired.

Traits

ArchiveWrite

Append pushed packfile bytes to an archive, verbatim.

Functions

acked_packs

The acked pack extents, in ordinal order — index i is pack i.

encode_journal_row

Serialize one journal batch message for (offset, len).

encode_journal_schema

Serialize the journal’s schema message — the header an Arrow IPC stream opens with, emitted once per segment.

journal_batch

One journal row for one extent.

journal_options

The IPC write options every arm’s journal uses, so the three arms’ journals are byte-comparable.

journal_rows

Interpret what read_journal returned.

journal_schema

The journal’s schema: the reference from a row to the blob bytes.

read_journal

Read every complete journal row back, tolerating a torn tail.

retire_packs

Retire these packs: one tombstone row each, durable before this returns.

retired_offsets

The offsets of the packs a gc has retired.

seal_generation_zero

Write generation 0.

Type Aliases

Extent

A byte range inside the archive: (offset, len). Same shape as crate::store::Extent.