Module archive_file

Expand description

One repository’s stored bytes in ONE FILE: replicate’s decisions with a body, and a file instead of a transport. archive_out / archive_in. A repository’s stored bytes, in ONE FILE. archive_out writes it, archive_in puts it back.

This is the file-shaped sibling of crate::replicate, and it exists because that module’s decisions were the right ones and its transport was the part nobody had settled. A file needs no transport: no socket, no protocol, no peer, no handshake — and a file can be copied to another appliance, to object storage, to a laptop or to a tape by whatever already moves files. Nothing here opens a socket either, for exactly the reason crate::replicate gives.

Every decision below is crate::replicate’s, restated because it now has a body:

STORE-side, one repository at a time. The unit is one repository’s bytes between two watermarks. It lands whole or it does not land.
from_generation is a WATERMARK, not a diff. ArchiveWrite only ever appends to objects.pack, so everything above a byte offset is new and nothing below it has changed. That is why the parameter is one u64 and not a manifest — and why an incremental archive is a pread of a suffix rather than a comparison of two trees.
NO INDEX TRAVELS. objects.tail (redb) and objects.exploded (Arrow) are derivations, and shipping a derivation is how two copies of a repository come to disagree about the same bytes. The receiver rebuilds its own: crate::git_ops::GitStore::open diffs the journal it is handed against the index rows it has (none, in a fresh store) and re-queues every pack for indexing. A restore is therefore three files and an open.

§What is truth, and what is not

From the table in crate::git_ops::GitStore:

file	what it is	travels?
`objects.pack`	the verbatim pushed pack bytes	yes — the truth (§14)
`objects.pack.journal`	the durable extent rows behind the ack	yes — the indexed bit is derived from it
`refs.log`	the ref transaction log, one Arrow IPC frame per push	yes — refs are not derivable from objects
`objects.tail`	the un-sealed index tail (redb)	no — derivation
`objects.exploded`	§14’s resolved content cache	no — derivation, explicitly droppable

Measured on a one-commit repository: the three that travel are 4 274 bytes; the directory on disk is 1.59 MiB, because objects.tail is a redb file with redb’s own floor. A block-level copy moves the second number.

§The journal and the ref log ship WHOLE, and that is deliberate

Only objects.pack is watermarked. objects.pack.journal is 24-byte rows and refs.log is one small Arrow frame per push: at a thousand pushes they are together a fraction of one pack. Cutting them too would buy little and would need three watermarks where the settled signature has one. The format nevertheless records from_offset per segment, so a later version can watermark them without a format change — the reader already honours it.

§A GC invalidates a watermark, and the file says so rather than lying

crate::gc compacts into a new generation: after it, byte 0 of objects.pack may not be the byte 0 the previous archive measured its watermark against. So every archive records base_prefix_sha256 — the SHA-256 of the first [PREFIX_BYTES] bytes of objects.pack at the instant of the cut — and an incremental cut checks it against the base’s. A mismatch is refused by name (the base has been rewritten…) and the answer is a full archive, not a silent half one.

§The file

magic "GUNNARAF"          8
version u32 LE            4   = FORMAT_VERSION
flags   u32 LE            4   = 0
from_generation u64 LE    8   the objects.pack watermark this file starts above
to_generation   u64 LE    8   objects.pack length at the cut
base_prefix_sha256       32   SHA-256 of objects.pack[0 .. min(len, PREFIX_BYTES)]
segment_count u32 LE      4
reserved      u32 LE      4
per segment:  name_len u16 LE, name bytes, from_offset u64 LE, len u64 LE, sha256 [32]
payloads, back to back, in the order the segments were listed
trailer_sha256           32   SHA-256 of every byte before it

Names are relative to the repository directory (OBJECT_FORMAT, gunnar-git.znippy/objects.pack, …), so a restore reconstructs the shape rather than guessing it, and a reader can say what a file contains without knowing which engine wrote it.

Structs§

ArchiveHeader: The header, as read back.
ArchiveInReceipt: What a completed archive_in did.
ArchiveOutReceipt: What a completed archive_out did.
Segment: One segment as the header describes it.

Constants§

FORMAT_VERSION: Bumped when the layout above changes. A reader refuses a version it does not know rather than reading the bytes it recognises out of a file it does not.
JOURNAL: The durable extent rows. The indexed bit is derived from this.
MAGIC: The first eight bytes of every archive file.
OBJECT_FORMAT: The hash-kind marker, one level up from the store directory.
PACK: The file that is watermarked. Append-only; see the module docs.
PREFIX_BYTES: How much of objects.pack the base fingerprint covers. One 64 KiB pread, which is cheap enough to do on every cut and long enough that two different bases do not collide by accident.
REFS: The ref transaction log.

Functions§

archive_in: Put an archive back into repo_dir.
archive_out: Write one repository’s bytes above from_generation into out.
archive_out_to: The same bytes, into any Write — a file, a socket, a gRPC stream.
base_prefix: SHA-256 of objects.pack’s first PREFIX_BYTES bytes: the base fingerprint an incremental cut is checked against. See the module docs on GC.
find_store_dir: Find the directory that holds objects.pack.
read_header: Read an archive’s header without reading its payload.