Module pushlog 

The append-only push log: the one mechanism __gunnar_refs__ and __gunnar_secrets__ are both built from.

Why there is no database here

D18 removed redb: it was a database wedged between two archive formats, and it bought nothing that the formats did not already provide. What a ref update actually needs is atomicity — a push is visible whole or not at all — and Arrow IPC already frames its messages. So the transaction boundary is the IPC frame boundary, and the rule is one RecordBatch per push.

The on-disk shape

The live log is a concatenation of self-contained Arrow IPC streams, one per push:

[ schema | RecordBatch | EOS ]  ← push 1, fsynced
[ schema | RecordBatch | EOS ]  ← push 2, fsynced
[ schema | RecordBatch          ← push 3, TORN by a crash

Repeating the schema per frame is what buys the property: every frame is independently parseable, so recovery needs no side-car, no length table and no journal.

What that costs, measured (refs schema, 4 refs per push, 2026-08-03): a frame is 2056 bytes — 448 of schema (21.8%) and 1600 of batch (77.8%, i.e. 400 bytes per ref). So the schema repetition is the smaller half of the overhead; the larger half is Arrow’s 64-byte buffer alignment paid six times over on a four-row batch. Merging frames into one stream would therefore recover only ~21.8%.

The real win is compaction — folding many small batches into one large one, which amortises the padding as well as the schema. gunnar-store’s arrow_log.rs already does exactly this (CompactionPolicy, a staged rewrite plus rename, and a reflog archive for the history it retires), and its single-stream live log carries the schema once rather than per push. That is the shape to adopt here rather than reinvent; see the note in the commit that added this measurement. scan_frames reads forward while frames parse and stops at the first one that does not — push 3 above is discarded whole, pushes 1 and 2 survive intact. That is the crash story, and it is a property of the framing rather than of any code that has to run.

A partially-written frame can never be mistaken for a complete one: Arrow prefixes every message with a continuation marker and a length, so a truncated frame ends in the middle of a message body and the reader errors instead of yielding a half-populated batch.

Live log vs sealed archive

znippy seals later. Until then the log is the file above. At seal, the recovered batches are folded into one reserved Arrow section that keeps one RecordBatch per push — so the push boundaries survive into the sealed archive, and DuckDB / Polars / DataFusion read the section straight out of its manifest byte range with no gunnar code.

Structs

CompactionPolicy

When a log should be compacted.

CompactionReport

What one compaction did.

PushLog

An append-only log file. One append call is one push.

PushLogScan

What a scan of a log recovered, and what it had to throw away.

Enums

Finish

Where a compaction is allowed to stop. Only the crash test constructs anything but Finish::Swap.

Functions

encode_frame

Serialize one push as a self-contained Arrow IPC stream frame.

read_sealed

Read a sealed push-log section back out of an archive. Ok(None) when the archive carries no such section — distinct from a section with no pushes.

scan_frames

Read forward over concatenated frames, stopping at the first that does not parse. Never errors on a torn tail — that is the expected state after a crash and the whole reason the framing was chosen.