Struct PushLog 

pub struct PushLog { /* private fields */ }

An append-only log file. One append call is one push.

Implementations

impl PushLog

pub fn new(path: impl Into<PathBuf>, schema: Arc<Schema>) -> Self

pub fn path(&self) -> &Path

pub fn schema(&self) -> &Arc<Schema>

pub fn append(&self, batch: &RecordBatch) -> Result<u64>

Append one push and fsync it. Returns the byte offset the frame starts at.

The frame is built fully in memory first, so the single write_all is the only thing the crash window covers; and the sync_all is what makes “the frame is complete on disk” — the durability claim this format rests on — actually true rather than merely likely.

pub fn scan(&self) -> Result<PushLogScan>

Recover the log. A missing file is an empty, clean log — a repository that has never been pushed to is not an error.

pub fn frame_count(&self) -> Result<usize>

How many complete frames the log holds.

pub fn compact(&self) -> Result<CompactionReport>

Fold every frame into one, atomically.

What it costs and what it buys

Rows are concatenated, never dropped: push_seq and updated_ms ride in the columns, so the full history survives and crate::refs::fold gives the identical answer before and after. What is given up is the frame boundary as a record of where one push ended — after compaction a frame is a run of pushes, and push_seq is the only thing that says where the seams were. That is why the ordering key was never the frame index.

Compatibility — no migration, no version bump

A compacted log is still exactly what an uncompacted one is: a sequence of self-contained frames, each carrying one batch and ending in EOS. There are simply fewer and larger ones. scan_frames already reads any number of them and its one-batch-per-frame invariant still holds, so a log written before this existed reads unchanged and a compacted log reads on any reader that could read the old one. This is deliberately not a versioned format change.

Crash safety

The same shape as gunnar-store’s arrow_log.rs — which owns the mature version of this problem — and deliberately so, since the code cannot be shared across the repository boundary: stage the whole replacement under [compacting_path], fsync it, rename it over the live log, then fsync the parent directory so the rename itself is durable. The live log is never written in place, so an interruption at any byte leaves the old log serving untouched; the worst outcome is a stray .compacting file that the next compaction overwrites.

pub fn compact_with(&self, finish: Finish) -> Result<CompactionReport>

compact, stopping where finish says. The crash test is the only caller that passes anything but Finish::Swap.

pub fn maybe_compact( &self, policy: CompactionPolicy, ) -> Result<Option<CompactionReport>>

Compact if policy says the log has grown enough. Returns None when it did not run.

pub fn seal_section(&self, module_name: &str) -> Result<ReservedSection>

Fold the recovered pushes into the reserved section to seal into an archive, preserving one RecordBatch per push.