Module wal 

Log format 3: one write-ahead log per device, shared by every shard.

A log group is a sequence of segment files. Each segment begins with one 4 KiB header block and holds batches back to back:

batch   = header  section*  padding
header  = len:u32  crc:u32  body_len:u32  flags:u32
          seq:u64  prev:u64  acked:u64  life:u64  sections:u32  reserved:u32
section = shard:u32  count:u32  first_lsn:u64  bytes:u32  reserved:u32  records
records = (varint length, canonical record body)*
crc     = crc32c(salt ++ batch[8 .. body_len])

len is the distance to the next batch (a multiple of 8, past any padding); body_len is what the checksum covers. seq numbers batches across the whole group. prev is the batch_checksum of the batch before it, so batches form a chain that stale bytes cannot join. acked is the highest sequence the writer had acknowledged when it submitted the batch. life is random per open of the log: two lives never write identical batches, so a stale batch’s checksum never matches a new one.

salt is in the segment header and is new for every life of a segment file (a recycled file gets a new header and salt), so bytes from a segment’s earlier life never pass the checksum.

Each shard’s records keep their own LSN sequence: a section carries one shard’s consecutive records starting at first_lsn. Records use the canonical encoding (crate::record::encode_canonical), the same bytes the run’s journal and replication use.

Acceptance (spec/FuseeLog.tla). Recovery accepts batches from a known position while each passes its checksum, has the next sequence and names the accepted batch before it. Where the chain ends, a later batch of the same segment life whose acked reaches past the end proves an acknowledged batch was lost: End::Damaged. Otherwise the tail was in flight and never acknowledged, and the next open resumes in place after it (End::Clean or End::Torn). Up to several writes may be in flight at once: the device may complete them in any order, which is why a valid batch after an invalid one proves nothing unless its acked says the invalid one had been acknowledged.

Batches of writes that may be in flight together never share a 4 KiB block: the writer pads a batch to the next block boundary when the next write may start before this one completes, so one write landing late can never overwrite another’s bytes.

Re-exports

pub use crate::wire::xxh64;

Structs

BatchHeader

The fields of a batch header.

BatchView

A batch that passed its checksum.

Large

A record over the offload threshold, found in a section by find_large.

OffloadRef

Where an offloaded record’s canonical body lives in the blob area.

Placed

Bytes to write at offset of segment; header is set when the batch starts a new segment (write the header block first, or together).

Scanned

The accepted chain through one segment.

SectionRecord

One record of a section: where its length prefix starts, and its body.

SectionRef

One shard’s consecutive records in a batch.

SectionWriter

One shard’s side of a shared log: assigns the shard’s LSNs and encodes its records, once, into the section its next batch carries. Pure.

WalHeader

The header block of one life of a segment file.

WalScanner

Follows the chain through one segment from a growing prefix of its bytes, like crate::segment::SegmentScanner.

WalWriter

Where the next batch goes. Pure: the I/O layer writes the bytes.

Enums

End

The end of the chain in a segment.

Progress

What WalScanner::advance concluded.

ResolveError

Why an offloaded record could not be read back.

WalError

Constants

BATCH_HEADER

BLOB_ALIGN

Blob offsets are multiples of this, so each blob starts on a block of its own (whole-block writes never touch another blob).

BLOCK

DAMAGE_WINDOW

How far past the end of the chain recovery looks for a batch proving that an acknowledged batch was lost. A proof is written within a few batches of the loss; damage with no proof that close is indistinguishable from a torn tail, as with any log.

FLAG_LIFE_START

Flag: the first batch of a writer life.

MAX_BATCH

A batch is at most this large (a segment must hold it).

MAX_SECTION

The most bytes of records a shard hands to its log at a time (one record at least, however long). A shard that fell behind (a stalled flush, a burst of starts) then writes its backlog as sections of this size, so no batch, replicated entry or protocol frame built from sections grows with the backlog; a flush that keeps up never comes near it.

OFFLOADED

Kind byte of a reference to an offloaded record. Never a record kind.

SECTION_HEADER

WAL_FORMAT

WAL_HEADER_BYTES

WAL_MAGIC

Functions

batch_body_len

Encoded size of a batch carrying sections, before padding.

batch_checksum

The chain checksum of a batch: its covered bytes, salt excluded, so the chain continues across segments.

batch_span

How far the next batch starts from a batch of body_len bytes placed at offset (a multiple of 8): the body rounded up to 8 bytes, or, when pipelined, up to the end of the block it ends in, so that the next batch starts on a block boundary even when this one did not. Writes in flight together then never share a block, which O_DIRECT (whole-block writes) needs. A batch that would grow past MAX_BATCH that way keeps the 8-byte rounding.

encode_batch

Appends one batch to out, padded to a multiple of align (8 or BLOCK). Returns its chain checksum.

encode_records

Encodes records for a section.

find_large

The records of a section’s bytes whose canonical body is longer than threshold, with their checksums. Pure and lock-free: the shard that wrote the section runs it, so the checksum of a large payload costs only that shard.

for_each_offload

Calls f with the index in the section and the reference of every offloaded record in a section’s bytes.

is_offloaded

Whether a section record body is a reference to an offloaded record.

section_records

Decodes a section’s records with their LSNs.

section_records_resolved

section_records for a section that may hold references, keeping only the records after after (the others are neither decoded nor read back): fetch reads the bytes a reference names (they are checked here).

section_spans

The records of a section’s bytes, in order, with their positions.