Module wal 

Record framing for the log-first storage substrate. This is the write-ahead log: crate::log_store appends these framed records to segment files, and that durable append is the commit’s ack boundary (not the buffered redb materialization). See the manual’s docs/manual/architecture/storage-engine.md.

Every record (an expanded op-set, optionally paired with its originating CommitBatch — encode_ops_record) is wrapped in a fixed header with magic, format version, shard id, LSN, payload length, and a CRC32C-style checksum. This lets replay/export/CDC distinguish valid old data, valid new data, partial new data, and corrupted data instead of hoping the filesystem write was atomic.

Greenfield: only the framed format is read or written. There is no legacy unframed format to migrate. An in-memory store (no WAL file) instead keeps a row-based command journal — the one remaining user of the bare __log__/<lsn> row format.

Structs

ChainAssembler

Reconstructs a (possibly chained) commit from consecutive framed WAL records, fed one at a time in physical log order. The overwhelmingly common case — a BARE/WITH_COMMAND record — reconstructs immediately. A chained commit’s CHAIN_CONTINUE fragments are buffered until the matching CHAIN_FINAL is fed, at which point the whole op-set is returned — in full, exactly once, never partially. This is the one reconstruction implementation crash recovery and replication apply both use, so there is never a second, potentially drifting copy of this logic.

Checkpoint

A published durability checkpoint: the durable LSN (header), the GC floor, and the wall-clock time the checkpoint was published. The record’s frame checksum makes a torn/partial checkpoint fail verification on load.

SnapshotManifest

Manifest header for a framed snapshot stream (W6, D9/P17). Written as the first framed chunk; consumers validate magic/format_version before reading tenant chunks. tenants_crc covers the concatenated tenant chunks, so a torn/corrupted stream is rejected at restore time.

TenantChunk

One tenant’s framed backup chunk: scope (TenantId bytes) + JSONL payload.

WalHeader

A decoded logical WAL record header.

WalSegmentManifest

Manifest describing one exported WAL segment range. Written first in a segment stream so a consumer can validate the range before reading records.

Enums

ChainFeed

Result of feeding one record to a ChainAssembler.

WalDecodeError

Error returned when a logical WAL record fails to decode or verify.

Constants

SEGMENT_MAGIC

Magic identifying a WAL segment manifest.

SNAPSHOT_MAGIC

Framed snapshot export magic: distinct from the WAL segment magic so a restore can reject a mismatched artifact type immediately.

WAL_FORMAT_VERSION

Current logical WAL framing format version.

Functions

decode_checkpoint

Decode a checkpoint from a framed WAL record.

decode_last_applied

Decode the LAST_APPLIED watermark row.

decode_logical_body

Decode a ChainFeed::Complete logical body into its op-set and optional command. Kept separate from ChainAssembler::feed (see ChainFeed::Complete’s doc) so framing/chain-structure completeness and content validity are never conflated.

decode_op_set

Decode an op-set encoded by encode_op_set.

decode_ops_record

Decode a framed op-set record, verifying the frame checksum. The returned header carries the routing shard and LSN; the CommitBatch is Some only for a record encoded with one (every real client commit through the normal publish path — see encode_ops_record).

decode_tenant_chunk

Decode a tenant chunk body (the inverse of encode_tenant_chunk).

decode_wal_record

Decode and verify a framed record; returns the header and payload.

encode_chained_ops_record

Split ops (optionally paired with command_payload) into one or more framed WAL records at (shard_id, lsn), each a complete, independently checksummed frame whose total length never exceeds max_record_bytes. All returned records share one LSN; only the last is tagged [OPS_RECORD_KIND_CHAIN_FINAL] (or, when everything already fits in one record — overwhelmingly the common case — the single returned record is byte-identical to encode_ops_record’s ordinary BARE/WITH_COMMAND output). When chaining is needed, the logical body (op-set plus optional command, laid out exactly like WITH_COMMAND’s own format) is cut at arbitrary raw byte boundaries — never aligned to op or command boundaries — which is what lets a command payload of any size be chained too, not just the op-set: nothing is decoded fragment-by-fragment, only once every fragment of the chain is reassembled (see ChainAssembler).

encode_checkpoint

Encode a checkpoint as a framed WAL record (LSN carried in the header, floor + timestamp in the payload).

encode_last_applied

Encode the LAST_APPLIED watermark row (a framed record with an empty payload carries the LSN in the header).

encode_manifest

Encode a manifest as JSON (used as the first framed chunk of a stream).

encode_op_set

Encode an expanded op-set as a compact binary WAL payload: an op count followed by (tag, key, [value]) entries. This is the log-first WAL’s record content (decision B): the log stores state transitions, so recovery applies them directly and needs no expander.

encode_ops_record

Encode an expanded op-set as a framed WAL record at (shard_id, lsn), optionally paired with the serialized CommitBatch that produced it (carrying the command in the WAL record is what makes the command journal — commit_key per LSN — fully derivable from the WAL, closing the __log__ redb row that used to be the only source of it). command_payload is the batch pre-serialized by the caller (already computed once for the outcome-payload row), not re-serialized here.

encode_tenant_chunk

Encode a tenant chunk: 4-byte tenant length, tenant bytes, JSONL bytes. The whole chunk is written as one framed chunk (CRC-verified on read).

encode_wal_record

Encode one logical record into a framed, checksummed byte vector.

now_ms

Wall-clock now in milliseconds since the Unix epoch (used for retention lease deadlines and checkpoint publication timestamps).

op_set_encoded_len

Exact byte length encode_op_set would produce for ops, computed without allocating — used for cheap pre-commit size checks (deciding whether a commit needs chaining, or exceeds the configured transaction ceiling entirely) without building the buffer just to measure it.

read_framed_chunk

Read one framed chunk written by write_framed_chunk.

read_snapshot

Read one framed snapshot stream, verifying the manifest (magic + version) and every tenant chunk CRC. Returns the manifest and decoded tenant chunks. Truncated or corrupted input is rejected: if the stream ends before tenant_count chunks are read, or a chunk fails its CRC, this errors.

read_snapshot_manifest

Read + validate only the snapshot manifest (magic/version), so a caller can stream the tenant chunks one at a time instead of materializing them all.

read_wal_segment

Read one WAL segment written by write_wal_segment, verifying the manifest and every record checksum. Returns (manifest, records).

write_framed_chunk

Framed-stream writer for export/backup: emits {len}{crc}{bytes} chunks so a consumer can detect truncation/corruption without reading the whole artifact. (P17 foundations.)

write_snapshot

Write one framed snapshot stream: manifest chunk, then one framed chunk per tenant (each independently checksummed; the manifest carries a CRC over all tenant chunks). Returns the manifest.

write_wal_segment

Write one full WAL segment: manifest chunk, then one framed chunk per record (each chunk independently checksummed; the manifest also carries a CRC over all records). Returns the manifest.