Module segment 

Log segment format, the in-memory log writer and the recovery scanner.

A shard’s log is a sequence of segment files. Each segment starts with one 4 KiB header block, followed by frames:

frame  = len:u32le  crc:u32le  lsn:u64le  batch:u32le  body[len]
crc    = crc32c(lsn ++ batch ++ body)

batch is the distance from the first record of the flush batch the frame was written in. Only one flush is in flight per log, so a batch is written only after the previous one was durable. That lets recovery tell a torn, never-acknowledged batch (whose writes the OS may have completed in any order) from a damaged record that had been acknowledged.

lsn is the record’s log sequence number: consecutive across the shard’s whole log, starting at the segment header’s base_lsn. The body is a record in segment encoding (see crate::record); names are interned per segment, so a segment can be read without any other file.

The writer here does no I/O. It turns records into addressed byte Chunks, and the I/O layer (real files or the simulator’s disk) writes them and flushes. Every open of a log starts a fresh segment file, so bytes left behind by a torn write are never followed by new records in the same file. (An open may reuse the number of a trailing segment that holds no valid record, after deleting that file.)

Recovery reads each segment with SegmentScanner over a growing prefix of the file, stopping just past the valid records (scan_segment is the same scan over a whole image). A frame that fails its checksum, carries the wrong LSN or does not decode ends the valid prefix. If a later frame in the segment is still valid, a durable record was lost in the middle of the log (Tail::Corrupt); otherwise the segment simply ends in a torn write that was never acknowledged (Tail::Torn).

Structs

Batch

Everything appended since the previous LogWriter::take.

Chunk

Bytes to write at offset of segment segment. A chunk that starts a segment begins with its header block at offset 0.

LogWriter

Appends records to a shard log in memory. Assigns LSNs, interns names per segment and decides segment boundaries. Pure: no files.

Scanned

SegmentHeader

SegmentNames

Decodes interned names while scanning one segment.

SegmentScanner

Scans a segment from a growing prefix of its bytes, so recovery reads only the valid records plus the first zero block after them instead of the whole preallocated file. Fed the complete image, it returns exactly what scan_segment returns.

TooLarge

Enums

Progress

What SegmentScanner::advance concluded from the bytes it was given.

ScanError

Tail

How a scanned segment ended.

Constants

DEFAULT_SEGMENT_BYTES

FORMAT_VERSION

The on-disk format version written into every segment header. A reader refuses segments with a newer version.

FRAME_HEADER

HEADER_BYTES

MAGIC

Functions

headerless_was_flushed

For a segment whose header is unreadable: whether its body proves the segment had been activated and flushed (a valid frame from a batch after the first). If not, the header was torn by the segment’s first, never acknowledged flush.

recycle_bytes

Keeps bytes for a later spare_bytes on this thread.

scan_segment

Scans a segment, calling visit for each valid record in order. Returns Ok(None) for a never-activated (all-zero header) segment.

spare_bytes

An empty byte buffer with room for at least min bytes, reused when this thread has one.