Skip to main content

Crate seglog

Crate seglog 

Source
Expand description

A simple, high-performance segment log implementation.

seglog provides low-level read and write operations for fixed-size segment files with built-in CRC-32 validation. It’s designed for use in event sourcing systems, write-ahead logs, and other append-only storage use cases.

§Architecture

§Single Writer, Multiple Readers

The segment log follows a single writer, multiple concurrent readers model:

  • One Writer can append records to a segment at a time
  • Multiple Readers can concurrently read from the same segment across threads
  • The FlushedOffset provides thread-safe coordination between writers and readers

This design ensures data consistency without locks on the read path, making it highly efficient for workloads with many concurrent readers.

§Following from outside the writer’s process

Tail extends that model past the process boundary. It opens a segment O_RDONLY and advances over the committed prefix on demand, so a reader that shares no memory with the writer (and may hold no write permission at all) can follow a live segment. A Reader for such a segment comes from Reader::open_read_only, since the ordinary Reader::open asks for a writable descriptor it needs only to rewrite headers.

A follower and crash recovery mean the same thing by “committed”: both walk the prefix through one shared implementation, so a Tail will never expose a record that Writer::open would have rolled back. What a follower reads is always a prefix, and never a torn record or an uncommitted batch. It is not a durability oracle: a batch becomes visible when its commit marker reaches the page cache, which is just before the writer’s fsync returns, so a power loss in that window erases events a follower has already seen.

§Record Format

Each record consists of:

┌─────────────┬─────────────┬────────────────┬─────────────────────┐
│ Length (4B) │ CRC-32 (4B) │ Header (H B)   │ Data (N bytes)      │
└─────────────┴─────────────┴────────────────┴─────────────────────┘
  • Length: 32-bit little-endian total payload length (H + N bytes). Bit 30 flags a control record; bit 31 is reserved and must be zero.
  • CRC-32: 32-bit checksum over length + header + data
  • Header: Fixed-size metadata (H bytes)
  • Data: Variable-length record payload

Total record header size is RECORD_HEAD_SIZE (8 bytes), not including the user header.

§Concurrent Safety via FlushedOffset

The FlushedOffset is an atomic counter that tracks how much data has been safely written to disk:

  • Writers update it after calling Writer::sync
  • Readers check it to avoid reading uncommitted data
  • It’s shared via Arc for efficient cloning across threads

This ensures readers never see partial writes or corrupted data.

§Performance Optimizations

§Read Hints

The ReadHint enum allows optimizing for different access patterns:

§Optimistic Reads

For random access, the reader performs an optimistic read of the header plus 2KB of data in a single syscall. If the record fits (most events in event sourcing do), this eliminates one syscall per read, improving performance by ~40% for small records.

§Examples

§Writing Records

use seglog::write::Writer;

let segment_size = 1024 * 1024; // 1 MB
// Writer with no header (H = 0)
let mut writer = Writer::<0>::create(&temp, segment_size, 0)?;

// Append records with empty header
let (offset, len) = writer.append(&[], b"event data")?;
writer.append(&[], b"more events")?;

// Sync to make data visible to readers
writer.sync()?;

§Reading Records (Sequential)

use seglog::read::Reader;

let mut reader = Reader::<0>::open(&temp, Some(flushed))?;

// Iterate over all records
let mut iter = reader.iter(0);
while let Some(record) = iter.next_record()? {
    println!("Record: {record:?}");
}

§Reading Records (Random Access)

use seglog::read::{Reader, ReadHint};

let mut reader = Reader::<0>::open(&temp, Some(flushed))?;

// Read specific record by offset
let record = reader.read_record(offset, ReadHint::Random)?;
println!("Record: {record:?}");

§Concurrent Readers

use seglog::read::Reader;
use std::thread;

let reader = Reader::<0>::open(&path, Some(flushed))?;

// Clone reader for use in another thread
let reader2 = reader.try_clone()?;

let handle = thread::spawn(move || {
    // Read from the segment in a different thread
    // Both readers share the same FlushedOffset
});

§Segment Lifecycle

  1. Create: Use Writer::<0>::create to initialize a new segment
  2. Write: Append records with Writer::append
  3. Sync: Periodically call Writer::sync to flush data to disk
  4. Read: Open readers with Reader::<0>::open using the shared FlushedOffset
  5. Truncate (optional): Use Writer::set_len to truncate the segment
  6. Close: Call Writer::close or Reader::close to ensure cleanup

§Error Handling

The crate defines two main error types:

  • ReadError: CRC mismatch, out of bounds reads, I/O errors
  • WriteError: Segment full, I/O errors, read errors during recovery

Modules§

control
Control record kinds (first byte of a control record’s payload).
parse
read
tail
Following a segment that someone else is writing.
write

Structs§

FlushedOffset
Thread-safe atomic offset tracking for flushed data.

Constants§

COMMIT_MARKER_PAYLOAD
Payload size of a batch commit record: kind byte + u64.
CONTROL_FLAG
Control-record flag bit in the length field (bit 30). When set, the record is internal framing, not caller data.
FLAG_MASK
Known flag bits (currently just the control flag). Bit 31 is reserved and must be zero: a record with it set is rejected as corrupt. Any bit outside FLAG_MASK | LENGTH_MASK is corruption.
LENGTH_MASK
Mask to extract actual length from the length field (bits 0..=29).
MAX_RECORD_LEN
Largest encodable record length (1 GiB - 1).
RECORD_HEAD_SIZE
Size of the record header in bytes, consisting of length and CRC-32 checksum.

Functions§

calculate_crc32c