Expand description
A simple, high-performance segment log implementation.
seglog provides low-level read and write operations for fixed-size segment files
with built-in CRC-32 validation. It’s designed for use in event sourcing systems,
write-ahead logs, and other append-only storage use cases.
§Architecture
§Single Writer, Multiple Readers
The segment log follows a single writer, multiple concurrent readers model:
- One
Writercan append records to a segment at a time - Multiple
Readers can concurrently read from the same segment across threads - The
FlushedOffsetprovides thread-safe coordination between writers and readers
This design ensures data consistency without locks on the read path, making it highly efficient for workloads with many concurrent readers.
§Following from outside the writer’s process
Tail extends that model past the process boundary. It opens a segment O_RDONLY and
advances over the committed prefix on demand, so a reader that shares no memory with the
writer (and may hold no write permission at all) can follow a live segment. A Reader
for such a segment comes from Reader::open_read_only, since the ordinary
Reader::open asks for a writable descriptor it needs only to rewrite headers.
A follower and crash recovery mean the same thing by “committed”: both walk the prefix
through one shared implementation, so a Tail will never expose a record that
Writer::open would have rolled back. What a follower reads is always a prefix, and
never a torn record or an uncommitted batch. It is not a durability oracle: a batch becomes
visible when its commit marker reaches the page cache, which is just before the writer’s
fsync returns, so a power loss in that window erases events a follower has already seen.
§Record Format
Each record consists of:
┌─────────────┬─────────────┬────────────────┬─────────────────────┐
│ Length (4B) │ CRC-32 (4B) │ Header (H B) │ Data (N bytes) │
└─────────────┴─────────────┴────────────────┴─────────────────────┘- Length: 32-bit little-endian total payload length (H + N bytes). Bit 30 flags a control record; bit 31 is reserved and must be zero.
- CRC-32: 32-bit checksum over length + header + data
- Header: Fixed-size metadata (H bytes)
- Data: Variable-length record payload
Total record header size is RECORD_HEAD_SIZE (8 bytes), not including the user header.
§Concurrent Safety via FlushedOffset
The FlushedOffset is an atomic counter that tracks how much data has been
safely written to disk:
- Writers update it after calling
Writer::sync - Readers check it to avoid reading uncommitted data
- It’s shared via
Arcfor efficient cloning across threads
This ensures readers never see partial writes or corrupted data.
§Performance Optimizations
§Read Hints
The ReadHint enum allows optimizing for different access patterns:
ReadHint::Sequential: Uses a 64KB read-ahead buffer for streaming accessReadHint::Random: Uses optimistic reads to reduce syscalls for small records
§Optimistic Reads
For random access, the reader performs an optimistic read of the header plus 2KB of data in a single syscall. If the record fits (most events in event sourcing do), this eliminates one syscall per read, improving performance by ~40% for small records.
§Examples
§Writing Records
use seglog::write::Writer;
let segment_size = 1024 * 1024; // 1 MB
// Writer with no header (H = 0)
let mut writer = Writer::<0>::create(&temp, segment_size, 0)?;
// Append records with empty header
let (offset, len) = writer.append(&[], b"event data")?;
writer.append(&[], b"more events")?;
// Sync to make data visible to readers
writer.sync()?;§Reading Records (Sequential)
use seglog::read::Reader;
let mut reader = Reader::<0>::open(&temp, Some(flushed))?;
// Iterate over all records
let mut iter = reader.iter(0);
while let Some(record) = iter.next_record()? {
println!("Record: {record:?}");
}§Reading Records (Random Access)
use seglog::read::{Reader, ReadHint};
let mut reader = Reader::<0>::open(&temp, Some(flushed))?;
// Read specific record by offset
let record = reader.read_record(offset, ReadHint::Random)?;
println!("Record: {record:?}");§Concurrent Readers
use seglog::read::Reader;
use std::thread;
let reader = Reader::<0>::open(&path, Some(flushed))?;
// Clone reader for use in another thread
let reader2 = reader.try_clone()?;
let handle = thread::spawn(move || {
// Read from the segment in a different thread
// Both readers share the same FlushedOffset
});§Segment Lifecycle
- Create: Use
Writer::<0>::createto initialize a new segment - Write: Append records with
Writer::append - Sync: Periodically call
Writer::syncto flush data to disk - Read: Open readers with
Reader::<0>::openusing the sharedFlushedOffset - Truncate (optional): Use
Writer::set_lento truncate the segment - Close: Call
Writer::closeorReader::closeto ensure cleanup
§Error Handling
The crate defines two main error types:
ReadError: CRC mismatch, out of bounds reads, I/O errorsWriteError: Segment full, I/O errors, read errors during recovery
Modules§
- control
- Control record kinds (first byte of a control record’s payload).
- parse
- read
- tail
- Following a segment that someone else is writing.
- write
Structs§
- Flushed
Offset - Thread-safe atomic offset tracking for flushed data.
Constants§
- COMMIT_
MARKER_ PAYLOAD - Payload size of a batch commit record: kind byte + u64.
- CONTROL_
FLAG - Control-record flag bit in the length field (bit 30). When set, the record is internal framing, not caller data.
- FLAG_
MASK - Known flag bits (currently just the control flag). Bit 31 is reserved and must be zero:
a record with it set is rejected as corrupt. Any bit outside
FLAG_MASK | LENGTH_MASKis corruption. - LENGTH_
MASK - Mask to extract actual length from the length field (bits 0..=29).
- MAX_
RECORD_ LEN - Largest encodable record length (1 GiB - 1).
- RECORD_
HEAD_ SIZE - Size of the record header in bytes, consisting of length and CRC-32 checksum.