WriteAhead

A partitioned WAL crate for building append-only, high-throughput durability systems.

A great component for a (distributed) data store. For distributed usage, consider FDB's model (you may want a custom in-memory index in front for custom ID→offset mappings).

Guarantees

1. Durability — a write is acknowledged only after it is fdatasync'd to the active log file.
2. Corruption detection — every record carries a 128-bit murmur3 hash of its data; any bit-flip in a record (including its length header, caught via bounds check + hash mismatch) is detected on read.
3. Crash recovery — start() restores a consistent WAL after a crash at any point: every file is validated, torn tails are truncated, files that missed their seal mid-rotation are healed and sealed, and sealed files are never appended to. Every acknowledged record survives.
4. Stable addressing — RecordID { file_id, file_offset } stays valid for the life of the file (until retention deletes it).
5. Sequential streaming — a stream replays all records across files in write order, yielding each record with its RecordID so consumers can checkpoint their position.

Quickstart

[dependencies]
writeahead = "0.2"
anyhow = "1"
futures = "0.3"
tokio = { version = "1", features = ["rt", "macros"] }

(Or track main directly: writeahead = { git = "https://github.com/danthegoodman1/WriteAhead" }.)

The full program below lives at examples/quickstart.rs and runs with cargo run --example quickstart:

use futures::StreamExt;
use std::sync::Arc;
use writeahead::{SimpleFile, WriteAhead, WriteAheadOptions};

#[tokio::main(flavor = "current_thread")]
async fn main() -> anyhow::Result<()> {
    let mut wal = WriteAhead::<SimpleFile>::with_options(WriteAheadOptions {
        log_dir: "./write_ahead".into(),
        ..Default::default()
    });
    wal.start()?; // recovers existing state; if this errors, crash
    let wal = Arc::new(wal); // WriteAhead is Sync: share it directly

    // Durable writes from any task or thread: clone a WriteHandle per
    // producer. Writes that arrive while an fsync is in flight coalesce
    // into one group commit automatically.
    let handle = wal.writer()?;
    let id = handle.write(b"hello".to_vec()).await?;
    let more = handle
        .write_batch(vec![b"a".to_vec(), b"b".to_vec()])
        .await?;
    println!("wrote {:?} and {:?}", id, more);

    // Point read by address
    let record = wal.read(id.file_id, id.file_offset)?;
    assert_eq!(record, b"hello");

    // Full replay, oldest first; each record comes with its address so
    // consumers can checkpoint how far they've gotten
    let mut stream = wal.create_stream()?;
    while let Some(entry) = stream.next().await {
        let (id, record) = entry?;
        println!("replayed {} bytes from {:?}", record.len(), id);
    }

    Ok(())
}

Retention is opt-in via RetentionOptions: max_total_size deletes the oldest sealed files once total disk usage exceeds the bound, and ttl deletes sealed files whose seal timestamp is older than the window. The active file is never deleted; reads into deleted files return LogfileNotFound. For consumer-driven cleanup, use trim_before instead — see the patterns below.

Active files are sparsely grown ahead of writes in 64 MiB allocation windows by default. This avoids making every durable append persist a new file length. Tune the window with WriteAheadOptions::preallocation_chunk_size: Option<u64>; set it to None to disable preallocation. Windows are clamped to max_file_size, and sealed files are truncated to their actual record end before their footer is written, so the sparse tail is only present on the active file. The physical file length shown by tools such as ls includes that sparse tail; allocated disk blocks remain small until records fill it.

max_file_size is a soft limit on the logical record region. The writer rotates before a queued commit group would cross it, keeping every group and submitted batch contiguous in one file. If one group is larger than an empty file's capacity, it is written intact to one oversized file and that file is rotated immediately afterward. The limit includes the 12 KiB version 3 header; a smaller configured limit is valid, but every non-empty group is consequently treated as oversized.

Patterns

The WAL never decides on its own that a record is done — that knowledge lives with whatever consumes the log. trim_before(file_id) is the primitive that lets the consumer feed that knowledge back: it deletes every sealed file strictly below file_id (the active file is never touched) and returns how many files and bytes were reclaimed. Two common shapes, both runnable from examples/:

Trim by consumer high-water mark

cargo run --example trim_hwm (examples/trim_hwm.rs)

A replaying consumer tracks the RecordID of the last record it has applied — its high-water mark. Records are yielded in order, so the moment the mark crosses into file N, every file below N is fully consumed and can be dropped:

let mut hwm: Option<RecordID> = None;
let mut stream = wal.create_stream()?;
while let Some(entry) = stream.next().await {
    let (id, record) = entry?;
    apply(&record); // your state machine / downstream sink

    if let Some(prev) = hwm {
        if id.file_id > prev.file_id {
            // Everything below the file we just entered is consumed
            handle.trim_before(id.file_id).await?;
        }
    }
    hwm = Some(id);
}

Checkpoint the mark durably (wherever your consumer state lives), and a restart resumes with create_stream_from(hwm.file_id, hwm.file_offset) instead of replaying from the beginning. Trim only after the checkpoint is durable — the trim is what makes the pre-mark log unrecoverable. Streams created before a trim are undisturbed (they hold their own open file handles); new reads into trimmed files return LogfileNotFound.

Explicit compaction

cargo run --example compaction (examples/compaction.rs)

The db-traditional shape, for when the log is the backing store of keyed state (an index maps each key to the RecordID of its latest value). Old files can't be trimmed by a high-water mark because a handful of cold, still-live records pin them while dead versions pile up around. Compaction rewrites the live records forward, then drops everything behind:

1. Pick a cutoff file id (e.g. the newest file the index points into).
2. Read every live record below the cutoff and write_batch the copies — they land in the active file, and the index is repointed at the new RecordIDs.
3. trim_before(cutoff) — nothing below it is referenced anymore.

The rewrite goes through the normal durable write path, so a crash at any point is safe: before the trim, both copies exist and the index (rebuilt by replay or checkpointed) still resolves; the trim only happens after the new copies are acknowledged.

Design

Single writer, page-cache readers

One dedicated writer thread owns the active file and the directory lifecycle: writes, rotation, and retention all happen there, so nothing ever contends on the write path. Any number of tasks submit through cloned WriteHandles and await their acks without blocking the async executor. Writes queued while an fsync is in flight are group-committed: the writer drains them into a single pwrite + fdatasync and fans replies back out, so concurrent producers coalesce automatically (see handle_concurrent_8w in the benchmarks). Each ack names the file the records landed in, so handles stay valid across rotations.

Readers use positional reads (pread) straight from the page cache — for WAL access patterns readers mostly want recently written data, which the page cache serves far better than direct-IO schemes like io_uring + O_DIRECT. Point reads speculatively fetch one small block so most records cost a single syscall; streams parse records out of 128 KiB readahead chunks.

File format (version 3)

| 4 KiB format block | 4 KiB slot 0 block | 4 KiB slot 1 block | records ... | 40B seal footer (sealed files only) |

format block: | 8B magic | 1B version | zero padding |
committed-end slot: | 8B generation | 8B records-end offset | 16B murmur3_128 of the previous 16B |
record: | 16B murmur3_128(data) | 4B bitwise-inverted length (u32 LE) | data |
footer: | 8B magic | 8B seal timestamp (unix ms) | 8B records-end offset | 16B murmur3_128 of the previous 24B |

The two committed-end slots alternate by generation and occupy distinct aligned filesystem blocks. Under the torn-write model, a damaged newest-slot write cannot damage the older slot's block; this does not claim recovery from arbitrary corruption spanning multiple blocks. A commit writes its record bytes and the alternate checksummed slot, then covers both with the same single fdatasync before acknowledging writers. Readers use this logical end instead of the active file's sparse physical length, so unused zero-filled space cannot be replayed as empty records. Inverting the stored length also makes an unwritten all-zero record header invalid while preserving support for real empty payloads. Data is stored raw — no escaping. A file is sealed iff its trailing 40 bytes parse as a footer whose own hash verifies and whose records-end field equals the footer's position, so record payloads (which may contain the magic bytes) cannot be mistaken for a seal.

Version 3 is a format break: older version 2 files are rejected as unsupported rather than upgraded in place.

The log directory is a sequence of 0000000000.log, 0000000001.log, … files. The highest id is the active file; rotation creates the next file first, then seals the previous one (the footer carries the seal timestamp used by ttl retention), so a crash mid-rotation always lands in a state recovery heals. Directory entries are fsync'd on file creation.

Recovery

On start():

• Non-active files that aren't sealed (crash mid-rotation, torn footer) are healed: valid record prefix kept, garbage truncated, footer written.
• The active file scans only through the newest valid committed-end slot, never through its sparse allocation tail. A torn newest commit is shortened to its valid record prefix, but never below the end protected by the older valid slot.
• If the active file turns out to be sealed (crash between seal and next-file creation), a fresh file is started rather than appending to it.
• Non-log files in the directory are skipped.

Benchmarks

cargo run --release --example bench (writes to the current directory — don't run it on tmpfs, where fsync is free and write numbers become fiction).

Snapshot from a desktop NVMe/ext4 box, 64-byte payloads:

Durable-write rates are dominated by fdatasync latency, so they scale with batch/group size, not payload size — compare seq_write_1rec (1 record per sync) against batch_write (1,000 per sync) and handle_concurrent_8w (however many are queued per sync).

Throughput scales with in-flight writes

handle_concurrent_8w is not a ceiling — it's the throughput of exactly 8 in-flight writers. Group commit converts concurrency into batch size: every write queued while an fsync is in flight rides the next one, so single-record handle.write() calls (each individually fsync'd before its ack) scale with the number of concurrent writers. Measured on the same box:

Per-write latency stays at flush scale (a few ms) throughout — concurrency buys throughput, never latency. If your data already arrives in batches, write_batch is the same lever with bigger settings: ~760k records/s at 4,096-record batches and ~2M records/s at 16,384 on this disk, every record fsync'd before ack.

Integrating with a thread-safe API framework (Axum, tonic, etc.)

Put the started WriteAhead in an Arc and share it — it's Sync. Clone a WriteHandle into each request handler (or once into your app state) and call handle.write(...) directly; there is no owner task to build and no upstream batching needed, because writes that arrive while a commit is in flight coalesce into the next group commit automatically.

WriteAheadStream is Send and owns its file handles, so you can create a stream and hand it to a response body directly (see test_write_ahead_stream_is_send). Use a bounded channel if you bridge it manually so you don't bloat memory.

Note: streams snapshot the set of log files at creation time and replay records acknowledged before that point; readers tailing the unsealed active file are best-effort and may surface transient errors if they race an in-flight write.