# WriteAhead
[](https://crates.io/crates/writeahead)
[](https://docs.rs/writeahead)
[](https://github.com/danthegoodman1/WriteAhead/actions/workflows/ci.yml)
A partitioned WAL crate for building append-only, high-throughput durability systems.
Requires Rust 1.89 or later (uses standard-library file locking on Unix).
A great component for a (distributed) data store. For distributed usage, consider FDB's model (you may want a custom in-memory index in front for custom ID→offset mappings).
## Guarantees
1. **Durability** — a write is acknowledged only after it is fdatasync'd to the active log file.
2. **Corruption detection** — every record carries a 128-bit murmur3 hash of its data; any bit-flip in a record (including its length header, caught via bounds check + hash mismatch) is detected on read.
3. **Crash recovery** — `start()` restores a consistent WAL after a crash at any point: every file is validated, torn tails are truncated, files that missed their seal mid-rotation are healed and sealed, and sealed files are never appended to. Every acknowledged record survives.
4. **Stable addressing** — `RecordID { file_id, file_offset }` stays valid for the life of the file (until retention deletes it).
5. **Sequential streaming** — a stream replays all records across files in write order, yielding each record with its `RecordID` so consumers can checkpoint their position.
## Quickstart
```toml
[dependencies]
writeahead = "0.3"
anyhow = "1"
futures = "0.3"
tokio = { version = "1", features = ["rt", "macros"] }
```
(Or track main directly: `writeahead = { git = "https://github.com/danthegoodman1/WriteAhead" }`.)
The full program below lives at [`examples/quickstart.rs`](examples/quickstart.rs) and runs with `cargo run --example quickstart`:
```rust
use futures::StreamExt;
use std::sync::Arc;
use writeahead::{SimpleFile, WriteAhead, WriteAheadOptions};
#[tokio::main(flavor = "current_thread")]
async fn main() -> anyhow::Result<()> {
let mut wal = WriteAhead::<SimpleFile>::with_options(WriteAheadOptions {
log_dir: "./write_ahead".into(),
..Default::default()
});
wal.start()?; // recovers existing state; if this errors, crash
let wal = Arc::new(wal); // WriteAhead is Sync: share it directly
// Durable writes from any task or thread: clone a WriteHandle per
// producer. Writes that arrive while an fsync is in flight coalesce
// into one group commit automatically.
let handle = wal.writer()?;
let id = handle.write(b"hello".to_vec()).await?;
let more = handle
.write_batch(vec![b"a".to_vec(), b"b".to_vec()])
.await?;
println!("wrote {:?} and {:?}", id, more);
// Point read by address
let record = wal.read(id.file_id, id.file_offset)?;
assert_eq!(record, b"hello");
// Full replay, oldest first; each record comes with its address so
// consumers can checkpoint how far they've gotten
let mut stream = wal.create_stream()?;
while let Some(entry) = stream.next().await {
let (id, record) = entry?;
println!("replayed {} bytes from {:?}", record.len(), id);
}
Ok(())
}
```
Retention is opt-in via `RetentionOptions`: `max_total_size` deletes the oldest sealed files once total disk usage exceeds the bound, and `ttl` deletes sealed files whose seal timestamp is older than the window. The active file is never deleted; reads into deleted files return `LogfileNotFound`. For consumer-driven cleanup, use `trim_before` instead — see the patterns below.
`start()` exclusively locks the log directory before recovery. Repeated starts
return `AlreadyStarted`; another manager or process using the same directory
gets `DirectoryLocked`. WriteHandles keep ownership after the manager is dropped.
Dropping the last handle waits for accepted commands and the writer thread to
finish, then releases the lock so the directory can be reopened.
Submission uses a bounded async queue: `queue_capacity` defaults to 64 commands,
and `max_batch_bytes` defaults to 4 MiB of encoded records (payloads plus 20 bytes
per record). A full queue suspends submission without blocking the executor;
larger batches return `BatchTooLarge`. Both options must be positive. Existing
complete `WriteAheadOptions` struct literals need these fields or
`..Default::default()`.
Queued encoded bytes are bounded by `queue_capacity * max_batch_bytes`.
A drain round takes at most 64 commands and stops after reaching 4 MiB; its last
batch may cross that target, so its encoded size is less than
`4 MiB + max_batch_bytes`. The in-flight round holds the original payloads and
an encoded copy, plus record/command bookkeeping. Caller-owned futures waiting
for admission and spare Vec capacity are outside these encoded-byte bounds.
Cancelling a request before admission prevents its write; after admission,
cancellation or an error does not imply the record is absent after recovery.
Active files are zero-filled ahead of writes in 256 KiB allocation windows by
default, so commits overwrite blocks that are already allocated and written. On
ext4 and XFS, an fdatasync that allocates blocks must also commit the filesystem
journal; overwriting written blocks skips that commit, which roughly halves
commit latency (see [PERFORMANCE.md](PERFORMANCE.md#allocation-windows)). The
commit that crosses into a new window also writes its zeros, so larger windows
mean rarer but longer stalls. Tune the window with
`WriteAheadOptions::preallocation_chunk_size: Option<u64>`. Set it to `None` on
copy-on-write filesystems (btrfs, ZFS, bcachefs, APFS): they allocate new
blocks on every write, so the zeros only add write traffic. Windows are clamped
to `max_file_size`, and sealed files are truncated to their record end before
their footer is written, so only the active file carries an allocation tail.
`max_file_size` is a soft limit on the logical record region. The writer
rotates before a queued commit group would cross it, keeping every group and
submitted batch contiguous in one file. If one group is larger than an empty
file's capacity, it is written intact to one oversized file and that file is
rotated immediately afterward. The limit includes the 12 KiB version 3 header;
a smaller configured limit is valid, but every non-empty group is consequently
treated as oversized.
## Patterns
The WAL never decides on its own that a record is *done* — that knowledge lives with whatever consumes the log. `trim_before(file_id)` is the primitive that lets the consumer feed that knowledge back: it deletes every sealed file strictly below `file_id` (the active file is never touched) and returns how many files and bytes were reclaimed. Two common shapes, both runnable from `examples/`:
### Trim by consumer high-water mark
`cargo run --example trim_hwm` ([examples/trim_hwm.rs](examples/trim_hwm.rs))
A replaying consumer tracks the `RecordID` of the last record it has applied — its high-water mark. Records are yielded in order, so the moment the mark crosses into file `N`, every file below `N` is fully consumed and can be dropped:
```rust
let mut hwm: Option<RecordID> = None;
let mut stream = wal.create_stream()?;
while let Some(entry) = stream.next().await {
let (id, record) = entry?;
apply(&record); // your state machine / downstream sink
if let Some(prev) = hwm {
if id.file_id > prev.file_id {
// Everything below the file we just entered is consumed
handle.trim_before(id.file_id).await?;
}
}
hwm = Some(id);
}
```
Checkpoint the mark durably (wherever your consumer state lives), and a restart resumes with `create_stream_from(hwm.file_id, hwm.file_offset)` instead of replaying from the beginning. Trim only *after* the checkpoint is durable — the trim is what makes the pre-mark log unrecoverable. Streams created before a trim are undisturbed (they hold their own open file handles); new reads into trimmed files return `LogfileNotFound`.
Resume is inclusive: skip the checkpointed record if it was already applied.
Trim evicts idle cached readers immediately. `TrimStats::bytes_reclaimed` counts
the lengths of unlinked files; their storage remains occupied while an existing
stream holds them open. Streams release exhausted files as replay advances.
### Explicit compaction
`cargo run --example compaction` ([examples/compaction.rs](examples/compaction.rs))
The db-traditional shape, for when the log is the backing store of keyed state (an index maps each key to the `RecordID` of its latest value). Old files can't be trimmed by a high-water mark because a handful of cold, still-live records pin them while dead versions pile up around. Compaction rewrites the live records forward, then drops everything behind:
1. Pick a cutoff file id (e.g. the newest file the index points into).
2. Read every live record below the cutoff and `write_batch` the copies — they land in the active file, and the index is repointed at the new `RecordID`s.
3. `trim_before(cutoff)` — nothing below it is referenced anymore.
The rewrite goes through the normal durable write path, so a crash at any point is safe: before the trim, both copies exist and the index (rebuilt by replay or checkpointed) still resolves; the trim only happens after the new copies are acknowledged.
## Design
### Single writer, page-cache readers
One dedicated writer thread owns the active file and the directory lifecycle: writes, rotation, and retention all happen there. It publishes durable end offsets to the reader cache under a short shared lock after sync succeeds and before replying. Any number of tasks submit through cloned `WriteHandle`s and await their acks without blocking the async executor. Writes queued while an fsync is in flight are **group-committed**: the writer drains them into a single record write + fdatasync and fans replies back out, so concurrent producers coalesce automatically (see `handle_concurrent_8w` in the benchmarks). Each ack names the file the records landed in, so handles stay valid across rotations.
Readers use positional reads (`pread`) straight from the page cache — for WAL access patterns readers mostly want recently written data, which the page cache serves far better than direct-IO schemes like io_uring + O_DIRECT. Point reads speculatively fetch one small block so most records cost a single syscall; streams parse records out of 128 KiB readahead chunks.
### File format (version 3)
```text
| 4 KiB format block | 4 KiB slot 0 block | 4 KiB slot 1 block | records ... | 40B seal footer (sealed files only) |
format block: | 8B magic | 1B version | zero padding |
committed-end slot: | 8B generation | 8B records-end offset | 16B murmur3_128 of the previous 16B |
record: | 16B murmur3_128(data) | 4B bitwise-inverted length (u32 LE) | data |
footer: | 8B magic | 8B seal timestamp (unix ms) | 8B records-end offset | 16B murmur3_128 of the previous 24B |
```
The two committed-end slots alternate by generation and occupy distinct aligned
filesystem blocks. Under the torn-write model, a damaged newest-slot write
cannot damage the older slot's block; this does not claim recovery from
arbitrary corruption spanning multiple blocks. A commit writes its record bytes
and the alternate checksummed slot, then covers both with the same single
`fdatasync` before acknowledging writers. Live manager readers use the end
published after successful sync; recovery reads the slots from disk. Both use
the logical end instead
of the active file's physical length, so the zero-filled allocation tail cannot
be replayed as empty records. Inverting the stored length also makes an
unwritten all-zero record header invalid while preserving support for real
empty payloads. Data is stored raw — no escaping. A file is sealed iff its
trailing 40 bytes parse as a footer whose own hash verifies and whose records-end
field equals the footer's position, so record payloads (which may contain the
magic bytes) cannot be mistaken for a seal.
Version 3 is a format break: older version 2 files are rejected as unsupported
rather than upgraded in place.
The log directory is a sequence of `0000000000.log`, `0000000001.log`, … files. The highest id is the active file; rotation creates the next file first, then seals the previous one (the footer carries the seal timestamp used by ttl retention), so a crash mid-rotation always lands in a state recovery heals. Directory entries are fsync'd on file creation.
Startup syncs ancestor directories and repeats the WAL-directory barrier even
when its header already exists, covering retries after interrupted creation.
### Recovery
On `start()`:
- Non-active files that aren't sealed (crash mid-rotation, torn footer) are healed: valid record prefix kept, garbage truncated, footer written.
- The active file scans only through the newest valid committed-end slot, never through its allocation tail. A torn newest commit is shortened to its valid record prefix, but never below the end protected by the older valid slot.
- If the active file turns out to be sealed (crash between seal and next-file creation), a fresh file is started rather than appending to it.
- Non-log files in the directory are skipped.
- Underlying read errors abort recovery without repairing slots or truncating
that file. Buffered recovery verifies payload slices without allocating a
separate payload per record.
## Benchmarks
`cargo run --release --example bench` (writes to the current directory — don't run it on tmpfs, where fsync is free and write numbers become fiction).
The harness also times active-file recovery. Current before/after measurements,
including storage details and raw results, are in [PERFORMANCE.md](PERFORMANCE.md).
The desktop measurements below are historical and predate bounded admission;
the new 64-command drain limit bounds group size at high concurrency.
Snapshot from a desktop NVMe/ext4 box, 64-byte payloads:
| scenario | what it measures | rate |
| --- | --- | --- |
| `seq_write_1rec` | one record per `write_batch`, strict ack-then-next: the floor set by one fdatasync per write | ~230/s (~4.4ms/op) |
| `batch_write` | 1,000-record batches through `write_batch`: one pwrite + one fdatasync per batch | ~225k records/s |
| `read_by_id` | point reads of individual records by `RecordID`, served from the page cache | ~2.1M/s |
| `stream_replay` | full sequential replay of the log through `create_stream()` | ~34M/s (~2.1GB/s) |
| `handle_concurrent_8w` | 8 threads each writing through cloned public `WriteHandle`s and awaiting their acks; queued writes share fsyncs via group commit | ~940/s |
Durable-write rates are dominated by fdatasync latency, so they scale with batch/group size, not payload size — compare `seq_write_1rec` (1 record per sync) against `batch_write` (1,000 per sync) and `handle_concurrent_8w` (however many are queued per sync).
### Throughput scales with in-flight writes
`handle_concurrent_8w` is not a ceiling — it's the throughput of exactly 8 in-flight writers. Group commit converts *concurrency* into batch size: every write queued while an fsync is in flight rides the next one, so single-record `handle.write()` calls (each individually fsync'd before its ack) scale with the number of concurrent writers. Measured on the same box:
| concurrent writers | durable writes/s |
| --- | --- |
| 8 | ~950 |
| 64 | ~7.2k |
| 512 | ~79k |
| 2,048 | ~290k |
Per-write latency stays at flush scale (a few ms) throughout — concurrency buys throughput, never latency. If your data already arrives in batches, `write_batch` is the same lever with bigger settings: ~760k records/s at 4,096-record batches and ~2M records/s at 16,384 on this disk, every record fsync'd before ack.
## Integrating with a thread-safe API framework (Axum, tonic, etc.)
Put the started `WriteAhead` in an `Arc` and share it — it's `Sync`. Clone a `WriteHandle` into each request handler (or once into your app state) and call `handle.write(...)` directly; there is no owner task to build and no upstream batching needed, because writes that arrive while a commit is in flight coalesce into the next group commit automatically.
`WriteAheadStream` is `Send` and owns its file handles, so you can create a stream and hand it to a response body directly (see `test_write_ahead_stream_is_send`). Use a bounded channel if you bridge it manually so you don't bloat memory.
Streams capture both file membership and successfully synced end offsets at
creation. Later appends do not extend a stream, and an in-flight sync cannot
expose tentative records through the manager's read/stream APIs. These APIs
perform synchronous page-cache reads; implementing `Stream` does not offload
cold disk I/O from an async executor. Direct low-level `Logfile::open` access
reads on-disk metadata and does not share the manager's publication boundary.