durability

Crash-consistent persistence primitives: directory abstraction, generic WAL, checkpoints, and recovery.

Quick start

[dependencies]
durability = "0.6"

use durability::storage::MemoryDirectory;
use durability::walog::{WalWriter, WalReader, WalEntry};

let dir = MemoryDirectory::arc();

// open() creates a fresh WAL or resumes an existing one.
let mut w = WalWriter::<WalEntry>::open(dir.clone()).unwrap();
w.append(&WalEntry::AddSegment { segment_id: 1, doc_count: 100 }).unwrap();
w.append(&WalEntry::DeleteDocuments { deletes: vec![(1, 42)] }).unwrap();
w.flush().unwrap();

assert_eq!(w.last_entry_id(), Some(2));
drop(w);

// Recover
let records = WalReader::<WalEntry>::new(dir).replay().unwrap();
assert_eq!(records.len(), 2);
assert_eq!(records[0].entry_id, 1);

WalWriter<E> and WalReader<E> are generic -- define your own entry type with #[derive(Serialize, Deserialize)] and use WalWriter::<YourType>::open(dir).

Thread-safe writer

SyncWalWriter wraps a WalWriter in a Mutex for concurrent access. append_durable provides group commit semantics -- concurrent callers share a single fsync:

use durability::storage::FsDirectory;
use durability::walog::SyncWalWriter;
use std::sync::Arc;

# #[derive(serde::Serialize, serde::Deserialize)] enum Op { Set(String) }
let dir = FsDirectory::arc("/tmp/wal-demo").unwrap();
let sw = Arc::new(SyncWalWriter::<Op>::open(dir).unwrap());

// Each thread calls append_durable; fsync is batched across callers.
let sw2 = sw.clone();
std::thread::spawn(move || {
    sw2.append_durable(&Op::Set("hello".into())).unwrap();
});
sw.append_durable(&Op::Set("world".into())).unwrap();

Generic recovery

recover_with_wal coordinates checkpoint loading and WAL replay for any entry type and checkpoint schema:

use durability::recover::{recover_with_wal, RecoveryOptions};
use durability::walog::WalWriter;
use durability::storage::MemoryDirectory;

#[derive(Default, serde::Serialize, serde::Deserialize)]
struct Snap { counter: u64 }

#[derive(serde::Serialize, serde::Deserialize)]
enum Op { Inc, Dec }

let dir = MemoryDirectory::arc();
let mut w = WalWriter::<Op>::new(dir.clone());
w.append(&Op::Inc).unwrap();
w.append(&Op::Inc).unwrap();
w.append(&Op::Dec).unwrap();
w.flush().unwrap();
drop(w);

let result = recover_with_wal::<Snap, Op, _>(
    &dir, None, RecoveryOptions::strict(),
    |ckpt| ckpt.unwrap_or_default().counter,
    |counter, _id, entry| match entry {
        Op::Inc => *counter += 1,
        Op::Dec => *counter = counter.saturating_sub(1),
    },
).unwrap();
assert_eq!(result.state, 1); // 0 + 1 + 1 - 1

Tuning

use durability::storage::{FsDirectory, FlushPolicy};
use durability::walog::WalWriter;
# #[derive(serde::Serialize, serde::Deserialize)] enum Op { X }

let dir = FsDirectory::arc("/tmp/wal-tuning").unwrap();
let mut w = WalWriter::<Op>::with_options(
    dir, FlushPolicy::Interval(std::time::Duration::from_millis(10)), 64 * 1024,
);
w.set_segment_size_limit_bytes(64 * 1024 * 1024); // 64 MiB segments
w.set_segment_max_age(std::time::Duration::from_secs(300)); // rotate after 5 min
w.set_preallocate_bytes(4 * 1024 * 1024); // 4 MiB preallocation
w.set_recycle_capacity(4); // reuse up to 4 truncated segments

Async support

The async feature provides AsyncDirectory and BlockingBridge for tokio:

[dependencies]
durability = { version = "0.6", features = ["async"] }

use durability::async_dir::{AsyncDirectory, BlockingBridge};
use durability::storage::FsDirectory;

# async fn example() {
let bridge = BlockingBridge::new(FsDirectory::new("/tmp/async-demo").unwrap());
bridge.atomic_write("data.bin", b"hello".to_vec()).await.unwrap();
let data = bridge.read_file("data.bin").await.unwrap();
# }

Modules

Not provided

• Multi-process locking: single-writer-per-directory assumed. Advisory lockfile catches in-process double-instantiation only.
• Strong consistency by default: writes are buffered. Use flush_and_sync() for a durability barrier.
• fsync failure recovery: a failed fsync poisons the writer. Callers should treat this as unrecoverable and restart from WAL.

Contract

• Prefix property: best-effort replay returns a prefix of the valid stream.
• Narrow best-effort: tolerance applies only to the final segment's torn tail. Corruption in non-final segments is an error.
• Deterministic checkpoints: payloads are written with stable ordering.

Running

• Tests: cargo test
• Property tests: PROPTEST_CASES=512 cargo test --test prop_wal_resume
• Benches: cargo bench
• Fuzzing: cargo fuzz run fuzz_wal_entry_decode (see fuzz/)