Crate powercut

Expand description

§powercut

Prove your acknowledged write survives the power cut. In cargo test.

powercut is a database-agnostic crash-injection harness. It runs your workload to the point where your code says “committed”, captures the state that would actually survive a failure at that instant, reboots a fresh instance of your store from only that state, and lets you assert that what you acked came back.

It has zero dependencies, contains no unsafe, needs no root, no FUSE mount, no kernel module and no special runtime, and it works with FFI engines (SQLite, DuckDB, RocksDB) as happily as with pure-Rust ones.

§The 10-line quickstart

use powercut::{Checker, ScratchDir, Scenario, SyncJournal, Tier};

let store = ScratchDir::new("quickstart").unwrap();   // your store's directory
let journal = SyncJournal::complete();                // records every successful fsync
let db = store.path().join("app.log");

let report = Scenario::new(store.path(), journal.clone())
    .label("append-and-durable-ack")
    .tier(Tier::PowerCut)                             // only fsync'd bytes survive
    .ack(|| {                                         // ...your real workload...
        std::fs::write(&db, b"balance=100").unwrap();
        journal.sync_path(&db).unwrap();              // the fsync that earns durability
        "balance=100"
    })
    .unwrap()
    .check(|crash, acked| {                           // reboot from the surviving bytes
        let mut c = Checker::new();
        c.survives("balance", Some(acked.to_string()),
                   std::fs::read_to_string(crash.dir().join("app.log")).ok());
        c.finish()
    });

report.assert_survived();

Delete the journal.sync_path(&db) line and that test goes red — because the durable set is derived from the syncs that actually happened, never declared by the test author. That is the whole design: removing an fsync from your write path removes a file from the durable set, with no separate mutation run and no list to keep in step.

§The tier model

powercut models two failures, and being sloppy about which one you are measuring is the single most common mistake in hand-rolled durability tests.

tier	models	what survives	a store that never calls `fsync`
`Tier::ProcessCrash`	`SIGKILL`, a panic, an OOM kill, an OS crash	a byte-for-byte copy of the store at the ack — everything the app wrote to the filesystem	passes
`Tier::PowerCut`	the plug coming out	only bytes a successful `fsync` certified before the ack, truncated to the length each file had at its sync	fails

Tier 1 reads through the OS page cache. It can only ever prove “the write reached the file”. Tier 2 is the one that means “the write reached stable storage”. Tier::PowerCut is the default because tier 1 is the one people measure by accident.

At tier 2 powercut also captures a tier-1 copy at the same instant, so a failure report can say which of the two very different bugs you have:

asymmetry: the acked data IS present in the tier-1 byte copy taken at the
same instant. It reached the filesystem and no further — the missing thing
is an fsync, not a write.

§Four outcomes, not two

Survived      the durable set honoured the ack
Lost          it did not, and coverage was good enough for that to mean something
NotMeasured   the check ran but its result is not sound.  NOT A PASS.
HarnessBroken powercut itself is the suspect, so this run accuses nobody

NotMeasured is load-bearing. If your witness admits it cannot see all of the store’s sync activity (Coverage::Partial), then a failed check proves nothing — an unobserved fsync might have saved the data — so powercut reports NotMeasured and never a loss. A passed check under partial coverage is still sound, because the durable set was a subset of the truth. That asymmetry is what stops this crate from libelling somebody’s database.

HarnessBroken exists for the same reason. The dangerous failure of a durability test is not “red when it should be green”; it is red for a reason that is not durability — a store path that misses a sidecar, a copy that will not reopen, a checker looking in the wrong place. Acked::with_teeth_control re-runs the identical capture and checker against an explicitly flushed store. That run must pass; if it does not, the verdict is HarnessBroken, never Lost.

§The CI assertion is a ratchet, not a conformance check

powercut::assert_no_worse_than(&reports, "tests/durability-baseline.tsv");

Engines legitimately differ, and a suite that demands uniformity gets switched off within a month. What CI actually wants is “nothing started differing in a new way today”. Three rules: a verdict that got worse fails; a new kind of violation fails even at the same verdict; and a case that vanishes from the run — or newly becomes NotMeasured — fails, because coverage is not allowed to quietly evaporate. Record with POWERCUT_UPDATE_BASELINE=1 cargo test.

That variable is ignored when CI is set. Recording a baseline accepts whatever happened today as the new truth, so a stray POWERCUT_UPDATE_BASELINE in a CI environment would turn the ratchet into a rubber stamp and no case would ever go red again. Under CI the ratchet warns on stderr and asserts against the committed baseline instead. Record locally, read the diff, commit it — that review is the point of having a baseline at all.

§The durability primitive, and its one honest residual

powercut does its syncing through std::fs::File::sync_all and sync_data, which dispatch as follows:

platform	`sync_all` issues	notes
macOS / iOS (Apple targets)	`fcntl(fd, F_FULLFSYNC)`	a real device-cache flush, not `fsync(2)`
Linux, other Unix	`fsync(2)`; `sync_data` → `fdatasync(2)`
Windows	`FlushFileBuffers`	needs a write handle, which is why `sync_path` opens read-write

This is worth stating plainly because the folklore is wrong in a way that matters: on Apple targets Rust’s standard library has issued F_FULLFSYNC for sync_all for years (library/std/src/sys/fs/unix.rs, #[cfg(target_vendor = "apple")]). You do not need a libc dependency or an unsafe block to get a real flush on macOS, and a crate that tells you otherwise is out of date.

Two consequences powercut models explicitly:

sync_data is weaker than sync_all. fdatasync need not flush metadata, so a file’s length may not be durable. powercut records the kind, so a report tells you which one you relied on.
A file’s name is a separate durability question. On Unix a newly created file only durably exists after its parent directory is fsynced. SyncJournal::sync_dir does that and records it, and the capture prints a note for every synced file whose directory was never synced.

The residual, stated once and stated honestly: the guarantee bottoms out at the drive honouring the flush. F_FULLFSYNC asks the device to empty its own write cache; Apple guarantees this for the internal storage on its own machines, and nobody guarantees it for a third-party external enclosure, a USB bridge, or a virtualised block device that has been told to lie. If your disk lies about the flush, powercut believes the lie. No user-space tool can do better — detecting a lying drive needs a hardware power interrupt and an oscilloscope, not a test harness.

§Capturing a store on Windows

powercut copies the store’s bytes while the engine is still open — no graceful shutdown, no checkpoint. On Unix that is always possible. On Windows it depends on how the engine locks its files:

engine style	example	live capture on Windows
no locks, or locks a range beyond EOF	SQLite (its locks live at byte `0x40000000`)	works
exclusive lock over the whole file	redb (`LockFileEx` for as long as the DB is open)	not possible

A byte range held under an exclusive Windows lock cannot be read through any other handle, and no user-space trick gets around it — backup semantics bypass share modes, not byte-range locks.

When this happens powercut returns Error::Locked from the capture, naming the file. It does not skip the unreadable file and carry on: a store file silently missing from a capture would manufacture a false Lost, which is precisely the failure this crate exists to prevent. An honest error beats a confident wrong answer.

The practical consequence: on Windows, measure engines that do not hold an exclusive whole-file lock, or drive the engine so that no handle is open at the ack. examples/redb_backend.rs prints this and declines to run on Windows for exactly this reason.

§Non-goals

Stated up front so they can be linked rather than re-argued:

Not a filesystem crash-consistency tester. It does not hunt for bugs in ext4/APFS/NTFS. That is CrashMonkey + ACE (OSDI ’18); go there.
Not a block-layer fault injector. LazyFS (VLDB ’24, “When Amnesia Strikes”) is the serious prior art at that layer and it is genuinely stronger than powercut: a FUSE filesystem with its own page cache that can drop unsynced writes on command, integrated into Jepsen. It is also Linux-only, needs a mount and privileges, and is orchestrated from outside your test binary. powercut deliberately trades that power for running inside a plain cargo test on macOS, Linux and Windows.
Not a distributed-systems simulator (see turmoil, madsim) and not a concurrency checker (see shuttle, loom).
Does not claim to test the drive’s own write cache. See the residual above.
Does not, in this version, observe syncs an FFI engine makes internally. There is no LD_PRELOAD / DYLD_INSERT_LIBRARIES interposition backend yet. This is exactly what Coverage::Partial and Finding::NotMeasured are for, and the SQLite example shows how to get a fully measured result anyway.

§Testing an engine that syncs where you cannot see

If your engine does its own fsync through libc, powercut cannot observe it and must not pretend to. You have two honest options, and the shipped SQLite example demonstrates both:

Declare it. SyncJournal::partial("libsqlite3 syncs internally through libc"). Passes stay sound; failures come back as NotMeasured with that sentence attached.
Take the syncs over. Configure the engine to perform no syncs of its own (PRAGMA synchronous=OFF) and do them yourself at the ack with SyncJournal::sync_path. Coverage is then complete by construction, and what you have measured is the durable-ack pattern your application actually ships.

§Examples

cargo run --example generic_file_store   # a hand-rolled store, no engine deps
cargo run --example sqlite_durable_ack   # rusqlite: all four verdicts, one file
cargo run --example redb_backend         # redb, observed through its StorageBackend
cargo run --example read_fork_close_tear # the flagship, as a matrix across engines

§The flagship: the read-fork close-tear

Open a second connection to your store. Close it. Now the data you already acked — and whose bytes were genuinely fsync’d — can stop being durable, while every subsequent COMMIT still returns Ok. Nothing errors. Nothing logs. A checkpoint or compaction triggered by that second handle’s lifecycle moves your committed rows out of the file that was synced and into one that was not.

cargo run --example read_fork_close_tear runs that as a matrix across the shipped engines and prints what each one actually does. powercut catches it because it captures the bytes each fsync actually certified: the file that carried the warrant has shrunk or vanished, the file the data moved into was rewritten in place and never re-synced, and neither one can honour the ack.

Here it is end to end, on a store simple enough to fit in a doctest:

use powercut::{Checker, ScratchDir, Scenario, SyncJournal, Tier, Verdict};

let store = ScratchDir::new("read-fork").unwrap();
let (dir, journal) = (store.path().to_path_buf(), SyncJournal::complete());
let log = dir.join("store.log");
let read_balance = move |d: &std::path::Path| -> Option<String> {
    ["store.base", "store.log"].iter().find_map(|n| {
        let text = std::fs::read_to_string(d.join(n)).ok()?;
        Some(text.lines().rev().find_map(|l| l.strip_prefix("balance="))?.to_string())
    })
};
let check = |crash: &powercut::CrashState, acked: &String| {
    let mut c = Checker::new();
    c.survives("balance", Some(acked.clone()), read_balance(crash.dir()));
    c.finish()
};

// 1. Commit and make it genuinely durable.
let file = std::fs::File::create(&log).unwrap();
std::io::Write::write_all(&mut (&file), b"balance=100\n").unwrap();
journal.sync_all(&file, &log).unwrap();
journal.sync_dir(&dir).unwrap();

let before = Scenario::new(&dir, journal.clone())
    .label("before").tier(Tier::PowerCut)
    .ack(|| "100".to_string()).unwrap().check(check);
assert_eq!(before.verdict(), Verdict::Survived);   // it really was durable

// 2. A reader opens, reads, and closes. Its close folds the log into a base
//    file and unlinks the log. The unlink is made durable; the base is not.
let after = Scenario::new(&dir, journal.clone())
    .label("after").tier(Tier::PowerCut)
    .ack(|| {
        let held = std::fs::read_to_string(&log).unwrap();
        std::fs::write(dir.join("store.base"), held).unwrap();  // never fsynced
        std::fs::remove_file(&log).unwrap();
        journal.sync_dir(&dir).unwrap();                        // the unlink IS durable
        "100".to_string()
    })
    .unwrap().check(check);

assert_eq!(after.verdict(), Verdict::Lost);        // nobody wrote. nobody errored.
assert!(after.to_text().contains("GONE from the store"));

§Contributing — an open, assertive posture

Bring the argument. This crate is an oracle: it tells people their database lost data, and it will sometimes be wrong. Every genuine argument, counterexample, engine report and feature request is welcome, and a well-argued improvement will be accepted — including one that says a core design decision here is wrong. “I think your tier model mismodels X” is the most useful issue you can file, not an imposition.

Concretely: file an issue with a runnable reproduction and we will read your storage engine. Send a PR without asking first. Disagree in public. The point of this crate is to be right about durability, and reach and correctness both come from other people pushing on it.

See CONTRIBUTING.md. Machine-readable entry points for coding agents: llms.txt and AGENTS.md.

Every public item carries its own compiling doctest — 113 of them across 109 public items, run in CI on Linux, macOS and Windows. That is a hard rule rather than a style preference: the docs are the adoption channel, coding agents read them, and an example that does not compile is worse than no example. If one on docs.rs fails to compile, that is a bug worth an issue.

§MSRV

Rust 1.82. Raising it is a minor-version change, and CI tests it.

§License

Dual-licensed under MIT or Apache-2.0, at your option.

Structs§

Acked: A workload that has acked, with the crash state already captured.
AssumedDurable: A path the harness is told to treat as durable without having observed a sync for it.
Barrier: A monotonic ordering point, handed out by a SyncJournal on every successful sync.
Baseline: A recorded set of expected outcomes, one per case label.
BaselineEntry: One recorded case.
CapturedFile: One file in a captured crash state.
Checker: A small helper for writing checkers that put the baseline check first.
CrashState: A materialised directory holding exactly the state that survives at a chosen Tier, plus the manifest explaining every byte of it.
Report: The result of one case: one store, one tier, one checker.
Scenario: One measurement: run a workload to an ack point, capture what survives at a Tier, and hand the captured directory to your checker.
ScratchDir: A uniquely named directory under the system temp dir, removed on drop.
SyncJournal: The cooperative witness: a cheap, cloneable, thread-safe journal of successful syncs.
SyncRecord: One successful sync: this path reached stable storage, at this length, at this point in the ordering.
Violation: One way the acked promise failed to survive.

Enums§

Coverage: How much of the store’s sync activity this witness can actually see.
Error: Everything that can go wrong inside the harness itself.
Finding: The outcome with its evidence attached.
SyncKind: Which flavour of sync was performed.
Tier: Which failure powercut is modelling.
Verdict: The four-valued outcome. Two of these are not “pass” and not “fail”.
ViolationKind: What kind of thing went wrong, so a ratchet can tell new breakage from breakage it already knows about.
Warrant: Why a file earned its place in a captured crash state.

Constants§

UPDATE_ENV: The environment variable that rewrites a baseline instead of asserting against it: POWERCUT_UPDATE_BASELINE=1 cargo test.

Traits§

DurabilityWitness: Something that can tell powercut which bytes reached stable storage.
IsAbsent: Lets Checker::survives tell “absent” apart from “different”.

Functions§

assert_all_survived: Print every report to stdout, then panic if any of them is not Verdict::Survived.
assert_no_worse_than: The CI assertion: today is no worse than the recorded baseline.
compare_to_baseline: Compare reports against the baseline recorded at path and return every way today is worse than the recorded day.
to_markdown: Render a set of reports as a markdown table — the matrix you paste into a README or a pull request.

Type Aliases§

Result: The crate’s result type.

Crate powercut

Crate powercut Copy item path

§powercut

§The 10-line quickstart

§The tier model

§Four outcomes, not two

§The CI assertion is a ratchet, not a conformance check

§The durability primitive, and its one honest residual

§Capturing a store on Windows

§Non-goals

§Testing an engine that syncs where you cannot see

§Examples

§The flagship: the read-fork close-tear

§Contributing — an open, assertive posture

§MSRV

§License

Structs§

Enums§

Constants§

Traits§

Functions§

Type Aliases§

Crate powercut