# powercut
**Prove your acknowledged write survives the power cut. In `cargo test`.**
[](https://crates.io/crates/powercut)
[](https://docs.rs/powercut)
[](https://github.com/Cservin69/powercut/actions/workflows/ci.yml)
`powercut` is a database-agnostic crash-injection harness. It runs your
workload to the point where your code says *"committed"*, captures the state
that would actually survive a failure at that instant, reboots a fresh instance
of your store from **only that state**, and lets you assert that what you acked
came back.
It has **zero dependencies**, contains no `unsafe`, needs no root, no FUSE
mount, no kernel module and no special runtime, and it works with FFI engines
(SQLite, DuckDB, RocksDB) as happily as with pure-Rust ones.
---
## The 10-line quickstart
```rust
use powercut::{Checker, ScratchDir, Scenario, SyncJournal, Tier};
let store = ScratchDir::new("quickstart").unwrap(); // your store's directory
let journal = SyncJournal::complete(); // records every successful fsync
let db = store.path().join("app.log");
let report = Scenario::new(store.path(), journal.clone())
.label("append-and-durable-ack")
.tier(Tier::PowerCut) // only fsync'd bytes survive
.ack(|| { // ...your real workload...
std::fs::write(&db, b"balance=100").unwrap();
journal.sync_path(&db).unwrap(); // the fsync that earns durability
"balance=100"
})
.unwrap()
.check(|crash, acked| { // reboot from the surviving bytes
let mut c = Checker::new();
c.survives("balance", Some(acked.to_string()),
std::fs::read_to_string(crash.dir().join("app.log")).ok());
c.finish()
});
report.assert_survived();
```
Delete the `journal.sync_path(&db)` line and that test goes red — because the
durable set is **derived from the syncs that actually happened**, never
declared by the test author. That is the whole design: removing an `fsync` from
your write path removes a file from the durable set, with no separate mutation
run and no list to keep in step.
---
## The tier model
`powercut` models two failures, and being sloppy about which one you are
measuring is the single most common mistake in hand-rolled durability tests.
| `Tier::ProcessCrash` | `SIGKILL`, a panic, an OOM kill, an OS crash | a byte-for-byte copy of the store at the ack — **everything** the app wrote to the filesystem | **passes** |
| `Tier::PowerCut` | the plug coming out | only bytes a **successful** `fsync` certified before the ack, truncated to the length each file had at its sync | fails |
Tier 1 reads through the OS page cache. It can only ever prove *"the write
reached the file"*. Tier 2 is the one that means *"the write reached stable
storage"*. `Tier::PowerCut` is the default because tier 1 is the one people
measure by accident.
At tier 2 `powercut` also captures a tier-1 copy at the same instant, so a
failure report can say which of the two very different bugs you have:
```text
asymmetry: the acked data IS present in the tier-1 byte copy taken at the
same instant. It reached the filesystem and no further — the missing thing
is an fsync, not a write.
```
## Four outcomes, not two
```text
Survived the durable set honoured the ack
Lost it did not, and coverage was good enough for that to mean something
NotMeasured the check ran but its result is not sound. NOT A PASS.
HarnessBroken powercut itself is the suspect, so this run accuses nobody
```
`NotMeasured` is load-bearing. If your witness admits it cannot see all of the
store's sync activity (`Coverage::Partial`), then a **failed** check proves
nothing — an unobserved `fsync` might have saved the data — so `powercut`
reports `NotMeasured` and never a loss. A **passed** check under partial
coverage is still sound, because the durable set was a subset of the truth.
That asymmetry is what stops this crate from libelling somebody's database.
`HarnessBroken` exists for the same reason. The dangerous failure of a
durability test is not "red when it should be green"; it is **red for a reason
that is not durability** — a store path that misses a sidecar, a copy that will
not reopen, a checker looking in the wrong place. `Acked::with_teeth_control`
re-runs the identical capture and checker against an explicitly flushed store.
That run *must* pass; if it does not, the verdict is `HarnessBroken`, never
`Lost`.
## The CI assertion is a ratchet, not a conformance check
```rust,no_run
# let reports: Vec<powercut::Report> = Vec::new();
powercut::assert_no_worse_than(&reports, "tests/durability-baseline.tsv");
```
Engines legitimately differ, and a suite that demands uniformity gets switched
off within a month. What CI actually wants is *"nothing started differing in a
new way today"*. Three rules: a verdict that got worse fails; a **new kind** of
violation fails even at the same verdict; and a case that vanishes from the run
— or newly becomes `NotMeasured` — fails, because coverage is not allowed to
quietly evaporate. Record with `POWERCUT_UPDATE_BASELINE=1 cargo test`.
That variable is **ignored when `CI` is set**. Recording a baseline accepts
whatever happened today as the new truth, so a stray `POWERCUT_UPDATE_BASELINE`
in a CI environment would turn the ratchet into a rubber stamp and no case
would ever go red again. Under `CI` the ratchet warns on stderr and asserts
against the committed baseline instead. Record locally, read the diff, commit
it — that review is the point of having a baseline at all.
---
## The durability primitive, and its one honest residual
`powercut` does its syncing through `std::fs::File::sync_all` and
`sync_data`, which dispatch as follows:
| macOS / iOS (Apple targets) | **`fcntl(fd, F_FULLFSYNC)`** | a real device-cache flush, not `fsync(2)` |
| Linux, other Unix | `fsync(2)`; `sync_data` → `fdatasync(2)` | |
| Windows | `FlushFileBuffers` | needs a write handle, which is why `sync_path` opens read-write |
This is worth stating plainly because the folklore is wrong in a way that
matters: on Apple targets Rust's standard library has issued `F_FULLFSYNC` for
`sync_all` for years (`library/std/src/sys/fs/unix.rs`, `#[cfg(target_vendor =
"apple")]`). You do **not** need a `libc` dependency or an `unsafe` block to get
a real flush on macOS, and a crate that tells you otherwise is out of date.
Two consequences `powercut` models explicitly:
- **`sync_data` is weaker than `sync_all`.** `fdatasync` need not flush
metadata, so a file's *length* may not be durable. `powercut` records the
kind, so a report tells you which one you relied on.
- **A file's *name* is a separate durability question.** On Unix a newly
created file only durably exists after its parent **directory** is fsynced.
`SyncJournal::sync_dir` does that and records it, and the capture prints a
note for every synced file whose directory was never synced.
**The residual, stated once and stated honestly: the guarantee bottoms out at
the drive honouring the flush.** `F_FULLFSYNC` asks the device to empty its own
write cache; Apple guarantees this for the internal storage on its own
machines, and nobody guarantees it for a third-party external enclosure, a USB
bridge, or a virtualised block device that has been told to lie. If your disk
lies about the flush, `powercut` believes the lie. No user-space tool can do
better — detecting a lying drive needs a hardware power interrupt and an
oscilloscope, not a test harness.
---
## Capturing a store on Windows
`powercut` copies the store's bytes **while the engine is still open** — no
graceful shutdown, no checkpoint. On Unix that is always possible. On Windows it
depends on how the engine locks its files:
| no locks, or locks a range beyond EOF | SQLite (its locks live at byte `0x40000000`) | works |
| exclusive lock over the whole file | redb (`LockFileEx` for as long as the DB is open) | **not possible** |
A byte range held under an exclusive Windows lock cannot be read through any
other handle, and no user-space trick gets around it — backup semantics bypass
*share modes*, not byte-range locks.
When this happens `powercut` returns `Error::Locked` from the capture, naming
the file. It does **not** skip the unreadable file and carry on: a store file
silently missing from a capture would manufacture a false `Lost`, which is
precisely the failure this crate exists to prevent. An honest error beats a
confident wrong answer.
The practical consequence: on Windows, measure engines that do not hold an
exclusive whole-file lock, or drive the engine so that no handle is open at the
ack. `examples/redb_backend.rs` prints this and declines to run on Windows for
exactly this reason.
---
## Non-goals
Stated up front so they can be linked rather than re-argued:
- **Not a filesystem crash-consistency tester.** It does not hunt for bugs in
ext4/APFS/NTFS. That is [CrashMonkey + ACE](https://github.com/utsaslab/crashmonkey)
(OSDI '18); go there.
- **Not a block-layer fault injector.** [LazyFS](https://github.com/dsrhaslab/lazyfs)
(VLDB '24, *"When Amnesia Strikes"*) is the serious prior art at that layer
and it is genuinely stronger than `powercut`: a FUSE filesystem with its own
page cache that can drop unsynced writes on command, integrated into Jepsen.
It is also Linux-only, needs a mount and privileges, and is orchestrated from
outside your test binary. `powercut` deliberately trades that power for
running inside a plain `cargo test` on macOS, Linux and Windows.
- **Not a distributed-systems simulator** (see `turmoil`, `madsim`) and **not a
concurrency checker** (see `shuttle`, `loom`).
- **Does not claim to test the drive's own write cache.** See the residual above.
- **Does not, in this version, observe syncs an FFI engine makes internally.**
There is no `LD_PRELOAD` / `DYLD_INSERT_LIBRARIES` interposition backend yet.
This is exactly what `Coverage::Partial` and `Finding::NotMeasured` are for,
and the SQLite example shows how to get a fully measured result anyway.
## Testing an engine that syncs where you cannot see
If your engine does its own `fsync` through libc, `powercut` cannot observe it
and must not pretend to. You have two honest options, and the shipped SQLite
example demonstrates both:
1. **Declare it.** `SyncJournal::partial("libsqlite3 syncs internally through
libc")`. Passes stay sound; failures come back as `NotMeasured` with that
sentence attached.
2. **Take the syncs over.** Configure the engine to perform *no* syncs of its
own (`PRAGMA synchronous=OFF`) and do them yourself at the ack with
`SyncJournal::sync_path`. Coverage is then complete *by construction*, and
what you have measured is the durable-ack pattern your application actually
ships.
## Examples
```console
cargo run --example generic_file_store # a hand-rolled store, no engine deps
cargo run --example sqlite_durable_ack # rusqlite: all four verdicts, one file
cargo run --example redb_backend # redb, observed through its StorageBackend
cargo run --example read_fork_close_tear # the flagship, as a matrix across engines
```
### The flagship: the read-fork close-tear
Open a second connection to your store. Close it. Now the data you already
acked — and whose bytes were genuinely `fsync`'d — can stop being durable,
while every subsequent `COMMIT` still returns `Ok`. Nothing errors. Nothing
logs. A checkpoint or compaction triggered by that second handle's lifecycle
moves your committed rows out of the file that was synced and into one that was
not.
`cargo run --example read_fork_close_tear` runs that as a matrix across the
shipped engines and prints what each one actually does. `powercut` catches it
because it captures the bytes each `fsync` **actually certified**: the file
that carried the warrant has shrunk or vanished, the file the data moved into
was rewritten in place and never re-synced, and neither one can honour the ack.
Here it is end to end, on a store simple enough to fit in a doctest:
```rust
use powercut::{Checker, ScratchDir, Scenario, SyncJournal, Tier, Verdict};
let store = ScratchDir::new("read-fork").unwrap();
let (dir, journal) = (store.path().to_path_buf(), SyncJournal::complete());
let log = dir.join("store.log");
let read_balance = move |d: &std::path::Path| -> Option<String> {
["store.base", "store.log"].iter().find_map(|n| {
let text = std::fs::read_to_string(d.join(n)).ok()?;
Some(text.lines().rev().find_map(|l| l.strip_prefix("balance="))?.to_string())
})
};
c.survives("balance", Some(acked.clone()), read_balance(crash.dir()));
c.finish()
};
// 1. Commit and make it genuinely durable.
let file = std::fs::File::create(&log).unwrap();
std::io::Write::write_all(&mut (&file), b"balance=100\n").unwrap();
journal.sync_all(&file, &log).unwrap();
journal.sync_dir(&dir).unwrap();
let before = Scenario::new(&dir, journal.clone())
.label("before").tier(Tier::PowerCut)
.ack(|| "100".to_string()).unwrap().check(check);
assert_eq!(before.verdict(), Verdict::Survived); // it really was durable
// 2. A reader opens, reads, and closes. Its close folds the log into a base
// file and unlinks the log. The unlink is made durable; the base is not.
let after = Scenario::new(&dir, journal.clone())
.label("after").tier(Tier::PowerCut)
.ack(|| {
let held = std::fs::read_to_string(&log).unwrap();
std::fs::write(dir.join("store.base"), held).unwrap(); // never fsynced
std::fs::remove_file(&log).unwrap();
journal.sync_dir(&dir).unwrap(); // the unlink IS durable
"100".to_string()
})
.unwrap().check(check);
assert_eq!(after.verdict(), Verdict::Lost); // nobody wrote. nobody errored.
assert!(after.to_text().contains("GONE from the store"));
```
---
## Contributing — an open, assertive posture
**Bring the argument.** This crate is an oracle: it tells people their database
lost data, and it will sometimes be wrong. Every genuine argument, counterexample,
engine report and feature request is welcome, and a well-argued improvement will
be accepted — including one that says a core design decision here is wrong.
"I think your tier model mismodels *X*" is the most useful issue you can file,
not an imposition.
Concretely: file an issue with a runnable reproduction and we will read your
storage engine. Send a PR without asking first. Disagree in public. The point of
this crate is to be *right about durability*, and reach and correctness both
come from other people pushing on it.
See [CONTRIBUTING.md](https://github.com/Cservin69/powercut/blob/main/CONTRIBUTING.md).
Machine-readable entry points for coding agents: [`llms.txt`](https://github.com/Cservin69/powercut/blob/main/llms.txt)
and [`AGENTS.md`](https://github.com/Cservin69/powercut/blob/main/AGENTS.md).
**Every public item carries its own compiling doctest** — 113 of them across
109 public items, run in CI on Linux, macOS and Windows. That is a hard rule
rather than a style preference: the docs are the adoption channel, coding
agents read them, and an example that does not compile is worse than no
example. If one on [docs.rs](https://docs.rs/powercut) fails to compile, that
is a bug worth an issue.
## MSRV
Rust **1.82**. Raising it is a minor-version change, and CI tests it.
## License
Dual-licensed under [MIT](https://github.com/Cservin69/powercut/blob/main/LICENSE-MIT)
or [Apache-2.0](https://github.com/Cservin69/powercut/blob/main/LICENSE-APACHE), at
your option.