faultbox

A production black-box recorder for Rust services: structured crash, corruption, and invariant-violation reports carrying a flight-recorder breadcrumb trail.

The goal is to debug a production failure from its report, without a reproduction and without shipping debug symbols to users.

[dependencies]
faultbox = "0.1"

Why this exists

Panic-only crash reporters miss the failures that matter most in a storage engine. Data corruption and violated invariants are usually returned errors — Err(ChecksumFailure), not a panic! — so nothing installs a hook for them, and they reach a user as a log line with no context. By the time anyone looks, the process that caused the damage is gone.

faultbox captures every failure class through one report format:

Each report carries the breadcrumb trail leading up to the failure, a build-id for offline symbolication, project-specific forensic context, and — for corruption — a preserved snapshot of the bad artifact.

When to use it

The question that decides it is not how long your process lives, it is: does your software detect failures it cannot reproduce?

Good fit:

• Storage engines, databases, file-format parsers, sync engines — anything that checks a checksum or an invariant at runtime and can therefore discover that its own data is wrong.
• Code that crosses into C or C++ through FFI, where a fault arrives as SIGSEGV rather than a panic. std::panic::set_hook cannot see those.
• Long-running services and desktop applications, where a crash otherwise leaves nothing behind.

Poor fit:

• no_std and embedded targets. This crate is std-only.
• Software whose only realistic failure is a panic, already reported somewhere. A panic-only reporter is simpler and does that job.

Libraries are a good fit, and are expected to be one. A library should not call init, install hooks, or choose the reports directory — that belongs to the binary. But a library that detects its own corruption should report it, and everything here stays inert until an application initializes, so depending on it costs a library nothing. See Using it across a dependency stack.

Usage

Initialize once at startup

use faultbox::{BasicRedactor, Config};

faultbox::init(
    Config::new("myapp", env!("CARGO_PKG_VERSION"), "/var/lib/myapp/reports")
        .git_sha(env!("GIT_SHA"))
        .redactor(Box::new(BasicRedactor::new())),
);

Full startup: tracing and native-crash capture together

Everything wired up, in the order it has to happen:

use tracing_subscriber::prelude::*;

fn main() {
    // 1. FIRST statement in main. This process may be the crash monitor — a
    //    re-exec of this same binary — in which case it serves minidumps here
    //    and never returns. Mandatory whenever the handler is armed below.
    if faultbox::run_crash_monitor_if_env() {
        return;
    }

    // 2. Initialize the recorder. Only the binary does this, and only once.
    faultbox::init(
        faultbox::Config::new("myapp", env!("CARGO_PKG_VERSION"), "/var/lib/myapp/reports")
            .redactor(Box::new(faultbox::BasicRedactor::new()))
            .features(["encryption", "compression"])
            // Spawns the out-of-process monitor. Safe because of step 1.
            .install_native_crash_handler(true),
    );

    // 3. Install the tracing layer, so events the app already emits become the
    //    flight recorder.
    tracing_subscriber::registry()
        .with(tracing_subscriber::fmt::layer())
        .with(faultbox::BreadcrumbLayer::new().only_targets(["myapp"]))
        .init();

    run_app();
}
# fn run_app() {}

Steps 2 and 3 can be swapped — BreadcrumbLayer feeds a ring that is inert until init runs, and init does not depend on tracing. Step 1 cannot move: it must precede all other work, including argument parsing.

Get a breadcrumb trail for free

If the project already uses tracing, the tracing feature turns existing instrumentation into the flight recorder — no manual breadcrumb calls:

use tracing_subscriber::prelude::*;

tracing_subscriber::registry()
    .with(faultbox::BreadcrumbLayer::new().only_targets(["myapp"]))
    .init();

Otherwise, place breadcrumbs by hand at significant operations. They are a no-op until init runs, so libraries may emit them unconditionally:

faultbox::breadcrumb!(Info, "myapp.commit", "committed", { "commit_id": 9558 });

Capture at the detection site

use faultbox::{Capture, EventKind};

let _ = Capture::new(EventKind::Corruption, "internal node references a non-btree page")
    .error_chain(faultbox::error_chain_of(&err))
    .domain(&ctx)                       // your DomainContext impl
    .preserve("store-snapshot", &store_path, "store.corrupt", None)
    .emit();

DomainContext is the per-project extension point. Its grouping_key should identify the class of failure, never the instance — "child_kind=0x09", not the page id that happened to be involved — so the same bug groups together across machines.

emit is synchronous and does real filesystem work (digest, copy, fsync, a lock). That is deliberate — it must be callable from a panic hook, where an async runtime may no longer be usable. In an async application, call it from spawn_blocking (or equivalent) so a failure report cannot stall an executor thread.

Read reports back

for group in faultbox::reader::list("/var/lib/myapp/reports")? {
    println!("{}", group.summary());
    // corruption  ×412  4121763d  store.dangling_child  internal node references …
}

Using it across a dependency stack

When several layers of one stack use faultbox — a storage engine, the database built on it, the application on top — they share a single recorder per process. Four rules follow.

Only the binary calls init. It is once-per-process and the first call wins; a later one returns false and changes nothing, not even partially. A library that calls init therefore either silently loses or silently decides the reports directory and redactor for everyone. Libraries only emit breadcrumbs, which are inert until the application initializes.

Breadcrumbs merge; they do not duplicate. Every layer records into one ring, so the trail interleaves in call order with each crumb recorded exactly once. This is the payoff: a report filed deep in the storage engine still carries the query that led to the bad read.

Report at the detection site. One emit is one report. If each layer emits on the way up, you get three reports for one root cause — faultbox cannot tell they are related, since each has its own domain kind and fingerprint. The layer that knows the invariant is the layer that reports it; layers above add breadcrumbs and propagate the error.

Make sure there is exactly one faultbox in the graph. This is the quiet one. If crate A depends on it by path and crate B by version, Cargo treats those as different packages and compiles both — giving two independent sets of process-wide state. The application initializes one; every emit through the other returns None and disappears, with no error and no report. The same applies to two semver-incompatible versions. Resolve all of them to one source ([patch], a shared path, or a single published version), and check with cargo tree -d.

Attribution still works across layers: meta.project names the process, while domain_kind records which layer detected the failure.

Feature flags are unioned by Cargo, so if any crate in the graph enables native-crash, it is compiled into everything. That is harmless — arming the handler is a separate, explicit decision the application makes.

Reports coalesce by bug, not by occurrence

A report directory is keyed by fingerprint and holds:

<reports_dir>/<fingerprint>/
├── report.json      first capture, plus occurrences / first_seen / last_seen
├── latest.json      most recent capture (only once a bug repeats)
└── store.corrupt/   preserved artifact, stored once

On unix, everything in there is created owner-only — directories 0700, files 0600. Redaction cleans the report's strings, but the things travelling beside them cannot be redacted and were never meant to be: a preserved artifact is a verbatim copy of your store, and a minidump is the crashed process's entire address space. On a multi-user host those must not be readable by other local accounts, so the recorder does not leave the mode to the umask. A preserved artifact does not inherit a permissive mode from its source, either. (On Windows there are no mode bits: files inherit the parent's ACL, so the reports directory is as private as the location you point it at — put it under %LOCALAPPDATA%.)

Artifact names are validated as a single plain path component: they are joined onto the report directory, so a name containing a separator or .. would place the artifact — and the removal that precedes it — somewhere else entirely. A rejected name is recorded on the report as NOT PRESERVED, not silently dropped.

A crash loop re-detecting one bug increments a counter instead of writing a new directory each time. This is not cosmetic: under a supervised restart loop, a per-occurrence layout writes thousands of directories for a single bug, each with its own copy of the store. Retention caps the directory, because a recorder that fills the disk of the process it monitors has done more damage than the bug it recorded.

Because coalescing hides the true failure count behind a small directory listing, reader::total_occurrences gives you the honest number.

The limits are all configurable:

use faultbox::Retention;

Config::new(/* … */)
    .breadcrumb_capacity(256)                  // ring size; default 128
    .preserve_max_bytes(64 * 1024 * 1024)      // artifact cap; default 256 MiB
    .retention(Retention {
        max_groups: 32,                        // default 64
        max_total_bytes: 512 * 1024 * 1024,    // default 2 GiB
    })

An artifact over the cap is not copied, and the report records that it was skipped and where the source is — a missing snapshot is stated, never merely absent. Digesting and copying happen before the group lock is taken, so preserving a large store never blocks another process's report.

Feature flags

All are off by default; the base crate depends only on serde and serde_json.

shared-ring: corruption caused elsewhere

An in-process breadcrumb ring can only show what this process did. When corruption is written by one process and detected when another opens the store, the detecting process's trail is structurally the wrong trail.

A SharedRing lives in a memory-mapped file beside the store, so every process touching it appends to one trail and a report can show the writes that actually caused the damage:

let ring = std::sync::Arc::new(faultbox::shared_ring::SharedRing::open(
    store_path.join(".faultbox-ring"),
    512,
)?);
faultbox::init(Config::new(/* … */).shared_ring(ring));

It uses no locks — a slot is claimed with one atomic increment and published with a seqlock — so a process killed mid-write costs exactly one breadcrumb and can never wedge the ring for anyone else.

An existing ring is joined as-is and its capacity is adopted, even if this caller asked for a different one; capacity() reports what you actually got. Resizing a file that other processes have mapped would leave their mappings pointing past end-of-file, and their next breadcrumb would take SIGBUS — a recorder must not be able to kill the processes it is watching because two of them disagreed about a number.

Because the ring is a file, it also outlives the process that wrote it. With native-crash enabled as well, the crash monitor recovers the trail after a fatal signal and writes it into the report — so a native crash arrives with the operations that led up to it instead of a bare minidump. The in-process recorder cannot do this: it dies with the process, and a signal handler could not read it in any case, because it sits behind a Mutex.

Crumbs are redacted on the way into the ring rather than on the way out of a report. The monitor is a separate process with no access to your Redactor, so anything written to the ring has to be safe already.

native-crash requires one line in main

The monitor is a re-exec of the host binary, so it needs main to identify itself. This call is mandatory whenever the handler is armed:

fn main() {
    if faultbox::run_crash_monitor_if_env() {
        return;
    }
    // ...normal startup
}

Without it, each spawned monitor runs the application again and spawns another monitor — exponentially. For that reason the handler is off by default even when the Cargo feature is enabled, and must be armed explicitly:

Config::new(/* … */).install_native_crash_handler(true)

init detects a failed-to-divert monitor and exits it rather than letting it multiply, but that is a backstop, not a substitute for the call.

Redaction

Every string entering a report — messages, error chains, breadcrumbs, domain values — passes through a Redactor. The default is NoopRedactor, so set one explicitly before reports leave a machine.

BasicRedactor masks:

• Paths — home directory to ~, username elsewhere to ~user. Case- and separator-insensitive, path-boundary aware.
• Credential assignments — the value under a key naming a credential (password, token, authorization, cookie, x-api-key, AZURE_OPENAI_KEY), quoted or bare, spaced or tight, or after an auth scheme (Bearer <token>). A composite value — Some("…"), ["…", "…"], Secret { inner: "…" } — is masked leaf by leaf, structure kept.
• JSON — the same keys as object members, in breadcrumb fields and DomainContext payloads. A masked leaf becomes the [redacted] string whatever its type was.
• Keyless credentials — issuer prefixes (sk-, ghp_, AKIA, …), JWTs, PEM blocks.
• Addresses — emails however punctuated, including object keys made of one.

Limits: no entropy scoring, so build-ids and hashes survive. An unquoted value is one token — token: abc def masks only abc; quote it or use an auth scheme for the full span. Credential-adjacent keys (grouping_key, auth) mask only credential-shaped values, so grouping_key=kind=0x09 stays readable.

It is a sensible default, not a compliance boundary: projects handling regulated data should compose their own on top.

Platform support

Requires std; there is no no_std build. Linux, macOS, and Windows run the full test suite in CI.

wasm32-unknown-unknown compiles and is inert: breadcrumbs are recorded in memory, but there is no filesystem to write a report to and no signals to catch. It is safe to have in the dependency tree of a crate that also targets wasm — linking it will not break the build or panic at runtime — but nothing is captured there. wasm32-wasip1 is compile-checked too.

native-crash and shared-ring are host-only. They are off by default, so a wasm or minimal build never pulls in crash-handler, minidumper, or memmap2.

Stability

Pre-1.0, the on-disk report shape is changed in place as the crate learns what triage actually needs. There are no migrations and no compatibility shims — reports are short-lived diagnostic artifacts, so delete a stale reports directory rather than parsing an older layout.

Notable changes are recorded in CHANGELOG.md.

Contributing

Bug reports and PRs are welcome — see CONTRIBUTING.md. A reproducing program is worth more than a description.

Security issues go through GitHub Security Advisories, not public issues; SECURITY.md covers what qualifies. A missed credential shape in BasicRedactor is an ordinary bug — file it publicly.

License

MIT OR Apache-2.0.