dev-flaky 0.1.0

Flaky-test detection for Rust. Repeated-run reliability tracking with per-test confidence scoring. Part of the dev-* verification suite.
Documentation
  • Coverage
  • 100%
    25 out of 25 items documented1 out of 15 items with examples
  • Size
  • Source code size: 48.27 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 557.03 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 23s Average build duration of successful builds.
  • all releases: 21s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Homepage
  • jamesgober/dev-flaky
    0 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • jamesgober

What it does

dev-flaky runs your test suite many times and tracks each test's pass/fail history. Stable tests pass every time. Flaky tests fail sometimes for no apparent reason. Broken tests fail every time.

After the run, every test gets a reliability score in [0.0, 1.0] and a classification (stable / flaky / broken).

Why flaky tests matter

Flaky tests are corrosive. After enough false alarms, developers start ignoring CI failures, and real failures get missed. Detecting flakiness automatically lets you quarantine the worst offenders before they erode trust in the suite.

Quick start

[dependencies]
dev-flaky = "0.9"

use dev_flaky::FlakyRun;

let run = FlakyRun::new("my-crate", "0.1.0").iterations(20);
let result = run.execute()?;

println!("Flaky tests: {}", result.flaky_count());
let report = result.into_report();
# Ok::<(), Box<dyn std::error::Error>>(())

Classification

Classification Pass count Fail count Verdict
Stable > 0 0 Pass
Flaky > 0 > 0 Warn
Broken 0 > 0 Fail

The per-test reliability percentage is attached as Evidence::Numeric so you can sort tests by flakiness in downstream reports.

The dev-* suite

See dev-tools for the full suite.

Status

v0.9.0 is the foundation release: API shape defined, subprocess integration lands in 0.9.1. Production use is discouraged until 1.0.

Minimum supported Rust version

1.85 — pinned in Cargo.toml and verified by CI.

License

Apache-2.0. See LICENSE.