Crate eval_flake_rs 

eval-flake-rs

Detect flaky LLM eval cases by tracking pass/fail outcomes across repeated runs of the same suite.

• A case is stable when every run agrees.
• A case is flaky when runs disagree (at least one pass and at least one fail).
• A case is always failing when every run failed.

Flip-rate = fraction of run-to-run transitions where the outcome changed. 0.0 = stable, 0.5 = maximally flaky.

Example

use eval_flake_rs::{detect, CaseResult};

let runs: Vec<Vec<(&str, bool)>> = vec![
    vec![("case-1", true),  ("case-2", true),  ("case-3", false)],
    vec![("case-1", true),  ("case-2", false), ("case-3", false)],
    vec![("case-1", true),  ("case-2", true),  ("case-3", false)],
];
let results = detect(&runs);
assert!(matches!(results.iter().find(|r| r.case == "case-2").unwrap().status, CaseResult::Flaky));

Structs

Report

Per-case report.

Enums

CaseResult

Per-case status.

Functions

detect

Aggregate per-case status from a list of run outcomes.