Skip to main content

Crate veridict

Crate veridict 

Source
Expand description

Core statistical decision library. Domain-agnostic: consumes baseline vs candidate trial data and returns a pass/fail/inconclusive verdict with enough detail to explain why. No stdout/stderr side effects; that is the CLI’s job (see main.rs).

Re-exports§

pub use error::VeridictError;
pub use report::MultiReport;
pub use report::Report;

Modules§

correction
Multiple-comparison correction for a compare --metric run’s family of simultaneous per-metric claims (--claim-correction bonferroni|holm, with --correction kept as a deprecated alias for one release - see main.rs’s CLI arg doc).
error
input
JSONL input parsing.
matrix
Comparison matrix across more than two candidates.
metrics
Turns raw records into a metric’s effect size and confidence interval.
plan
veridict plan: given the same input matrix accepts, recommends which pairs would most benefit from more trials to narrow their Elo-difference confidence interval to --min-elo half-width - a ranked “what to test next” list, built directly on top of matrix::run’s output rather than duplicating its edge-aggregation logic.
power
veridict power: pre-experiment sample-size estimation for compare --metric winrate/ sign-test/elo. The counterpart to verdict::estimate_additional_trials (which asks “how many more trials would likely resolve an already-run inconclusive result”) but before any trial has run: “how many trials would I need for a target probability of reaching a passing verdict at all.”
report
The machine-readable verdict report, plus a Markdown rendering for humans. JSON field set matches AGENTS.md’s “Minimum report fields”, plus reason, pass_above/fail_below, and failure_breakdown (all justified by AGENTS.md’s own “every verdict must include enough information to understand why” and “failure classification summary” requirements).
sprt
Sequential probability ratio test: accumulate a log-likelihood ratio over decisive trials and stop as soon as it crosses one of Wald’s boundaries. Unlike compare, there’s no confidence interval or threshold to configure - the test’s own alpha/beta are the guaranteed error rates, by construction (see stats::sprt for the math and its documented “decisive games only” assumption).
stats
time_sensitive
Time-sensitive anytime-valid testing: an early rejection is worth more than a late one, and this module lets a caller say so directly, as a reward assigned to when the wealth process crosses 1/alpha, rather than only whether it does.
verdict
Turns an effect size + confidence interval into a pass/fail/inconclusive decision.
verify_run
verify-run: checks that a completed run’s raw data (a manifest.toml of declared expectations plus a games.jsonl of per-record observations) is structurally sound before any statistics (compare/sprt) are trusted on it - pairing, ordering, contamination, and opaque-identifier consistency, not a metric or a verdict.

Structs§

FailureCaps
Hard per-run caps on technical failure counts (timeout/crash/ invalid - the same categories TrialStatus/FailureBreakdown already track; no new domain-specific categories). None (the Default) means uncapped - today’s existing behavior, unchanged, unless a cap is opted into. These are zero-tolerance-style gates (e.g. max_crashes = 0), not a rate threshold: a single technical failure can matter regardless of how many clean trials surround it, unlike data_quality.high_failure_rate (a rate-based, purely advisory warning that never changes verdict).

Enums§

BootstrapMethod
Which bootstrap variant mean-diff uses. Percentile is the default (unchanged from before this existed, so existing output doesn’t shift); Bca corrects for bias and skewness at the cost of a little extra computation (a jackknife pass, still O(n)); Basic reflects the percentile interval around the point estimate - simpler than Bca, but with no bias-correction of its own (see stats::bootstrap’s doc).
CiMethod
Which confidence-interval method winrate/sign-test use. Exact (Clopper-Pearson) and Jeffreys don’t apply to elo (fractional successes) or mean-diff (not a binomial proportion at all) - both are derived from a true Beta-Binomial model, so requesting either for those metrics is a config error (VeridictError::IncompatibleCiMethod), not a silent fallback to Wilson.
FailurePolicy
How a failed trial (baseline_status/candidate_status other than ok) affects a win/loss/draw-shaped metric (winrate/elo) or sprt. ReportOnly (default) is exactly today’s existing behavior, unchanged: a failure is still tallied into failure_breakdown, but never itself contributes an outcome - only a literal result field does, and a status- only record (the common case) already contributes nothing today regardless of this enum. Exclude/Loss only diverge from ReportOnly in the less common case of a record carrying both a failure status and a result (the schema doesn’t forbid this combination). Only meaningful for outcome-based metrics - mean-diff/sign-test have no win/loss/draw outcome for a failed numeric trial to become, so requesting Exclude/Loss with either is a config error (VeridictError::IncompatibleFailurePolicy), not an arbitrary numeric penalty.
InconclusiveKind
Sub-classifies a Verdict::Inconclusive result by whether its CI excludes zero - the two situations read identically as "inconclusive" otherwise, but call for different next steps. None (not this enum) covers every case where the split doesn’t apply: Pass/Fail, an Inconclusive caused by zero usable trials rather than a real CI, or one forced by a breached FailureCaps - see verdict::classify_inconclusive and build_report.
MatchOutcome
Result of a single named-competitor match (see input::MatchRecord, matrix --matches).
MetricConfig
Which knob(s) a metric actually uses, replacing the (MetricKind, CiMethod, BootstrapMethod) trio compare_one/compare_many used to take as three independent parameters. elo reads neither ci_method nor bootstrap_method, and mean-diff doesn’t read ci_method - passing one anyway used to be silently ignored or a runtime IncompatibleCiMethod error. Carrying only the field(s) a metric actually reads makes an invalid pairing a compile error instead of a runtime one. MetricKind-keyed code (Report.metric, build_report, estimate_additional_trials) is unchanged - call MetricConfig::kind to recover it. PartialEq only, not Eq: QuantileDiff’s quantile: f64 field has no total equality (f64 isn’t Eq), so the derive would fail across the whole enum otherwise.
MetricKind
Which statistical method computed the effect size and confidence interval.
Outcome
Result of a single win/loss/draw comparison.
Promotion
The one field a deployment/promotion pipeline should actually gate on - collapses Validity and Verdict into a single go/no-go: Promoted only when the run is both Validity::Valid and Verdict::Pass. Every other combination (invalid data, a fail, an inconclusive result) is NotPromoted.
TrialStatus
Health of a single trial’s execution, independent of any score it produced.
Validity
Whether a report’s underlying data is trustworthy enough to read a Verdict off of at all - independent of what that verdict says. Invalid means a hard technical-failure cap (FailureCaps) was breached: not “the evidence was weak” (that’s Inconclusive, still Valid), but “the run itself can’t be trusted to have measured the candidate at all.” Keeping this a separate axis from Verdict is the point: a Pass/Fail produced from data that also breached a failure cap (possible under --failure-policy loss, where a crash can tip the numeric verdict) must never be reported as a clean Pass/Fail - see verdict::apply_failure_caps.
Verdict
Final decision returned for a comparison run.

Constants§

MIN_DECISIVE_UNITS
A fixed-sample result needs at least this many independent analysis units before it may return pass or fail. One observation produces a point bootstrap distribution and can otherwise look certain despite containing no empirical variance information.

Traits§

IntoRecordResult
Lets compare_one/compare_many (and metrics::compute/ metrics::compute_many) accept either a streaming Result-yielding iterator (the CLI’s real use case - parsing JSONL/CSV can fail mid- stream) or a plain iterator over already-valid (usize, Record) pairs (a caller that already has a validated slice/Vec in memory, with no Result to thread through) through the same function, instead of forcing every in-memory caller to write .map(Ok) just to satisfy the type. A blanket impl<T> From<T> for Result<T, E> isn’t available here (implementing a foreign trait, From, for a foreign type, Result - even parameterized by a local type - violates Rust’s orphan rule), hence this small local trait instead.

Functions§

compare_many
Runs several metrics against the same records in a single pass over records (see metrics::compute_many), each against its own threshold (thresholds[i] for metrics[i] - a per-metric CLI override, or the same value repeated if the caller wants one threshold shared across all of them), and combines them into one overall verdict: Fail if any metric fails, else Inconclusive if any metric is inconclusive, else Pass. Matches the “a false pass is worse than an inconclusive result” rule: one metric failing sinks the whole run.
compare_one
Runs one metric end to end: classify records, compute its effect and confidence interval, and apply the pass/fail thresholds. paired_by_id enables paired-testcase variance reduction (see metrics::compute). cluster_by_id enables a cluster-robust bootstrap CI instead (winrate/elo only - mutually exclusive with paired_by_id, see VeridictError::IncompatibleClusterById/ClusterByIdConflictsWithPairedById).