Expand description
Core statistical decision library. Domain-agnostic: consumes baseline vs
candidate trial data and returns a pass/fail/inconclusive verdict with
enough detail to explain why. No stdout/stderr side effects; that is the
CLI’s job (see main.rs).
Re-exports§
pub use error::VeridictError;pub use report::MultiReport;pub use report::Report;
Modules§
- correction
- Multiple-comparison correction for a
compare --metricrun’s family of simultaneous per-metric claims (--claim-correction bonferroni|holm, with--correctionkept as a deprecated alias for one release - seemain.rs’s CLI arg doc). - error
- input
- JSONL input parsing.
- matrix
- Comparison matrix across more than two candidates.
- metrics
- Turns raw records into a metric’s effect size and confidence interval.
- plan
veridict plan: given the same inputmatrixaccepts, recommends which pairs would most benefit from more trials to narrow their Elo-difference confidence interval to--min-elohalf-width - a ranked “what to test next” list, built directly on top ofmatrix::run’s output rather than duplicating its edge-aggregation logic.- power
veridict power: pre-experiment sample-size estimation forcompare --metric winrate/ sign-test/elo. The counterpart toverdict::estimate_additional_trials(which asks “how many more trials would likely resolve an already-run inconclusive result”) but before any trial has run: “how many trials would I need for a target probability of reaching a passing verdict at all.”- report
- The machine-readable verdict report, plus a Markdown rendering for
humans. JSON field set matches AGENTS.md’s “Minimum report fields”, plus
reason,pass_above/fail_below, andfailure_breakdown(all justified by AGENTS.md’s own “every verdict must include enough information to understand why” and “failure classification summary” requirements). - sprt
- Sequential probability ratio test: accumulate a log-likelihood ratio
over decisive trials and stop as soon as it crosses one of Wald’s
boundaries. Unlike
compare, there’s no confidence interval or threshold to configure - the test’s own alpha/beta are the guaranteed error rates, by construction (seestats::sprtfor the math and its documented “decisive games only” assumption). - stats
- time_
sensitive - Time-sensitive anytime-valid testing: an early rejection is worth more than a late one, and
this module lets a caller say so directly, as a reward assigned to when the wealth process
crosses
1/alpha, rather than only whether it does. - verdict
- Turns an effect size + confidence interval into a pass/fail/inconclusive decision.
- verify_
run verify-run: checks that a completed run’s raw data (amanifest.tomlof declared expectations plus agames.jsonlof per-record observations) is structurally sound before any statistics (compare/sprt) are trusted on it - pairing, ordering, contamination, and opaque-identifier consistency, not a metric or a verdict.
Structs§
- Failure
Caps - Hard per-run caps on technical failure counts (
timeout/crash/invalid- the same categoriesTrialStatus/FailureBreakdownalready track; no new domain-specific categories).None(theDefault) means uncapped - today’s existing behavior, unchanged, unless a cap is opted into. These are zero-tolerance-style gates (e.g.max_crashes = 0), not a rate threshold: a single technical failure can matter regardless of how many clean trials surround it, unlikedata_quality.high_failure_rate(a rate-based, purely advisory warning that never changesverdict).
Enums§
- Bootstrap
Method - Which bootstrap variant
mean-diffuses.Percentileis the default (unchanged from before this existed, so existing output doesn’t shift);Bcacorrects for bias and skewness at the cost of a little extra computation (a jackknife pass, still O(n));Basicreflects the percentile interval around the point estimate - simpler thanBca, but with no bias-correction of its own (seestats::bootstrap’s doc). - CiMethod
- Which confidence-interval method
winrate/sign-testuse.Exact(Clopper-Pearson) andJeffreysdon’t apply toelo(fractional successes) ormean-diff(not a binomial proportion at all) - both are derived from a true Beta-Binomial model, so requesting either for those metrics is a config error (VeridictError::IncompatibleCiMethod), not a silent fallback toWilson. - Failure
Policy - How a failed trial (
baseline_status/candidate_statusother thanok) affects a win/loss/draw-shaped metric (winrate/elo) orsprt.ReportOnly(default) is exactly today’s existing behavior, unchanged: a failure is still tallied intofailure_breakdown, but never itself contributes an outcome - only a literalresultfield does, and a status- only record (the common case) already contributes nothing today regardless of this enum.Exclude/Lossonly diverge fromReportOnlyin the less common case of a record carrying both a failure status and aresult(the schema doesn’t forbid this combination). Only meaningful for outcome-based metrics -mean-diff/sign-testhave no win/loss/draw outcome for a failed numeric trial to become, so requestingExclude/Losswith either is a config error (VeridictError::IncompatibleFailurePolicy), not an arbitrary numeric penalty. - Inconclusive
Kind - Sub-classifies a
Verdict::Inconclusiveresult by whether its CI excludes zero - the two situations read identically as"inconclusive"otherwise, but call for different next steps.None(not this enum) covers every case where the split doesn’t apply:Pass/Fail, anInconclusivecaused by zero usable trials rather than a real CI, or one forced by a breachedFailureCaps- seeverdict::classify_inconclusiveandbuild_report. - Match
Outcome - Result of a single named-competitor match (see
input::MatchRecord,matrix --matches). - Metric
Config - Which knob(s) a metric actually uses, replacing the
(MetricKind, CiMethod, BootstrapMethod)triocompare_one/compare_manyused to take as three independent parameters.eloreads neitherci_methodnorbootstrap_method, andmean-diffdoesn’t readci_method- passing one anyway used to be silently ignored or a runtimeIncompatibleCiMethoderror. Carrying only the field(s) a metric actually reads makes an invalid pairing a compile error instead of a runtime one.MetricKind-keyed code (Report.metric,build_report,estimate_additional_trials) is unchanged - callMetricConfig::kindto recover it.PartialEqonly, notEq:QuantileDiff’squantile: f64field has no total equality (f64isn’tEq), so the derive would fail across the whole enum otherwise. - Metric
Kind - Which statistical method computed the effect size and confidence interval.
- Outcome
- Result of a single win/loss/draw comparison.
- Promotion
- The one field a deployment/promotion pipeline should actually gate on -
collapses
ValidityandVerdictinto a single go/no-go:Promotedonly when the run is bothValidity::ValidandVerdict::Pass. Every other combination (invalid data, a fail, an inconclusive result) isNotPromoted. - Trial
Status - Health of a single trial’s execution, independent of any score it produced.
- Validity
- Whether a report’s underlying data is trustworthy enough to read a
Verdictoff of at all - independent of what that verdict says.Invalidmeans a hard technical-failure cap (FailureCaps) was breached: not “the evidence was weak” (that’sInconclusive, stillValid), but “the run itself can’t be trusted to have measured the candidate at all.” Keeping this a separate axis fromVerdictis the point: aPass/Failproduced from data that also breached a failure cap (possible under--failure-policy loss, where a crash can tip the numeric verdict) must never be reported as a cleanPass/Fail- seeverdict::apply_failure_caps. - Verdict
- Final decision returned for a comparison run.
Constants§
- MIN_
DECISIVE_ UNITS - A fixed-sample result needs at least this many independent analysis units before it may return
passorfail. One observation produces a point bootstrap distribution and can otherwise look certain despite containing no empirical variance information.
Traits§
- Into
Record Result - Lets
compare_one/compare_many(andmetrics::compute/metrics::compute_many) accept either a streamingResult-yielding iterator (the CLI’s real use case - parsing JSONL/CSV can fail mid- stream) or a plain iterator over already-valid(usize, Record)pairs (a caller that already has a validated slice/Vecin memory, with noResultto thread through) through the same function, instead of forcing every in-memory caller to write.map(Ok)just to satisfy the type. A blanketimpl<T> From<T> for Result<T, E>isn’t available here (implementing a foreign trait,From, for a foreign type,Result- even parameterized by a local type - violates Rust’s orphan rule), hence this small local trait instead.
Functions§
- compare_
many - Runs several metrics against the same records in a single pass over
records(seemetrics::compute_many), each against its own threshold (thresholds[i]formetrics[i]- a per-metric CLI override, or the same value repeated if the caller wants one threshold shared across all of them), and combines them into one overall verdict:Failif any metric fails, elseInconclusiveif any metric is inconclusive, elsePass. Matches the “a false pass is worse than an inconclusive result” rule: one metric failing sinks the whole run. - compare_
one - Runs one metric end to end: classify records, compute its effect and
confidence interval, and apply the pass/fail thresholds.
paired_by_idenables paired-testcase variance reduction (seemetrics::compute).cluster_by_idenables a cluster-robust bootstrap CI instead (winrate/elo only - mutually exclusive withpaired_by_id, seeVeridictError::IncompatibleClusterById/ClusterByIdConflictsWithPairedById).