1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
//! # Rusty Eval
//!
//! Rusty Eval is the foundation of Rusty's Agent TestOps plane: it turns the
//! Flight Recorder's run evidence into release decisions. Composable pieces,
//! usable on its own:
//!
//! - **Datasets** ([`dataset`]) — versioned JSONL datasets: a header
//! (`name`, `version`, `format_version`) plus one [`dataset::EvalCase`]
//! per line, each carrying an input payload, an expected tool-call
//! trajectory with argument matchers, final-state predicates, and tags.
//! Loading validates the schema version; serialization is canonical, so
//! datasets diff cleanly in git.
//! - **Assertions** ([`assertion`]) — deterministic checks over a recorded
//! run's [`evidence::RunEvidence`]: tool-call order (subsequence match),
//! tool-call count, state predicates by JSON pointer, tool blacklists,
//! cost and latency bounds. Every verdict carries its evidence — expected
//! vs observed — into the report.
//! - **Experiments** ([`experiment`]) — [`experiment::ExperimentRunner`]
//! drives one agent over a dataset N times per case through the real
//! `rusty-agent-runtime` executor, distills each run's journal into
//! evidence, grades it, and aggregates an
//! [`experiment::ExperimentReport`]: pass rate per assertion, per-case
//! detail, latency percentiles, total cost. Runs are sequential by default
//! or bounded-parallel with deterministic report ordering.
//! - **Comparison** ([`mod@compare`]) — [`compare::compare`] diffs a candidate
//! report against a baseline: per-assertion deltas, per-case regressions
//! and improvements, and threshold-based regression flags (pass-rate drop,
//! p95 latency growth).
//! - **Failure clustering** ([`clustering`]) — deterministic signatures group
//! failed runs by termination, assertions, and judge outcome, preserving
//! source evidence under stable cluster ids.
//! - **Judges** ([`judge`]) — [`judge::RuleBasedJudge`] scores deterministic
//! expectations, while [`judge::ModelJudge`] adapts any runtime `ChatModel`
//! into a strict structured evaluator with local threshold enforcement.
//! - **Human feedback** ([`feedback`]) — deterministic pairwise annotation
//! queues with reviewer leases, rubric validation, consensus resolution,
//! disagreement adjudication, corrections, and promotion into versioned
//! evaluation cases.
//! - **Release gates** ([`gate`]) — versioned policies turn candidate reports
//! and baseline comparisons into deterministic allow/block decisions with
//! machine-readable evidence for every configured check.
//!
//! ## Quick sketch
//!
//! ```no_run
//! use rusty_eval::{Dataset, ExperimentConfig, ExperimentRunner, PreparedRun, compare, CompareThresholds};
//!
//! # async fn demo() -> rusty_eval::Result<()> {
//! let dataset = Dataset::load("evals/math_tools_v1.jsonl")?;
//!
//! let runner = ExperimentRunner::new(ExperimentConfig::new().with_runs_per_case(3));
//! let baseline = runner
//! .run(&dataset, |case, journal| {
//! // Build the agent under test for this case run; `journal` is the
//! // run's Flight Recorder journal (wire it into recording graphs so
//! // model/tool calls become assertion evidence).
//! # let _ = (case, journal);
//! # unimplemented!()
//! })
//! .await?;
//!
//! let candidate = /* run the changed agent the same way */ baseline.clone();
//! let verdict = compare(&baseline, &candidate, &CompareThresholds::default());
//! assert!(!verdict.regressed);
//! # Ok(())
//! # }
//! ```
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;
pub use ;