Expand description
§Rusty Eval
Rusty Eval is the foundation of Rusty’s Agent TestOps plane: it turns the Flight Recorder’s run evidence into release decisions. Composable pieces, usable on its own:
- Datasets (
dataset) — versioned JSONL datasets: a header (name,version,format_version) plus onedataset::EvalCaseper line, each carrying an input payload, an expected tool-call trajectory with argument matchers, final-state predicates, and tags. Loading validates the schema version; serialization is canonical, so datasets diff cleanly in git. - Assertions (
assertion) — deterministic checks over a recorded run’sevidence::RunEvidence: tool-call order (subsequence match), tool-call count, state predicates by JSON pointer, tool blacklists, cost and latency bounds. Every verdict carries its evidence — expected vs observed — into the report. - Experiments (
experiment) —experiment::ExperimentRunnerdrives one agent over a dataset N times per case through the realrusty-agent-runtimeexecutor, distills each run’s journal into evidence, grades it, and aggregates anexperiment::ExperimentReport: pass rate per assertion, per-case detail, latency percentiles, total cost. Runs are sequential by default or bounded-parallel with deterministic report ordering. - Comparison (
compare) —compare::comparediffs a candidate report against a baseline: per-assertion deltas, per-case regressions and improvements, and threshold-based regression flags (pass-rate drop, p95 latency growth). - Failure clustering (
clustering) — deterministic signatures group failed runs by termination, assertions, and judge outcome, preserving source evidence under stable cluster ids. - Judges (
judge) —judge::RuleBasedJudgescores deterministic expectations, whilejudge::ModelJudgeadapts any runtimeChatModelinto a strict structured evaluator with local threshold enforcement. - Human feedback (
feedback) — deterministic pairwise annotation queues with reviewer leases, rubric validation, consensus resolution, disagreement adjudication, corrections, and promotion into versioned evaluation cases. - Release gates (
gate) — versioned policies turn candidate reports and baseline comparisons into deterministic allow/block decisions with machine-readable evidence for every configured check.
§Quick sketch
use rusty_eval::{Dataset, ExperimentConfig, ExperimentRunner, PreparedRun, compare, CompareThresholds};
let dataset = Dataset::load("evals/math_tools_v1.jsonl")?;
let runner = ExperimentRunner::new(ExperimentConfig::new().with_runs_per_case(3));
let baseline = runner
.run(&dataset, |case, journal| {
// Build the agent under test for this case run; `journal` is the
// run's Flight Recorder journal (wire it into recording graphs so
// model/tool calls become assertion evidence).
})
.await?;
let candidate = /* run the changed agent the same way */ baseline.clone();
let verdict = compare(&baseline, &candidate, &CompareThresholds::default());
assert!(!verdict.regressed);Re-exports§
pub use assertion::Assertion;pub use assertion::AssertionResult;pub use clustering::cluster_failures;pub use clustering::AssertionFailureKey;pub use clustering::ExecutionFailureCategory;pub use clustering::FailureCause;pub use clustering::FailureCluster;pub use clustering::FailureClusterReport;pub use clustering::FailureEvidenceRef;pub use clustering::FailureOccurrence;pub use clustering::FailureSignature;pub use clustering::FailureTermination;pub use clustering::FAILURE_CLUSTER_REPORT_FORMAT_VERSION;pub use compare::compare;pub use compare::AssertionDelta;pub use compare::CaseChange;pub use compare::CaseDelta;pub use compare::CompareThresholds;pub use compare::ComparisonReport;pub use compare::LatencyDelta;pub use compare::Regression;pub use dataset::Dataset;pub use dataset::EvalCase;pub use dataset::Expectation;pub use dataset::ExpectedToolCall;pub use dataset::StatePredicate;pub use dataset::DATASET_FORMAT_VERSION;pub use error::EvalError;pub use error::Result;pub use evidence::RunEvidence;pub use evidence::RunStatus;pub use evidence::ToolCallRecord;pub use experiment::AssertionPassRate;pub use experiment::CaseReport;pub use experiment::CaseRunReport;pub use experiment::ExperimentConfig;pub use experiment::ExperimentReport;pub use experiment::ExperimentRunner;pub use experiment::LatencyStats;pub use experiment::PreparedRun;pub use experiment::ReportSummary;pub use experiment::REPORT_FORMAT_VERSION;pub use feedback::AnnotationQueue;pub use feedback::AnnotationStatus;pub use feedback::AnnotationTask;pub use feedback::ResolutionAuthority;pub use feedback::ReviewCandidate;pub use feedback::ReviewDecision;pub use feedback::ReviewLease;pub use feedback::ReviewResolution;pub use feedback::ReviewRubric;pub use feedback::ReviewSubmission;pub use feedback::RubricCriterion;pub use feedback::StoredReview;pub use feedback::TraceRef;pub use feedback::FEEDBACK_FORMAT_VERSION;pub use gate::evaluate_gate;pub use gate::GateCheck;pub use gate::GateDecision;pub use gate::GateMetric;pub use gate::GateOutcome;pub use gate::GatePolicy;pub use gate::GATE_DECISION_FORMAT_VERSION;pub use gate::GATE_POLICY_FORMAT_VERSION;pub use judge::JudgeModel;pub use judge::JudgeRequest;pub use judge::JudgeVerdict;pub use judge::ModelJudge;pub use judge::RuleBasedJudge;pub use judge::DEFAULT_MODEL_JUDGE_MAX_REQUEST_BYTES;pub use judge::DEFAULT_MODEL_JUDGE_MAX_RESPONSE_BYTES;pub use judge::DEFAULT_MODEL_JUDGE_PASS_SCORE;pub use judge::MAX_MODEL_JUDGE_RATIONALE_BYTES;pub use statistics::detect_pass_rate_regression;pub use statistics::StatisticalDecision;pub use statistics::StatisticalRegressionConfig;pub use statistics::StatisticalRegressionReport;pub use statistics::STATISTICAL_REGRESSION_FORMAT_VERSION;
Modules§
- assertion
- Deterministic assertions over run evidence.
- clustering
- Deterministic clustering of failed experiment runs.
- compare
- Baseline-vs-candidate comparison of experiment reports.
- dataset
- Versioned evaluation datasets.
- error
- Error types for Rusty Eval.
- evidence
- What a run left behind: the evidence assertions evaluate against.
- experiment
- The experiment runner: dataset in, evidence-graded report out.
- feedback
- Deterministic human-feedback operations for evaluation and learning loops.
- gate
- Deterministic release gates over evaluation evidence.
- judge
- The judge seam: scored evaluation beyond deterministic assertions.
- statistics
- Statistical regression detection for matched experiment reports.