Skip to main content

Crate rusty_eval

Crate rusty_eval 

Source
Expand description

§Rusty Eval

Rusty Eval is the foundation of Rusty’s Agent TestOps plane: it turns the Flight Recorder’s run evidence into release decisions. Composable pieces, usable on its own:

  • Datasets (dataset) — versioned JSONL datasets: a header (name, version, format_version) plus one dataset::EvalCase per line, each carrying an input payload, an expected tool-call trajectory with argument matchers, final-state predicates, and tags. Loading validates the schema version; serialization is canonical, so datasets diff cleanly in git.
  • Assertions (assertion) — deterministic checks over a recorded run’s evidence::RunEvidence: tool-call order (subsequence match), tool-call count, state predicates by JSON pointer, tool blacklists, cost and latency bounds. Every verdict carries its evidence — expected vs observed — into the report.
  • Experiments (experiment) — experiment::ExperimentRunner drives one agent over a dataset N times per case through the real rusty-agent-runtime executor, distills each run’s journal into evidence, grades it, and aggregates an experiment::ExperimentReport: pass rate per assertion, per-case detail, latency percentiles, total cost. Runs are sequential by default or bounded-parallel with deterministic report ordering.
  • Comparison (compare) — compare::compare diffs a candidate report against a baseline: per-assertion deltas, per-case regressions and improvements, and threshold-based regression flags (pass-rate drop, p95 latency growth).
  • Failure clustering (clustering) — deterministic signatures group failed runs by termination, assertions, and judge outcome, preserving source evidence under stable cluster ids.
  • Judges (judge) — judge::RuleBasedJudge scores deterministic expectations, while judge::ModelJudge adapts any runtime ChatModel into a strict structured evaluator with local threshold enforcement.
  • Human feedback (feedback) — deterministic pairwise annotation queues with reviewer leases, rubric validation, consensus resolution, disagreement adjudication, corrections, and promotion into versioned evaluation cases.
  • Release gates (gate) — versioned policies turn candidate reports and baseline comparisons into deterministic allow/block decisions with machine-readable evidence for every configured check.

§Quick sketch

use rusty_eval::{Dataset, ExperimentConfig, ExperimentRunner, PreparedRun, compare, CompareThresholds};

let dataset = Dataset::load("evals/math_tools_v1.jsonl")?;

let runner = ExperimentRunner::new(ExperimentConfig::new().with_runs_per_case(3));
let baseline = runner
    .run(&dataset, |case, journal| {
        // Build the agent under test for this case run; `journal` is the
        // run's Flight Recorder journal (wire it into recording graphs so
        // model/tool calls become assertion evidence).
    })
    .await?;

let candidate = /* run the changed agent the same way */ baseline.clone();
let verdict = compare(&baseline, &candidate, &CompareThresholds::default());
assert!(!verdict.regressed);

Re-exports§

pub use assertion::Assertion;
pub use assertion::AssertionResult;
pub use clustering::cluster_failures;
pub use clustering::AssertionFailureKey;
pub use clustering::ExecutionFailureCategory;
pub use clustering::FailureCause;
pub use clustering::FailureCluster;
pub use clustering::FailureClusterReport;
pub use clustering::FailureEvidenceRef;
pub use clustering::FailureOccurrence;
pub use clustering::FailureSignature;
pub use clustering::FailureTermination;
pub use clustering::FAILURE_CLUSTER_REPORT_FORMAT_VERSION;
pub use compare::compare;
pub use compare::AssertionDelta;
pub use compare::CaseChange;
pub use compare::CaseDelta;
pub use compare::CompareThresholds;
pub use compare::ComparisonReport;
pub use compare::LatencyDelta;
pub use compare::Regression;
pub use dataset::Dataset;
pub use dataset::EvalCase;
pub use dataset::Expectation;
pub use dataset::ExpectedToolCall;
pub use dataset::StatePredicate;
pub use dataset::DATASET_FORMAT_VERSION;
pub use error::EvalError;
pub use error::Result;
pub use evidence::RunEvidence;
pub use evidence::RunStatus;
pub use evidence::ToolCallRecord;
pub use experiment::AssertionPassRate;
pub use experiment::CaseReport;
pub use experiment::CaseRunReport;
pub use experiment::ExperimentConfig;
pub use experiment::ExperimentReport;
pub use experiment::ExperimentRunner;
pub use experiment::LatencyStats;
pub use experiment::PreparedRun;
pub use experiment::ReportSummary;
pub use experiment::REPORT_FORMAT_VERSION;
pub use feedback::AnnotationQueue;
pub use feedback::AnnotationStatus;
pub use feedback::AnnotationTask;
pub use feedback::ResolutionAuthority;
pub use feedback::ReviewCandidate;
pub use feedback::ReviewDecision;
pub use feedback::ReviewLease;
pub use feedback::ReviewResolution;
pub use feedback::ReviewRubric;
pub use feedback::ReviewSubmission;
pub use feedback::RubricCriterion;
pub use feedback::StoredReview;
pub use feedback::TraceRef;
pub use feedback::FEEDBACK_FORMAT_VERSION;
pub use gate::evaluate_gate;
pub use gate::GateCheck;
pub use gate::GateDecision;
pub use gate::GateMetric;
pub use gate::GateOutcome;
pub use gate::GatePolicy;
pub use gate::GATE_DECISION_FORMAT_VERSION;
pub use gate::GATE_POLICY_FORMAT_VERSION;
pub use judge::JudgeModel;
pub use judge::JudgeRequest;
pub use judge::JudgeVerdict;
pub use judge::ModelJudge;
pub use judge::RuleBasedJudge;
pub use judge::DEFAULT_MODEL_JUDGE_MAX_REQUEST_BYTES;
pub use judge::DEFAULT_MODEL_JUDGE_MAX_RESPONSE_BYTES;
pub use judge::DEFAULT_MODEL_JUDGE_PASS_SCORE;
pub use judge::MAX_MODEL_JUDGE_RATIONALE_BYTES;
pub use statistics::detect_pass_rate_regression;
pub use statistics::StatisticalDecision;
pub use statistics::StatisticalRegressionConfig;
pub use statistics::StatisticalRegressionReport;
pub use statistics::STATISTICAL_REGRESSION_FORMAT_VERSION;

Modules§

assertion
Deterministic assertions over run evidence.
clustering
Deterministic clustering of failed experiment runs.
compare
Baseline-vs-candidate comparison of experiment reports.
dataset
Versioned evaluation datasets.
error
Error types for Rusty Eval.
evidence
What a run left behind: the evidence assertions evaluate against.
experiment
The experiment runner: dataset in, evidence-graded report out.
feedback
Deterministic human-feedback operations for evaluation and learning loops.
gate
Deterministic release gates over evaluation evidence.
judge
The judge seam: scored evaluation beyond deterministic assertions.
statistics
Statistical regression detection for matched experiment reports.