rusty-eval 0.1.5

Agent TestOps for Rusty — versioned evaluation datasets, deterministic assertions over recorded runs, experiment reports, and baseline-vs-candidate comparison.
Documentation

rusty-eval

Agent TestOps for Rusty: versioned evaluation datasets, deterministic assertions over recorded runs, experiment reports, and baseline-vs-candidate comparison — built directly on the rusty-agent-runtime executor and Flight Recorder journal. No simulation harness, no live model required.

The pipeline

dataset (JSONL, versioned)
      │
      ▼
ExperimentRunner ── runs the agent N× per case through Executor::run,
      │            each run journaling into its own Flight Recorder journal;
      │            sequential by default or bounded-parallel
      ▼
RunEvidence ── ordered tool-call trajectory + final state + latency/cost,
      │        distilled from the journal
      ▼
Assertion::evaluate ── deterministic pass/fail with expected-vs-observed evidence
      │
      ▼
ExperimentReport ── per-case detail, pass rate per assertion, p50/p95 latency, cost
      │
      ▼
compare(baseline, candidate) ── per-assertion deltas, per-case regressions,
                                threshold-flagged release verdict
      │
      ▼
detect_pass_rate_regression ── paired exact significance + practical effect
      │
      ▼
cluster_failures ── stable failure signatures, ranked clusters, evidence links

Dataset format

JSONL: line 1 is the header, every following line is a case. Serialization is canonical (field order fixed, map keys sorted), so datasets are diffable in version control and load → save is byte-stable.

{"kind":"header","format_version":1,"name":"math-tools","version":"1.0.0"}
{"kind":"case","id":"add-two","input":{"messages":[{"role":"user","content":"2+3?"}]},"expect":{"tool_trajectory":[{"name":"calculator","args":{"/op":"add"}}],"state":[{"pointer":"/messages/3/content","expected":"the answer is 5"}]},"tags":["smoke"]}

Each case: id, input (merged into the run's initial state by default), expect (trajectory as an ordered subsequence with optional JSON-pointer argument matchers, state predicates, forbidden tools, cost/latency bounds), and tags. Loading validates format_version and refuses unknown versions.

Assertions

Deterministic checks only — no model in the loop:

Assertion Checks
tool_call_order Expected calls appear as an ordered subsequence, argument matchers satisfied
tool_call_count Exact call count for one tool
state[...] Final-state value at a JSON pointer equals the expected value
no_tool_call Blacklisted tools never called
max_cost / max_latency Run totals within bounds

Every verdict returns { assertion, passed, expected, observed, detail } — the report shows why a run failed, not just that it did.

Experiments run sequentially by default so latency measurements remain uncontended. ExperimentConfig::with_max_concurrency(n) opts into bounded parallel execution for larger suites. The bound covers graph execution and judge calls; every run still gets a fresh graph and journal. Reports always sort by dataset case and repetition, and concurrent infrastructure failures resolve to the earliest case/repetition rather than whichever future happens to finish first. Once a parallel failure is observed, no new runs are admitted; only the already-active window is drained. Sequential execution retains its original fail-fast behavior.

Baseline gates require matching concurrency settings. This keeps latency, cost, and rate-limit evidence comparable instead of silently mixing sequential and parallel workloads.

Statistical regression detection

detect_pass_rate_regression pairs baseline and candidate outcomes by case id and repetition, then applies a one-sided exact McNemar test. It fails closed when datasets, repetitions, or run keys do not match. A regression is reported only when the configured sample size, minimum pass-rate drop, and significance level are all satisfied; underpowered comparisons are explicitly insufficient_evidence, never silently treated as safe.

Failure clustering

cluster_failures turns a validated experiment report into deterministic failure groups keyed by termination mode, failed assertions, and judge outcome. Normalized categories and fingerprints distinguish root causes while removing volatile ids. Clusters are ranked by frequency and retain safe source references, never raw errors, observations, or judge rationales. Stable content-derived ids track the same failure shape across experiments. The versioned JSON artifact is recomputed against its source report when loaded, so supplied counts, members, signatures, or shares cannot survive tampering.

Judges

JudgeModel (one async judge method, mirroring the runtime's ChatModel) is the evaluator seam. RuleBasedJudge scores the fraction of expectations met without a live model. ModelJudge adapts any runtime ChatModel into a structured LLM judge: the trusted rubric stays in the system instruction, evidence travels as untrusted JSON, output is accepted only as one schema-constrained tool call or strict JSON fallback, scores and rationale sizes are bounded before parsing, response roles are verified, and pass/fail is derived locally.

Status

Foundation release (v0.1.0): library only, no CLI. Deliberately absent for now: provider-specific judge clients (use the runtime's ChatModel adapters), dataset/report storage backends beyond JSON files, and assertion kinds beyond the deterministic set above.