Expand description
Async test and eval execution: expand fixtures into Case rows, apply mocks or live
generation, optionally run evaluators (rules / golden / process / judge), and produce a Run.
Entry points:
run_fixtures—test-style runs (assertions + optional rules on each case).run_eval—eval-style runs with configurable evaluator and multi-run aggregation.
Requires a Tokio runtime (multi-thread recommended for parallel cases).
Structs§
- Eval
Config - Extends
RunnerConfigwith evaluation-specific options (evaluator id, judge, thresholds, multi-run seeding, optional progress bar). - Runner
Config - Tunables for
run_fixtures(concurrency, timeouts, mocks directory, offline mode, etc.). - Secret
Scrubber
Enums§
- Generation
Mode - How synthetic traces are produced for each case.
- Generator
Provider - Sandbox
Error
Functions§
- run_
case - Run a single
crate::Case: replay tool expectations againstcrate::MockEngine, apply output rules when enabled, and return acrate::CaseResult. - run_
eval - Like
run_fixtures, but runs the eval pipeline (repeated runs, effectiveness threshold, selectedcrate::Evaluator) and attachescrate::EvalScoreswhen applicable. - run_
fixtures - Execute all cases from the given fixtures using
RunnerConfigand return a completedRun.