Skip to main content

Module runner

Module runner 

Source
Expand description

Async test and eval execution: expand fixtures into Case rows, apply mocks or live generation, optionally run evaluators (rules / golden / process / judge), and produce a Run.

Entry points:

  • run_fixturestest-style runs (assertions + optional rules on each case).
  • run_evaleval-style runs with configurable evaluator and multi-run aggregation.

Requires a Tokio runtime (multi-thread recommended for parallel cases).

Structs§

EvalConfig
Extends RunnerConfig with evaluation-specific options (evaluator id, judge, thresholds, multi-run seeding, optional progress bar).
RunnerConfig
Tunables for run_fixtures (concurrency, timeouts, mocks directory, offline mode, etc.).
SecretScrubber

Enums§

GenerationMode
How synthetic traces are produced for each case.
GeneratorProvider
SandboxError

Functions§

run_case
Run a single crate::Case: replay tool expectations against crate::MockEngine, apply output rules when enabled, and return a crate::CaseResult.
run_eval
Like run_fixtures, but runs the eval pipeline (repeated runs, effectiveness threshold, selected crate::Evaluator) and attaches crate::EvalScores when applicable.
run_fixtures
Execute all cases from the given fixtures using RunnerConfig and return a completed Run.