Struct EvalRunner 

pub struct EvalRunner { /* private fields */ }

Orchestrates evaluation: runs agents, captures trajectories, and scores results. Default: sequential, num_runs=1, no cache, no cancellation.

Implementations

impl EvalRunner

pub fn new(registry: EvaluatorRegistry) -> Self

Create a runner with a custom evaluator registry.

pub fn with_defaults() -> Self

Create a runner pre-loaded with built-in evaluators.

pub fn with_parallelism(self, n: usize) -> Self

Maximum number of concurrent case executions (FR-036).

Panics

Panics if n == 0.

pub fn with_num_runs(self, n: u32) -> Self

Repeat judge-side scoring n times per case (FR-037 / Q2).

Panics

Panics if n == 0.

pub fn with_cache(self, store: Arc<dyn EvaluationDataStore>) -> Self

Attach a pluggable EvaluationDataStore for cached invocations (FR-038).

pub fn with_cancellation(self, token: CancellationToken) -> Self

Attach a CancellationToken honored at every await point (FR-040).

pub fn with_initial_session_file(self, path: PathBuf) -> Self

Load the given JSON file as an initial SessionState before each case (FR-039 / R-023). Missing / malformed files surface as EvalError::InvalidCase — never a panic.

pub fn with_telemetry(self, telemetry: Arc<EvalsTelemetry>) -> Self

Attach an EvalsTelemetry (spec 043 US7 / FR-035). When present, Self::run_set emits the three-level span tree swink.eval.run_set → swink.eval.case → swink.eval.evaluator.

pub fn agent_invocation_count(&self) -> usize

Number of times an agent was actually invoked (cache miss count).

pub fn reset_agent_invocation_count(&self)

Reset the agent-invocation counter to zero.

pub async fn run_case( &self, case: &EvalCase, factory: &dyn AgentFactory, ) -> Result<EvalCaseResult, EvalError>

Run a single eval case and return the scored result.

pub async fn run_set( &self, eval_set: &EvalSet, factory: &dyn AgentFactory, ) -> Result<EvalSetResult, EvalError>

Run an entire eval set and return aggregated results.