Struct Evaluator 

pub struct Evaluator { /* private fields */ }

Evaluates a subject model against a benchmark dataset using an LLM judge.

Implementations

impl Evaluator

pub fn new( judge: Arc<AnyProvider>, benchmark: BenchmarkSet, budget_tokens: u64, ) -> Result<Self, EvalError>

Create a new Evaluator.

Errors

Returns EvalError::EmptyBenchmarkSet if the benchmark has no cases.

pub fn with_parallel_evals(self, n: usize) -> Self

Override the default concurrency limit for judge calls.

pub async fn evaluate( &self, subject: &AnyProvider, ) -> Result<EvalReport, EvalError>

Run the full benchmark against subject, returning aggregate scores.

Subject calls are sequential; judge calls are parallelized up to parallel_evals concurrent tasks. A per-invocation token budget is enforced across all judge calls.

Errors

Returns EvalError::Llm if any subject call fails fatally. Budget exhaustion and judge errors are handled gracefully (excluded from scores).