Struct Evaluator 

pub struct Evaluator { /* private fields */ }

Evaluates a subject model against a benchmark dataset using an LLM judge.

Evaluator runs each BenchmarkCase against a subject model to obtain a response, then scores all responses in parallel using a separate judge model. The judge is prompted to return a JudgeOutput with a score in [1, 10].

Token Budget

A cumulative token budget is enforced across all judge calls in a single evaluate invocation. When the budget is exceeded the report has is_partial = true and the remaining futures are drained (any that already completed successfully are included in the scores).

Concurrency

Subject calls are sequential; judge calls are parallelized up to parallel_evals (default: 3) via a tokio semaphore.

Examples

let judge = Arc::new(AnyProvider::Mock(MockProvider::with_responses(vec![
    r#"{"score": 8.0, "reason": "mostly correct"}"#.into(),
])));
let subject = AnyProvider::Mock(MockProvider::with_responses(vec!["42".into()]));
let benchmark = BenchmarkSet {
    cases: vec![BenchmarkCase {
        prompt: "What is 6×7?".into(),
        context: None,
        reference: Some("42".into()),
        tags: None,
    }],
};
let evaluator = Evaluator::new(judge, benchmark, 50_000)?;
let report = evaluator.evaluate(&subject).await?;
assert_eq!(report.cases_scored, 1);

Implementations

impl Evaluator

pub fn new( judge: Arc<AnyProvider>, benchmark: BenchmarkSet, budget_tokens: u64, ) -> Result<Self, EvalError>

Create a new Evaluator.

Errors

Returns EvalError::EmptyBenchmarkSet if the benchmark has no cases.

pub fn with_parallel_evals(self, n: usize) -> Self

Override the default concurrency limit for judge calls.

The default is 3. A value of 0 is silently promoted to 1 (at least one judge call can run at a time).

Examples

let judge = Arc::new(AnyProvider::Mock(MockProvider::with_responses(vec![])));
let benchmark = BenchmarkSet {
    cases: vec![BenchmarkCase {
        prompt: "hi".into(), context: None, reference: None, tags: None,
    }],
};
let evaluator = Evaluator::new(judge, benchmark, 10_000)?.with_parallel_evals(5);

pub async fn evaluate( &self, subject: &AnyProvider, ) -> Result<EvalReport, EvalError>

Run the full benchmark against subject, returning aggregate scores.

Subject calls are sequential; judge calls are parallelized up to parallel_evals concurrent tasks. A per-invocation token budget is enforced across all judge calls.

Errors

Returns EvalError::Llm if any subject call fails fatally. Budget exhaustion and judge errors are handled gracefully (excluded from scores).