Skip to main content

run_eval

Function run_eval 

Source
pub fn run_eval<H, S>(
    harness: &H,
    scorer: &S,
    cases: &[EvalCase<H::Setup, S::Expect>],
) -> EvalReport
where H: Harness, S: Scorer<World = H::World>,
Expand description

Run every case through harness + scorer and aggregate into an EvalReport, using default RunMeta (temperature 0, empty backend/system-prompt labels).

This is the simple entry point for hosts that don’t track LLM run metadata. For LLM/agent runs that want the report to record a backend label, temperature, or shared system prompt, use run_eval_with_meta.

Semantics per case: build a fresh world via Harness::setup, time Harness::run (wall-clock for the whole run), then score every predicate via Scorer::score. A case PASSES iff run returned Ok AND every predicate passed. The whole build+run+score is isolated behind catch_unwind, so a panicking case fails only itself (with the panic message recorded in CaseOutcome::error). Progress is emitted to stderr.