Skip to main content

Module runner

Module runner 

Source
Expand description

The generic benchmark runner: ties a Harness + Scorer over a shared World, runs every EvalCase, times each, isolates panics, and assembles an EvalReport.

This is the domain-agnostic port of AetherCore’s concrete eval runner: same per-case timing, same catch_unwind panic isolation (one bad case fails alone), same panic-hook suppression around the loop, and the same stderr progress lines — but with the game/LLM specifics (which backend, which world, which predicates) pushed out behind the Harness/Scorer traits.

Structs§

AgentHarness
An internal adapter turning an Agent into a Harness<World = (), Setup = ()> so the easy run_suite path reuses the exact same panic-isolated runner as the full Harness path.
RunMeta
Run-level metadata recorded on the EvalReport but NOT intrinsic to the generic runner.

Functions§

run_eval
Run every case through harness + scorer and aggregate into an EvalReport, using default RunMeta (temperature 0, empty backend/system-prompt labels).
run_eval_with_meta
As run_eval, but with explicit run RunMeta (backend label, temperature, shared system prompt) recorded on the resulting EvalReport.
run_suite
The easy path: run a suite of Expectation-based cases against an Agent, scoring with the built-in BuiltinScorer — no World, no Setup, no host Scorer impl.
run_suite_with_meta
As run_suite, but records explicit RunMeta (backend label, temperature, shared system prompt) on the report.