Expand description
The generic benchmark runner: ties a Harness + Scorer over a shared World, runs every
EvalCase, times each, isolates panics, and assembles an EvalReport.
This is the domain-agnostic port of AetherCore’s concrete eval runner: same per-case timing, same
catch_unwind panic isolation (one bad case fails alone), same panic-hook suppression around the
loop, and the same stderr progress lines — but with the game/LLM specifics (which backend, which
world, which predicates) pushed out behind the Harness/Scorer traits.
Structs§
- Agent
Harness - An internal adapter turning an
Agentinto aHarness<World = (), Setup = ()>so the easyrun_suitepath reuses the exact same panic-isolated runner as the fullHarnesspath. - RunMeta
- Run-level metadata recorded on the
EvalReportbut NOT intrinsic to the generic runner.
Functions§
- run_
eval - Run every
casethroughharness+scorerand aggregate into anEvalReport, using defaultRunMeta(temperature0, empty backend/system-prompt labels). - run_
eval_ with_ meta - As
run_eval, but with explicit runRunMeta(backend label, temperature, shared system prompt) recorded on the resultingEvalReport. - run_
suite - The easy path: run a suite of
Expectation-based cases against anAgent, scoring with the built-inBuiltinScorer— noWorld, noSetup, no hostScorerimpl. - run_
suite_ with_ meta - As
run_suite, but records explicitRunMeta(backend label, temperature, shared system prompt) on the report.