Skip to main content

Harness

eval_core::harness

Trait Harness 

Source 
pub trait Harness {
    type World;
    type Setup;

    // Required methods
    fn setup(&self, setup: &Self::Setup) -> Self::World;
    fn run(
        &self,
        instruction: &str,
        world: &mut Self::World,
    ) -> Result<RunArtifacts>;
}
Expand description

The thing being benchmarked: the host’s agent harness. The host implements this over its own world.

The runner calls setup once per case to build a FRESH world, then run once against that world. Implementations should be deterministic given the same Setup (the eval forces a deterministic configuration where it can) so runs are comparable.

Method calls are isolated behind catch_unwind by the runner, so a panic inside setup/run fails only the offending case; implementors need not catch their own panics.

For the common “score what the agent DID, against () world + the built-in Expectations” case, prefer the simpler Agent trait + run_suite — no World/Setup/Scorer to implement.

Required Associated Types§

Source

type World

The mutable world a case runs against (e.g. an executor over a bare game world). Built by setup, mutated by run, then scored by a Scorer whose Scorer::World must match this type.

Source

type Setup

The host’s per-case setup description (the input that builds a World). Matches EvalCase::setup.

Required Methods§

Source

fn setup(&self, setup: &Self::Setup) -> Self::World

Build a fresh world for one case from its setup. Called exactly once per case, immediately before run. A panic here fails only this case.

Source

fn run( &self, instruction: &str, world: &mut Self::World, ) -> Result<RunArtifacts>

Run instruction against world, mutating it, and return what the run produced (tool calls, final text, tokens, transcript). Wall-clock latency is timed by the runner around this call.

Return Err for a run-level failure (backend error, etc.); the case is then marked failed and the error recorded. Predicates are still scored against whatever world state exists, but a case can never PASS when run returned Err.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§