Struct BenchRunner 

pub struct BenchRunner { /* private fields */ }

Drives Agent<BenchmarkChannel> over a dataset and collects scored results.

Each call to run_dataset creates a fresh agent per scenario (baseline mode: no tools, no MCP). Memory is optionally wired via BenchRunner::with_memory_params and RunOptions::memory_mode.

Examples

use zeph_bench::runner::BenchRunner;
use zeph_llm::{any::AnyProvider, mock::MockProvider};

let provider = AnyProvider::Mock(MockProvider::with_responses(vec!["Paris".into()]));
let runner = BenchRunner::new(provider, false);

Implementations

impl BenchRunner

pub fn new(provider: AnyProvider, _no_deterministic: bool) -> Self

Create a new runner with the given provider.

The no_deterministic argument is unused at runtime but kept in the public API so the bench command can pass it through for future use (e.g., logging or config). Apply deterministic overrides to provider before calling this if needed.

Examples

use zeph_bench::runner::BenchRunner;
use zeph_llm::{any::AnyProvider, mock::MockProvider};

let provider = AnyProvider::Mock(MockProvider::with_responses(vec![]));
let runner = BenchRunner::new(provider, false);

pub fn with_memory_params(self, params: BenchMemoryParams) -> Self

Attach SemanticMemory parameters for memory-on benchmark runs.

When set, a per-scenario SQLite-backed SemanticMemory is constructed inside run_one whenever opts.memory_mode == MemoryMode::On.

Examples

use std::path::PathBuf;
use zeph_bench::runner::{BenchRunner, BenchMemoryParams};
use zeph_llm::{any::AnyProvider, mock::MockProvider};

let provider = AnyProvider::Mock(MockProvider::with_responses(vec![]));
let params = BenchMemoryParams {
    data_dir: PathBuf::from("/tmp/bench-data"),
    embedding_model: "nomic-embed-text".into(),
    run_id: "bench-abc".into(),
    dataset: "locomo".into(),
};
let runner = BenchRunner::new(provider, false).with_memory_params(params);

pub async fn run_dataset<L, E>( &self, loader: &L, evaluator: &E, path: &Path, opts: RunOptions, ) -> Result<BenchRun, BenchError>

where L: DatasetLoader, E: Evaluator,

Run all matching scenarios from path through the agent and return a BenchRun.

For each scenario:

1. Builds a fresh Agent<BenchmarkChannel> with no tools or memory.
2. Feeds the scenario prompt and collects the agent’s response.
3. Scores the response with evaluator.
4. Appends a ScenarioResult and recomputes aggregate statistics.

The returned BenchRun has status = Running until the caller sets it to Completed or Interrupted.

Errors

Returns BenchError if the dataset cannot be loaded or a scenario run fails.

pub async fn run_dataset_with_env_factory<L, F, X>( &self, loader: &L, env_factory: F, path: &Path, opts: RunOptions, ) -> Result<BenchRun, BenchError>

where L: DatasetLoader, F: Fn(&Scenario) -> Result<(X, ActionTrace), BenchError>, X: ToolExecutor + Send + Sync + 'static,

Run all scenarios from path through a per-scenario env executor and return a BenchRun.

This is the execution path for tool-driven datasets (tau2-bench). For each scenario:

1. Calls env_factory(scenario) to build a fresh (ToolExecutor, ActionTrace).
2. Builds a fresh TauBenchEvaluator from the scenario metadata and the trace.
3. Runs the agent with the env executor and the tool-use system prompt.
4. Scores the response via the evaluator (reads the populated trace).

Errors

Returns BenchError if the dataset cannot be loaded, the env factory fails, or TauBenchEvaluator::from_scenario fails (malformed metadata).