zeph-bench

Benchmark harness for evaluating Zeph agent performance on standardized datasets.

Feeds LOCOMO, GAIA, FRAMES, LongMemEval, and tau2-bench tasks through the full Zeph agent loop and records correctness, latency, and token usage. The default run is a reproducible baseline — no tools, no memory, no MCP, temperature pinned to 0.0 — measuring raw model capability; SemanticMemory can be wired in per run to measure what memory adds (see Memory A/B mode).

Baseline Results

gpt-5.4-mini, baseline mode, 2026-04-25:

[!NOTE] Baseline mode injects a concise-answer system prompt and post-processes responses (first-line extraction, markdown strip) before scoring. This is the primary driver of score quality — without it, verbose answers fail both Token F1 and exact-match evaluators.

CLI Usage

zeph-bench is invoked through the main zeph binary (requires the bench feature):

# List available datasets and their cache status
zeph bench list

# Download tau2-bench into the local cache (other datasets must be fetched manually —
# `zeph bench list` prints their source URLs)
zeph bench download --dataset tau2-bench

# Run GAIA sample
zeph bench run \
  --dataset gaia \
  --data-file path/to/gaia.jsonl \
  --provider my-provider \
  --output results/

# Run a single scenario for debugging
zeph bench run \
  --dataset locomo \
  --data-file path/to/locomo.json \
  --scenario s1_0 \
  --output results/

# Resume an interrupted run
zeph bench run \
  --dataset gaia \
  --data-file path/to/gaia.jsonl \
  --resume \
  --output results/

# Print a summary of a previous run
zeph bench show --results results/results.json

[!TIP] --provider references a named entry from [[llm.providers]] in your config. If omitted, the default provider is used. Use a fast, cheap model for large evaluation runs.

Additional run flags:

Output directory receives two files: results.json (machine-readable) and summary.md (human-readable markdown table).

Memory A/B mode (--baseline)

--baseline runs the dataset twice — once with MemoryMode::Off and once with a per-scenario SQLite-backed SemanticMemory (MemoryMode::On) — and writes each pass to its own subdirectory plus a delta report:

<output>/baseline/memory-off/{results.json,summary.md}
<output>/baseline/memory-on/{results.json,summary.md}
<output>/baseline/comparison.json

comparison.json is a BaselineComparison: per-scenario ScenarioDelta records plus the aggregate score difference between the two passes.

Library Usage

use std::path::Path;
use zeph_bench::runner::{BenchRunner, RunOptions};
use zeph_bench::loaders::{GaiaLoader, GaiaEvaluator};
use zeph_llm::{any::AnyProvider, mock::MockProvider};

# async fn example() -> Result<(), zeph_bench::BenchError> {
let provider = AnyProvider::Mock(MockProvider::with_responses(vec!["1945".into()]));
let runner = BenchRunner::new(provider);
let opts = RunOptions::default();
let run = runner
    .run_dataset(&GaiaLoader::all_levels(), &GaiaEvaluator, Path::new("gaia.jsonl"), opts)
    .await?;
println!("mean score: {:.4}", run.aggregate.mean_score);
# Ok(())
# }

Implementing a custom dataset

use zeph_bench::scenario::{DatasetLoader, Evaluator, EvalResult, Scenario};
use std::path::Path;

struct MyLoader;

impl DatasetLoader for MyLoader {
    fn name(&self) -> &'static str { "my-dataset" }

    fn load(&self, path: &Path) -> Result<Vec<Scenario>, zeph_bench::BenchError> {
        // parse your file format here
        todo!()
    }
}

struct MyEvaluator;

impl Evaluator for MyEvaluator {
    fn evaluate(&self, scenario: &Scenario, response: &str) -> EvalResult {
        let passed = response.trim() == scenario.expected.trim();
        EvalResult {
            scenario_id: scenario.id.clone(),
            score: if passed { 1.0 } else { 0.0 },
            passed,
            details: format!("exact_match={passed}"),
        }
    }
}

Supported Datasets

[!NOTE] tau2-bench is tool-use, not knowledge retrieval: it runs under ResponseMode::ToolUse against a simulated environment (RetailEnv / AirlineEnv) and is scored on the action trace rather than the response text. Every other dataset runs under ResponseMode::TerseAnswer.

[!IMPORTANT] Requires Rust 1.97 or later.

Architecture

The harness is built on three composable traits:

• DatasetLoader — reads a dataset file, returns Vec<Scenario>
• Evaluator — scores one agent response against a Scenario
• BenchmarkChannel — headless Channel impl that drives the agent loop without a terminal

BenchRunner wires them together: one fresh Agent<BenchmarkChannel> per scenario, no shared state between runs. Results accumulate into a BenchRun and are persisted by ResultWriter.

RunOptions controls each run: scenario_filter (single-scenario debugging), completed_ids (resume), and memory_mode. With MemoryMode::On, BenchRunner::with_memory_params supplies the BenchMemoryParams (data dir, embedding model, run ID, dataset) used to build a per-scenario SQLite-backed SemanticMemory; MemoryMode::Off is the default.

Features

Installation

[dependencies]
zeph-bench = "0.22"

This crate is part of the Zeph workspace. See the API documentation for the complete reference.

License

Licensed under MIT OR Apache-2.0 — see LICENSE for details.