Aether Evals

Aether Evals is a Rust library that makes writing agent evals easier to express as ordinary cargo-nextest tests.

Table of Contents

• Quick Start
• Test organization
• Core API
• Dockerized Aether evals
• Assertions
• Failure output and debugging
• Git-backed evals
• Development

Quick Start

use aether_evals::{FakeAgent, Workspace, run_eval};

#[tokio::test]
async fn hello_world_test() -> Result<(), aether_evals::EvalRunError> {
    let report = run_eval(
        &FakeAgent::writes_file("hello.txt", "Hello, World!"),
        "Write 'Hello, World!' to hello.txt",
        Workspace::empty()?,
    )
    .await?;

    assert!(report.path("hello.txt").exists(), "{}", report.failure_context());
    Ok(())
}

Run evals with nextest:

cargo nextest run -p aether-evals --all-features
cargo nextest run --profile ci --all-features --workspace

The CI profile writes JUnit XML to target/nextest/ci/junit.xml.

Test organization

Aether Evals' fake-agent coverage should use normal test names. Reserve _eval suffixes for real provider-backed evals selected by the repository evals nextest group.

Core API

• run_eval(&agent, prompt, workspace) runs one eval and returns an EvalReport.
• DockerAetherAgent::new(image) runs aether headless --output json in a fresh Testcontainers container for each eval run.
• Workspace::empty() creates an isolated temp directory.
• Workspace::from_dir(path) copies fixture directory contents into a temp directory.
• Workspace::from_git_repo(GitRepoSpec { url, start_commit, gold_commit, subdir }) clones and checks out a git repository.
• EvalReport exposes the prompt, workspace, agent messages, tool-call helpers, and git diff summaries.

Dockerized Aether evals

Use DockerAetherAgent when an eval should exercise the real Aether CLI in an isolated container. Aether Evals mounts the workspace root at /workspace; git-backed workspaces with GitRepoSpec::subdir mount the repository root so .git remains available while headless runs from the matching relative cwd under /workspace.

use aether_evals::{DockerAetherAgent, DockerImage, GitRepoSpec, Workspace, run_eval};
use aether_project::AetherSettings;
use std::path::Path;

let settings = AetherSettings::load_file_for_export(Path::new(".aether/settings.json"))?;
let agent = DockerAetherAgent::new(DockerImage::new("aether-sandbox", "latest"))
    .with_settings(settings)
    .with_agent("Fast");

let report = run_eval(
    &agent,
    "fix the failing test",
    Workspace::from_git_repo(GitRepoSpec {
        url: "https://github.com/example/repo".to_string(),
        start_commit: "start_sha".to_string(),
        gold_commit: "gold_sha".to_string(),
        subdir: None,
    })?,
)
.await?;

By default the Docker agent forwards provider API key env vars, OLLAMA_HOST, and AETHER_* except host AETHER_HOME. Each run gets a fresh temporary container AETHER_HOME; configured settings are passed with --settings-json, and .with_agent(...) passes --agent.

Assertions

Use normal Rust assertions over the returned EvalReport and files on disk:

let report = run_eval(&agent, prompt, Workspace::empty()?).await?;

assert!(report.tool_called("write_file"), "{}", report.failure_context());
assert_eq!(report.tool_call_count("bash"), 1, "{}", report.failure_context());
assert_eq!(std::fs::read_to_string(report.path("output.txt"))?, "expected text\n");

Aether Evals also exports small #[track_caller] helpers for common tool assertions:

aether_evals::assert_tool_called(&report, "write_file");
aether_evals::assert_tool_call_count(&report, "bash", 1);
aether_evals::assert_tool_call_with_args(&report, "write_file", &serde_json::json!({ "path": "output.txt" }));

LLM judging runs through declarative eval files: add a rubric judge block to expect in a *.eval.json file. Each criterion is scored 0.0–1.0 by the judge model, weighted into an overall score, and blocking criteria below their threshold fail the eval. See the evals documentation for the file format.

Judge faults — transient LLM stream failures, unparseable responses, criterion sets that don't match the rubric — are recorded as failures on the eval's outcome rather than conflated with a "the agent didn't do it" verdict.

Failure output and debugging

EvalReport::failure_context() returns deterministic plain text suitable for assertion messages, nextest output, and JUnit failure bodies. It includes prompt, workspace path, agent messages, and git diff stats when available.

Temp directories are deleted when the report is dropped.

Git-backed evals

use aether_evals::GitRepoSpec;

let report = run_eval(
    &agent,
    "Make the test pass",
    Workspace::from_git_repo(GitRepoSpec {
        url: "https://github.com/example/repo".to_string(),
        start_commit: "start_sha".to_string(),
        gold_commit: "gold_sha".to_string(),
        subdir: Some("packages/api".into()),
    })?,
)
.await?;

assert!(report.tool_called("bash"), "{}", report.failure_context());

For git-backed evals, EvalReport exposes the agent diff from HEAD and the reference diff from start_commit..gold_commit when git commands succeed.

Development

just test -p aether-evals
just test-ci -p aether-evals
just lint -p aether-evals
just doc-check -p aether-evals