Aether Evals

Aether Evals is a Rust library that makes writing agent evals easier to express as ordinary cargo-nextest tests.

Table of Contents

• Quick Start
• Test organization
• Core API
• Dockerized evals
• Assertions
• Debugging failed evals
• Git-backed evals
• Development

Quick Start

use aether_evals::{Agent, FakeAgent, Task, Transcript, Workspace};

#[tokio::test]
async fn hello_world_test() -> Result<(), aether_evals::WorkspaceError> {
    let workspace = Workspace::empty()?;
    let prompt = "Write 'Hello, World!' to hello.txt";
    let agent = FakeAgent::writes_file("hello.txt", "Hello, World!").with_workspace(workspace.path());
    let transcript = Transcript::from_stream(agent.run(Task::new(prompt))).await.unwrap();

    assert!(!transcript.events().is_empty());
    assert!(workspace.join("hello.txt").exists());
    Ok(())
}

Run evals with nextest:

cargo nextest run -p aether-evals --all-features
cargo nextest run --profile ci --all-features --workspace

The CI profile writes JUnit XML to target/nextest/ci/junit.xml.

Test organization

Aether Evals' fake-agent coverage should use normal test names. Reserve _eval suffixes for real provider-backed evals selected by the repository evals nextest group.

Core API

• Task::new(prompt) describes the problem to solve.
• Agent::run(task) streams AgentEvents from an agent's configured execution environment.
• Transcript::from_stream(agent.run(task)) collects the stream into a Transcript.
• Transcript::default() plus transcript.add(message) supports observing each streamed message while building the transcript manually.
• Container::builder(image).start(&workspace) starts a caller-owned Docker container with the workspace mounted at /workspace.
• DockerAgent::new(container, command) runs an in-container command whose stdout is newline-delimited AgentEvent JSON.
• Container::exec_shell(script) runs follow-up assertions or test commands in the same caller-owned container.
• eval_env_vars(&model) forwards only the credentials for the providers model runs on into Docker, and sets AETHER_HOME=/root/.aether.
• Workspace::empty() creates an isolated temp directory.
• Workspace::from_dir(path) copies fixture directory contents into a temp directory.
• Workspace::from_git(GitSpec { source, start_commit, gold_commit, subdir, branch, diff_exclude }) checks out the start commit on branch (default main), fetched from GitSource::Url(url) or GitSource::Bundle(path) (see create_git_bundle). The workspace holds only the start commit's history, with no remote, so the agent can't find the gold commit in git. That history is fetched in full, blobs included, so reuse one bundle across runs for large repositories.
• Transcript exposes collected agent messages and tool-call helpers. transcript.session_usage() returns the session's cumulative tokens and estimated cost, sub-agents' calls included, from the last usage event.

Dockerized evals

Aether Evals mounts the workspace root at /workspace and runs the command from the effective eval cwd. It does not wrap or alter prompts; the command receives the caller-provided prompt verbatim:

• AETHER_EVAL_TASK_PROMPT — the caller-provided prompt to send to the agent unchanged.
• AETHER_EVAL_WORKSPACE_ROOT=/workspace — the mounted workspace root.
• AETHER_EVAL_CWD — the command's effective cwd inside the workspace.

Stdout is reserved for AgentEvent JSON lines and must end with a terminal event such as {"category":"turn","event":{"type":"ended","outcome":{"status":"completed"}}}. Diagnostic logging should go to stderr.

use aether_evals::{
    Agent, CONTAINER_AETHER_HOME, Container, DockerAgent, GitSource, GitSpec, Image, Task, Transcript, Workspace,
    eval_env_vars,
};

let workspace = Workspace::from_git(GitSpec {
    source: GitSource::Url("https://github.com/example/repo".to_string()),
    start_commit: "start_sha".to_string(),
    gold_commit: "gold_sha".to_string(),
    subdir: None,
    branch: None,
    diff_exclude: Vec::new(),
})?;
let container = Container::builder(Image::new("aether-sandbox", "latest"))
    .with_env_vars(eval_env_vars(&"anthropic:claude-opus-4-6".parse()?))
    .with_ephemeral_mount(CONTAINER_AETHER_HOME)
    .start(&workspace)
    .await?;
let agent = DockerAgent::new(container.clone(), vec!["/usr/local/bin/aether-eval-agent".to_string()]);

let prompt = "fix the failing test";
let transcript = Transcript::from_stream(agent.run(Task::new(prompt))).await?;
let tests = container.exec_shell("cargo test").await?;
assert_eq!(tests.exit_code, 0, "stdout:\n{}\nstderr:\n{}", tests.stdout, tests.stderr);
assert!(!transcript.events().is_empty());

Run follow-up container.exec_shell(...) assertions before dropping the caller-owned container and workspace.

A wrapper script for evaluating Aether itself can delegate to headless JSON output:

#!/bin/sh
exec aether headless \
  --output json \
  "$AETHER_EVAL_TASK_PROMPT"

A declarative eval:

{
  "agent": {
    "command": ["node", "/app/dist/eval-agent.js"]
  }
}

Assertions

Use normal Rust assertions over the returned Transcript and files on disk:

let workspace = Workspace::empty()?;
let transcript = Transcript::from_stream(agent.run(Task::new(prompt))).await?;

assert!(transcript.tool_called("write_file"));
assert_eq!(transcript.tool_call_count("bash"), 1);
assert_eq!(std::fs::read_to_string(workspace.join("output.txt"))?, "expected text\n");

Aether Evals also exports small #[track_caller] helpers for common tool assertions:

aether_evals::assert_tool_called(&transcript, "write_file");
aether_evals::assert_tool_call_count(&transcript, "bash", 1);
aether_evals::assert_tool_call_with_args(&transcript, "write_file", &serde_json::json!({ "path": "output.txt" }));

LLM judging runs through declarative eval files: add a rubric judge block to expect in a *.eval.json file. Each criterion is scored 0.0–1.0 by the judge model, weighted into an overall score, and blocking criteria below their threshold fail the eval. See the evals documentation for the file format.

Judge faults — transient LLM stream failures, unparseable responses, criterion sets that don't match the rubric — are recorded as failures on the eval's outcome rather than conflated with a "the agent didn't do it" verdict.

Debugging failed evals

TranscriptError preserves the partial transcript if an agent stream returns an error. Use error.transcript().events() or error.into_parts() to inspect events that were emitted before the failure.

Workspace removes its temporary directory when dropped. To retain files for debugging, call workspace.persist() and remove the returned path manually when finished.

For git-backed workspaces, workspace.capture_agent_diff()? returns the agent's commits, staged and unstaged edits, and untracked files since the start commit, and errors if git can't read the repository. workspace.reference_diff() returns the start-to-gold diff, computed before the gold commit was discarded. Both are None for workspaces without a start commit, and both leave out diff_exclude pathspecs such as *pnpm-lock.yaml.

Git-backed evals

use aether_evals::{Agent, GitSource, GitSpec, Task, Transcript, Workspace};

let workspace = Workspace::from_git(GitSpec {
    source: GitSource::Bundle("fixtures/repo.bundle".into()),
    start_commit: "start_sha".to_string(),
    gold_commit: "gold_sha".to_string(),
    subdir: Some("packages/api".into()),
    branch: None,
    diff_exclude: vec!["*pnpm-lock.yaml".to_string()],
})?;
let prompt = "Make the test pass";
let transcript = Transcript::from_stream(agent.run(Task::new(prompt))).await?;

assert!(transcript.tool_called("bash"));

Development

cargo test -p aether-evals