# Aether Evals
Aether Evals is a Rust library that makes writing agent evals easier to express as ordinary `cargo-nextest` tests.
## Table of Contents
- [Quick Start](#quick-start)
- [Test organization](#test-organization)
- [Core API](#core-api)
- [Dockerized evals](#dockerized-evals)
- [Assertions](#assertions)
- [Debugging failed evals](#debugging-failed-evals)
- [Git-backed evals](#git-backed-evals)
- [Development](#development)
## Quick Start
```rust
use aether_evals::{Agent, FakeAgent, Task, Transcript, Workspace};
#[tokio::test]
async fn hello_world_test() -> Result<(), aether_evals::WorkspaceError> {
let workspace = Workspace::empty()?;
let prompt = "Write 'Hello, World!' to hello.txt";
let agent = FakeAgent::writes_file("hello.txt", "Hello, World!").with_workspace(workspace.path());
let transcript = Transcript::from_stream(agent.run(Task::new(prompt))).await.unwrap();
assert!(!transcript.events().is_empty());
assert!(workspace.join("hello.txt").exists());
Ok(())
}
```
Run evals with nextest:
```bash
cargo nextest run -p aether-evals --all-features
cargo nextest run --profile ci --all-features --workspace
```
The CI profile writes JUnit XML to `target/nextest/ci/junit.xml`.
## Test organization
Aether Evals' fake-agent coverage should use normal test names. Reserve `_eval` suffixes for real provider-backed evals selected by the repository `evals` nextest group.
## Core API
- `Task::new(prompt)` describes the problem to solve.
- `Agent::run(task)` streams `AgentEvent`s from an agent's configured execution environment.
- `Transcript::from_stream(agent.run(task))` collects the stream into a `Transcript`.
- `Transcript::default()` plus `transcript.add(message)` supports observing each streamed message while building the transcript manually.
- `Container::builder(image).start(&workspace)` starts a caller-owned Docker container with the workspace mounted at `/workspace`.
- `DockerAgent::new(container, command)` runs an in-container command whose stdout is newline-delimited `AgentEvent` JSON.
- `Container::exec_shell(script)` runs follow-up assertions or test commands in the same caller-owned container.
- `eval_env_vars(&model)` forwards only the credentials for the providers `model` runs on into Docker, and sets `AETHER_HOME=/root/.aether`.
- `Workspace::empty()` creates an isolated temp directory.
- `Workspace::from_dir(path)` copies fixture directory contents into a temp directory.
- `Workspace::from_git(GitSpec { source, start_commit, gold_commit, subdir, branch, diff_exclude })` checks out the start commit on `branch` (default `main`), fetched from `GitSource::Url(url)` or `GitSource::Bundle(path)` (see `create_git_bundle`). The workspace holds only the start commit's history, with no remote, so the agent can't find the gold commit in git. That history is fetched in full, blobs included, so reuse one bundle across runs for large repositories.
- `Transcript` exposes collected agent messages and tool-call helpers. `transcript.session_usage()` returns the session's cumulative tokens and estimated cost, sub-agents' calls included, from the last usage event.
## Dockerized evals
Aether Evals mounts the workspace root at `/workspace` and runs the command from the effective eval cwd. It does not wrap or alter prompts; the command receives the caller-provided prompt verbatim:
- `AETHER_EVAL_TASK_PROMPT` — the caller-provided prompt to send to the agent unchanged.
- `AETHER_EVAL_WORKSPACE_ROOT=/workspace` — the mounted workspace root.
- `AETHER_EVAL_CWD` — the command's effective cwd inside the workspace.
Stdout is reserved for `AgentEvent` JSON lines and must end with a terminal event such as `{"category":"turn","event":{"type":"ended","outcome":{"status":"completed"}}}`. Diagnostic logging should go to stderr.
```rust
use aether_evals::{
Agent, CONTAINER_AETHER_HOME, Container, DockerAgent, GitSource, GitSpec, Image, Task, Transcript, Workspace,
eval_env_vars,
};
let workspace = Workspace::from_git(GitSpec {
source: GitSource::Url("https://github.com/example/repo".to_string()),
start_commit: "start_sha".to_string(),
gold_commit: "gold_sha".to_string(),
subdir: None,
branch: None,
diff_exclude: Vec::new(),
})?;
let container = Container::builder(Image::new("aether-sandbox", "latest"))
.with_env_vars(eval_env_vars(&"anthropic:claude-opus-4-6".parse()?))
.with_ephemeral_mount(CONTAINER_AETHER_HOME)
.start(&workspace)
.await?;
let agent = DockerAgent::new(container.clone(), vec!["/usr/local/bin/aether-eval-agent".to_string()]);
let prompt = "fix the failing test";
let transcript = Transcript::from_stream(agent.run(Task::new(prompt))).await?;
let tests = container.exec_shell("cargo test").await?;
assert_eq!(tests.exit_code, 0, "stdout:\n{}\nstderr:\n{}", tests.stdout, tests.stderr);
assert!(!transcript.events().is_empty());
```
Run follow-up `container.exec_shell(...)` assertions before dropping the caller-owned container and workspace.
A wrapper script for evaluating Aether itself can delegate to headless JSON output:
```sh
#!/bin/sh
exec aether headless \
--output json \
"$AETHER_EVAL_TASK_PROMPT"
```
A declarative eval:
```json
{
"agent": {
"command": ["node", "/app/dist/eval-agent.js"]
}
}
```
## Assertions
Use normal Rust assertions over the returned `Transcript` and files on disk:
```rust
let workspace = Workspace::empty()?;
let transcript = Transcript::from_stream(agent.run(Task::new(prompt))).await?;
assert!(transcript.tool_called("write_file"));
assert_eq!(transcript.tool_call_count("bash"), 1);
assert_eq!(std::fs::read_to_string(workspace.join("output.txt"))?, "expected text\n");
```
Aether Evals also exports small `#[track_caller]` helpers for common tool assertions:
```rust
aether_evals::assert_tool_called(&transcript, "write_file");
aether_evals::assert_tool_call_count(&transcript, "bash", 1);
aether_evals::assert_tool_call_with_args(&transcript, "write_file", &serde_json::json!({ "path": "output.txt" }));
```
LLM judging runs through declarative eval files: add a rubric `judge` block to `expect` in a `*.eval.json` file. Each criterion is scored 0.0–1.0 by the judge model, weighted into an overall score, and blocking criteria below their threshold fail the eval. See the [evals documentation](https://aether-agent.io/aether/running/evals/) for the file format.
Judge faults — transient LLM stream failures, unparseable responses, criterion sets that don't match the rubric — are recorded as failures on the eval's outcome rather than conflated with a "the agent didn't do it" verdict.
## Debugging failed evals
`TranscriptError` preserves the partial transcript if an agent stream returns an error. Use `error.transcript().events()` or `error.into_parts()` to inspect events that were emitted before the failure.
`Workspace` removes its temporary directory when dropped. To retain files for debugging, call `workspace.persist()` and remove the returned path manually when finished.
For git-backed workspaces, `workspace.capture_agent_diff()?` returns the agent's commits, staged and unstaged edits, and untracked files since the start commit, and errors if git can't read the repository. `workspace.reference_diff()` returns the start-to-gold diff, computed before the gold commit was discarded. Both are `None` for workspaces without a start commit, and both leave out `diff_exclude` pathspecs such as `*pnpm-lock.yaml`.
## Git-backed evals
```rust
use aether_evals::{Agent, GitSource, GitSpec, Task, Transcript, Workspace};
let workspace = Workspace::from_git(GitSpec {
source: GitSource::Bundle("fixtures/repo.bundle".into()),
start_commit: "start_sha".to_string(),
gold_commit: "gold_sha".to_string(),
subdir: Some("packages/api".into()),
branch: None,
diff_exclude: vec!["*pnpm-lock.yaml".to_string()],
})?;
let prompt = "Make the test pass";
let transcript = Transcript::from_stream(agent.run(Task::new(prompt))).await?;
assert!(transcript.tool_called("bash"));
```
## Development
```bash
cargo test -p aether-evals
```