Crate aether_evals

Structs§

AgentConfig: Configuration for running an agent on a specific task
AgentSpec
DiffStats
DockerAgent
DockerImage
DockerImageParseError
EvalFilesReport: Outcome of running a collection of eval files.
EvalOutcome
EvalSpec: A JSON serializable representation of a single eval.
EvalSpecLoadOptions: Options for discovering and loading resolved eval specs.
EvalToolCall
Expect: Expectations checked against the agent’s run. All are optional; an empty expect passes as long as the agent runs to completion.
FakeAgent
GitDiff
GitRepoSpec
JudgeCriterionSummary
JudgeSpec: An LLM-as-judge rubric for an eval.
JudgeSummary
ResolvedEvalSpec: An EvalSpec resolved against its base directory.
RetainedWorkspaceInfo
Task
TaskRun
TaskRunError
TaskSpec
ToolCall
Transcript
Workspace

DockerSpec: Docker image configuration for an eval sandbox: either a prebuilt image, or a Dockerfile file to build (optionally tagged with image).
EvalFileError: Eval-file setup errors. Setup — loading, image resolution and builds, settings — happens before any eval runs and aborts the collection. Faults inside an individual eval (workspace setup, the agent run, the judge call) do not abort a collection run; they are recorded on that eval’s crate::spec::EvalOutcome instead.
EvalRunError
GitRepoError
ImageBuildError
RunError: Errors that can occur when running an agent
ToolCallExpectation
WorkspaceError
WorkspaceRetention: Whether the eval workspace is retained on disk after the run.
WorkspaceSource
WorkspaceSpec: The starting workspace for an eval. Omitted means an empty temporary directory.

AETHER_EVAL_CWD_ENV
AETHER_EVAL_WORKSPACE_ROOT_ENV
AETHER_EVAL_WRAPPED_TASK_PROMPT_ENV: Environment variable through which DockerAgent hands the wrapped task prompt to the in-container command.
CONTAINER_AETHER_HOME

assert_tool_call_count
assert_tool_call_with_args
assert_tool_called
default_eval_env_vars
run_eval_specs: Run resolved eval specs. Agents and judge models are assembled before anything runs, so a broken case fails fast. Docker image builds are deduplicated and run concurrently before eval execution. Outcomes are returned in input order.