Structs§
- Container
- Container
Builder - Context
Usage - Context/token usage reported after an LLM call.
- Diff
Stats - Docker
Agent - Exec
Output - Fake
Agent - GitDiff
- GitRepo
Spec - Image
- Image
Parse Error - Judge
- A built judge: the assembled prompt plus the normalized rubric it grades against. Run it with
Judge::runagainst a model, or grade a parsedJudgeRubricResponsewithJudge::summarize. - Judge
Builder - Judge
Context - Evidence the judge grades against: the agent transcript, a workspace diff, and/or final files.
- Judge
Criterion Response - Judge
Criterion Spec - A single rubric criterion scored on a normalized 0.0..=1.0 scale.
- Judge
Criterion Summary - Judge
Rubric Response - The raw rubric response the judge model is expected to return.
- Judge
Summary - The graded result of running a judge: an overall pass/score plus per-criterion detail.
- Retained
Workspace Info - Task
- Tool
Call - Transcript
- Transcript
Error - Workspace
Enums§
- Container
Error - Eval
RunError - GitRepo
Error - Judge
Error - RunError
- Errors that can occur when running an agent
- Workspace
Error - Workspace
Source
Constants§
- AETHER_
EVAL_ CWD_ ENV - AETHER_
EVAL_ TASK_ PROMPT_ ENV - Environment variable through which
DockerAgenthands the caller-provided task prompt to the in-container command. - AETHER_
EVAL_ WORKSPACE_ ROOT_ ENV - CONTAINER_
AETHER_ HOME
Traits§
- Agent
- Trait for an agent under evaluation.
Functions§
- assert_
tool_ call_ count - assert_
tool_ call_ with_ args - assert_
tool_ called - default_
eval_ env_ vars - judge
- Start building an LLM-as-judge from structured context and rubric criteria.