Expand description
HTTP API contract for the Evaluate service.
This service provides a programmatic interface for running agent evaluation cases, collecting metrics, and producing reports. The eval harness replays predefined conversation turns against an agent runtime instance and judges outcomes against oracles.
§Layering
agent-evaluate-contract paths + DTOs (this crate)
agent-evaluate-runtime eval runner + judge
agent-evaluate-server router + auth + binStructs§
- ApiError
- Error body shared by every non-2xx response.
- Assertion
Verdict - Cancel
RunRequest - Case
Outcome Diff - Case
RunPage - Case
RunSummary - Claim
Review Request - Compare
Reports Request - Complete
Review Request - Create
Eval RunRequest - Enqueue
Review Request - Eval
Access Scope - Eval
Budget - Eval
Case - Eval
Case File - Eval
Case Result - Eval
Case Summary - Eval
Operation - Eval
Oracle - Eval
Report - Eval
Suite Version - Eval
Turn - Evidence
Ref - Immutable evidence object owned by the scoped Evaluate namespace.
- Experiment
Variant - GetCase
Request - GetOperation
Request - GetReport
Request - Human
Review - List
Case Runs Request - List
Cases Request - List
Cases Response - Load
Cases Request - Oracle
Verdict - Outcome
Assertion Spec - Strictly parsed compatibility schema for
outcomeAssertions. - Plugin
Evaluation Evidence - Register
Suite Request - Report
Comparison - RunEval
Request - Sandbox
Execution Evidence - Bounded evidence returned by the isolated executor. Verdict calculation remains owned by Evaluate and is deterministic from this typed structure.
- Sandbox
Execution Request - Turn
Assertion Spec - Verify
Command - Verify
Command Result - Versioned
Eval Case - Workspace
File Assertion Spec
Enums§
- Case
Failure Kind - Case
RunState - Operation
State - Review
State - Typed
Assertion - Versioned typed assertion schema used by durable suites. The legacy
Valuefields onEvalCaseremain only for v1 wire compatibility.
Constants§
- AUDIENCE
- CANCEL_
RUN_ PATH - CLAIM_
REVIEW_ PATH - COMPARE_
REPORTS_ PATH - COMPLETE_
REVIEW_ PATH - DEFAULT_
PORT - DURABLE_
REPORT_ PATH - GET_
CASE_ PATH - GET_
REPORT_ PATH - HEALTH_
PATH - LIST_
CASES_ PATH - LOAD_
CASES_ PATH - MAX_
RESOURCE_ ID_ CHARS - OPERATIONS_
PATH - REVIEW_
QUEUE_ PATH - RUNS_
PATH - RUN_
CASES_ PATH - RUN_
EVAL_ PATH - SERVICE_
NAME - SERVICE_
TOKEN_ ENV - SUITES_
PATH