Expand description
Eval suite + fixtures (#233): the deterministic task definitions an A/B run scores.
A suite is an NDJSON file (one Task per line, #-comments + blank lines allowed).
Each task carries everything the harness needs to (a) assemble context from a workspace,
(b) prompt the pinned model, and (c) score the answer objectively. Two domains are
supported today: free-form Domain::Qa (scored with EM / F1 / containment) and
Domain::Code (scored by running a unit-test command against the model output).
Structs§
- Eval
Suite - A loaded, validated suite: the tasks plus the directory used to resolve relative workspaces.
- Task
- One scored unit of work. Fixtures are stored as NDJSON (one task per line) so suites are diff-friendly and stream without loading the whole file into a single JSON value.
Enums§
- Domain
- What kind of task this is — selects the scorer and how the model output is interpreted.