Expand description
Evaluations. The judge-backed kinds pose a criterion in plain English and
ask the provider’s judge to score the transcript (a boolean assertion, or a
numeric score compared against a threshold). The deterministic kinds
(called / not_called) assert on the mock/spy channel’s observed tool
calls — no judge, no flakiness — referencing a mocks: declaration by name.
Structs§
- Eval
Outcome - The result of running one eval against a transcript.
Enums§
- Comparator
- How a numeric score is compared to its threshold.
- Eval
- An eval specification, as written in a test case’s YAML.
- Eval
Detail - The kind-specific detail of an eval outcome, for reporting.
- Judge
Value - The raw value a judge returns: either a boolean or a number, matching the
eval kind. Deserialized untagged from the provider’s
valuefield.