Expand description
Scoring one expectation against a run’s result, behind the Scorer trait — plus the
batteries-included BuiltinScorer that needs NO host scoring code at all.
The host implements Scorer over the SAME World its Harness produces. For
each of a case’s expect predicates the runner calls Scorer::score, collecting the returned
(label, passed) pairs into the case’s CaseOutcome::predicates. A
case passes iff the run succeeded AND every predicate’s passed is true.
score now also receives the run’s RunArtifacts, so a scorer can assert on
what the agent DID (tool calls, params, final text) and not only on the post-run world. The common
case — “score the built-in Expectations over the artifacts, ignoring
the world” — is BuiltinScorer, which a host gets for free via run_suite.
Structs§
- Builtin
Scorer - The batteries-included scorer: evaluates the built-in
Expectationassertions over a run’sRunArtifacts, ignoring the world entirely (World = ()).
Traits§
- Scorer
- Scores one expectation against the world AND artifacts a case produced. The host implements this for a
custom predicate/world; for the built-in assertions use
BuiltinScorer.