Module classifier_eval 

Deterministic, structured label-match eval for classifier agents. Pure scoring — no I/O, no provider, no agent driver. A classifier’s output is a closed label set, so exact-label-match gates the actual decision while tolerating the prose a real LLM wraps it in.

Structs

ClassificationCase

One golden row. expected must be the canonical form of a LabelSpec the scorer is given, or the case can never be counted correct.

ClassificationReport

Aggregate result over a golden set.

LabelSpec

A canonical label plus the surface forms that resolve to it.

LabelStat

Per-label tally (named fields — a 3-tuple would be an opaque API surface).

Miss

One missed case: expected label vs what parsed out (None = output did not resolve to exactly one known label).

Functions

extract_label

Resolve a raw agent output to a known label. Scans for each spec’s canonical form and aliases as whole tokens, case-insensitively, and returns Some(canonical) iff EXACTLY ONE distinct known label appears. Zero matches or an ambiguous multi-label output → None (a miss). This separates a real misclassification from format noise (prose wrapping, alias rendering).

score_classification

Score predictions against the golden cases (aligned by index). A prediction of None, or one not equal to the case’s expected, is a miss. A prediction missing for a case (short slice) counts as None; predictions beyond cases.len() are ignored — the report is defined over the golden set.