Module evaluate 

Offline calibration evaluation for --score predictions.

--score already writes one JSONL line per row with the full probability vector. This module turns those files (plus a gold file) into the numbers that say whether the probabilities are usable — not only whether the answers are right: accuracy, balanced accuracy, NLL, Brier (both flavours), ECE and the reliability curve.

The bridge itself performs no calibration and recommends no thresholds; this module only measures. A deployment that wants to act on the probabilities can validate them on its own workload first.

Definitions (frozen)

• predicted slot = argmax of the probability vector, ties broken by the first slot in the row’s declared option order (the same first-max rule the wire layer reports its answers with). Rows without a vector fall back to their label.
• accuracy = correct / predicted rows.
• balanced accuracy = mean of per-class recall over the classes that occur as gold (macro-recall; classes with no gold row are excluded).
• NLL = mean of -ln(max(p_gold, 1e-12)) over scored rows.
• brier_multiclass = mean of Σ_k (p_k - 1[gold == slot_k])² over scored rows; defined for any number of slots.
• brier_binary = mean of (p_positive - 1[gold == positive])² over the scored rows that declare a positive class. n_binary says how many rows that was. A two-slot true/false row defaults positive to true (the noul contract), so the binarisation is never arbitrary.
• confidence = max(p), the probability mass behind the predicted slot.
• ECE = Σ_b (n_b / N) · |acc_b − conf_b| over bins equal-width bins in [0,1]; bin index = min(bins-1, floor(conf · bins)), so conf == 1.0 lands in the last bin rather than overflowing. N is the number of scored rows.
• reliability curve = the per-bin (lower, upper, n, mean_confidence, accuracy) tuples ECE is summed from. Empty bins are omitted.

Rows that carry no probability vector are counted in n_items/n_missing and contribute to accuracy, but they are excluded from NLL / Brier / ECE — n_scored names exactly how many rows did contribute, so coverage is never hidden.

Every metric is computed per stratum (the gold file’s family, with rows that declare none reported under unlabelled), and the report also carries the overall stratum. Strata are never silently pooled into one number by this module; a caller that wants an aggregate has to say so itself.

Verification

verify recomputes a written report from the same inputs and compares it field by field: the report is a claim, the prediction and gold files are the evidence. The tolerance belongs to the caller (--tol) and is never read out of the report, so a rewritten number cannot be laundered by declaring a looser tolerance.

Structs

Diff

One disagreement between a report and a fresh recomputation.

GoldLine

One gold line: the right answer for one row.

Metrics

Every number computed for one stratum.

PredictionLine

One prediction line, as --score writes it.

ReliabilityBin

One bin of the reliability curve.

Report

A written evaluation report.

Sample

One joined item reduced to exactly the quantities the metrics need.

VerifyOutcome

The result of a verification run.

Constants

DEFAULT_BINS

Default number of equal-width confidence bins for ECE and the reliability curve.

NLL_PROBABILITY_FLOOR

Probability floor for NLL: keeps -ln(0) from becoming +inf.

REPORT_SCHEMA

The schema identifier every report carries.

UNLABELLED_FAMILY

Stratification key for gold rows that declare no family.

Functions

evaluate

Evaluate predictions against gold: join, validate, and summarise every stratum plus the overall one.

join

Join predictions with gold by id, validating both sides.

load_gold

Parse a gold JSONL document (one GoldLine per line).

load_predictions

Parse a predictions JSONL document (one PredictionLine per line).

render_table

A short human-readable summary, for the CLI.

summarize

Reduce one stratum’s samples to Metrics.

verify

Recompute nothing: compare a claimed report against a freshly computed one, field by field.