Module evaluate 

Scoring a rubric against states someone has already labelled.

One call tells you what the model said about one state; that is what jev run is for. It does not tell you where to put the threshold, or how confident a choice has to be before a script may act on it — the README leaves both calls to you, and this is the module that turns them into a table. Feed it a page and a file of labelled states and it reports what the rubric got right, at every threshold worth trying.

Everything here is pure: cases come in as text, the answers come from a function you pass, and the report goes out as lines and JSON. Reading files, hashing request bodies and talking to the API belong to the caller.

Structs

Best

The threshold that scored best, and what it scored.

CalibratedQuestion

What calibration made of one question: the bar it found, or why it left the question alone.

Calibration

Case

One labelled state: what to judge, and what the rubric should say about it.

CaseError

A case that never produced a full set of answers, and why.

CompareOptions

What a comparison is read at, which both reports repeat back.

Comparison

Everything a comparison found, with the numbers unrounded.

Flip

A case the two pages answered differently.

GateRow

One cut of the confidence gate: how much of the set survives it, and how right it is.

Labelled

One page’s report, with the name it goes by.

Labels

What the two pages of a comparison are called in its messages and its report.

Latency

How early or late a noul notices, over the conversations labelled per turn.

Leg

One page’s share of a comparison: its session, its cases, and how to ask it.

McNemar

Metric

One metric on both sides, and what moved.

Mismatch

A name both pages use for questions of different kinds.

Preflight

What a whole run would send, before any of it is sent.

Report

Everything the run found out, with the numbers unrounded.

ReportOptions

What the run was asked for, which the report repeats back.

ReportUsage

The tokens the run spent, counted when the API counted them and estimated when it did not.

Shared

A question both pages ask the same way, measured on the cases both pages scored.

Side

One page’s run, as a comparison needs it.

SweepRow

One row of a noul’s threshold sweep: the confusion counts, and what they come to.

ThreadLatency

One conversation labelled per turn: the turn it should have said yes, and the turn it did.

Enums

ByTurn

A per-turn label: the turn a noul becomes true from, or never.

Expectation

What one question is expected to answer, in the shape its kind is scored in.

Outcome

What one case’s request came back as.

QuestionReport

What one question scored, in the numbers its kind is judged by.

Verdict

What the exact McNemar test made of the discordant pairs.

Constants

ALPHA

The level McNemar’s test is read at. Not a flag: a comparison should mean the same everywhere.

DEFAULT_TARGET

The accuracy a choice’s or score’s bar has to reach when --target-accuracy is not given.

MIN_DISCORDANT

Below this many discordant pairs no two-sided exact p can reach ALPHA: 2 / 2^5 > 0.05.

Functions

below_bar

Questions whose accuracy is below bar, for –min-accuracy.

calibrate

The bars a run supports, one per scored question.

calibration_cuts

The confidence bars calibration tries, k / 20 for k from 0 to 19: finer than the report’s gate, and computed by division so each prints as the short decimal it is.

calibration_json

The calibration as JSON, for the report’s calibration key.

calibration_lines

What calibration changed, as lines for under the report: one per question, the new directive as it now reads on the page, what it replaced, and the evidence for it.

calibration_text

The calibration as plain text, for under the report.

compare

Put two runs of the same cases side by side.

compare_json

The comparison as JSON, ready for to_string_pretty: both reports whole, and what moved between them.

compare_lines

The comparison as lines: a legend, a block per shared question, what could not be compared.

compare_text

The comparison as the plain text a pipe wants.

mcnemar

The exact McNemar test on the discordant pairs of a paired comparison.

not_calibrating

Why a run with errors writes nothing back: the bars would be fitted to the cases that worked.

parse_cases

Parse JSON Lines into cases, checking every expectation against session.

parse_compare_cases

Parse one cases file for two pages at once: a case per page, each holding the expectations for that page’s questions.

preflight

The preflight estimate: tokens summed over every case, priced when rates are known.

regressions

The questions b is significantly worse at, for --fail-on-regression.

report

Score the outcomes against the cases.

report_json

The JSON report, ready for to_string_pretty. Numbers keep their precision; what is undefined is null.

report_lines

The text report, as lines the terminal draws.

report_text

Styled lines as the plain text a pipe wants.

run

Send every case through ask, at most concurrency at a time; results are in case order.

run_compare

Both pages of a comparison through one pool of workers: page a’s cases first, then b’s.

signed

A change, signed either way, so a regression reads as one; -0.00 is no change, so +0.00.

three

Three decimals, the way toFixed(3) writes them; see two.

two

Two decimals, the way the TypeScript port’s toFixed(2) writes them.

with_state

The session as one case sends it: the page’s questions, the case’s state.