Module evaluate 

Scoring a rubric against states someone has already labelled.

One call tells you what the model said about one state; that is what jev run is for. It does not tell you where to put the threshold, or how confident a choice has to be before a script may act on it — the README leaves both calls to you, and this is the module that turns them into a table. Feed it a page and a file of labelled states and it reports what the rubric got right, at every threshold worth trying.

Everything here is pure: cases come in as text, the answers come from a function you pass, and the report goes out as lines and JSON. Reading files, hashing request bodies and talking to the API belong to the caller.

Structs

Best

The threshold that scored best, and what it scored.

Case

One labelled state: what to judge, and what the rubric should say about it.

CaseError

A case that never produced a full set of answers, and why.

GateRow

One cut of the confidence gate: how much of the set survives it, and how right it is.

Preflight

What a whole run would send, before any of it is sent.

Report

Everything the run found out, with the numbers unrounded.

ReportOptions

What the run was asked for, which the report repeats back.

ReportUsage

The tokens the run spent, counted when the API counted them and estimated when it did not.

SweepRow

One row of a noul’s threshold sweep: the confusion counts, and what they come to.

Enums

Expectation

What one question is expected to answer, in the shape its kind is scored in.

Outcome

What one case’s request came back as.

QuestionReport

What one question scored, in the numbers its kind is judged by.

Functions

below_bar

Questions whose accuracy is below bar, for –min-accuracy.

parse_cases

Parse JSON Lines into cases, checking every expectation against session.

preflight

The preflight estimate: tokens summed over every case, priced when rates are known.

report

Score the outcomes against the cases.

report_json

The JSON report, ready for to_string_pretty. Numbers keep their precision; what is undefined is null.

report_lines

The text report, as lines the terminal draws.

report_text

Styled lines as the plain text a pipe wants.

run

Send every case through ask, at most concurrency at a time; results are in case order.

two

Two decimals, rounding a tie away from zero the way the TypeScript port’s toFixed(2) does.

with_state

The session as one case sends it: the page’s questions, the case’s state.