Module bench 

Offline scoring and explicit, bounded benchmark execution.

A benchmark runner invokes its provider: default warm-up adds one invocation. Provider failures abort; they are never folded into abstention denominators.

Structs

Case

One labeled input; omitted question labels do not contribute to metrics.

FlowMiss

One wrong completed flow result.

FlowReport

Scored flow outcomes with their traces, expected results and latencies.

Miss

A wrong decided answer; abstentions are not misses.

QuestionStats

Counts and probabilities for the labeled subset of one question.

Report

A scored batch, retaining answers, decisions and observed latency.

RunOptions

Explicit benchmark work budget settings. Warm-up defaults to one extra call.

Enums

Decision

One discrete non-abstention label. Scores use zero-based rubric indices.

Functions

at_min_confidence

Apply a confidence threshold without provider calls or restoring old abstentions.

decide

Reduce a normalized answer to a label. Score rounding is half up. Box is unsupported even when its answer is undecided.

ndcg

Discounted cumulative gain at k; missing IDs have zero grade. Returns None when ideal gain is zero. Grades and intermediate gains must be finite.

ndcg_default

NDCG with the Python default cutoff of ten.

run

Invoke a provider for all cases, aborting on failure and dropping pending work. Validate labels and request assets before any warm-up or measured call.

run_flow

Run caller-supplied flow futures sequentially, retaining traces and aborting on failure. There is no automatic flow warm-up or retry. Inputs are borrowed by each future.

score

Score already-collected answers. Label subsets and empty case lists are valid. Answer and latency lengths must exactly match cases, and answer keys must match questions.

score_flows

Score existing flow outcomes. Inconsistent completion/value states and failed traces are rejected.

stability

Compare decisions across aligned nonempty reports, including identical abstentions. One report is perfectly stable; stability does not measure correctness.