Module ltt 

Learn-then-Test (LTT) threshold calibration — distribution-free, finite-sample risk control on the gate serving threshold (SPEC §10.1).

Method

Angelopoulos et al. (2021), “Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control”; Bates et al. (2021), “Testing for Outliers with Conformal P-values” (RCPS).

The guarantee: the returned threshold λ carries a finite-sample risk certificate — served-failure rate ≤ α at family-wise confidence 1 − δ — without assuming exchangeability or a parametric model of the gate’s error distribution.

Algorithm

1. Build candidate thresholds from the observed score grid in descending order (strictest first). One candidate per distinct observed score suffices — finer grids do not tighten the guarantee (only the binomial test size δ and sample n do).
2. For each λ compute: the served set (pairs with score ≥ λ), empirical risk = n_failures / n_served, and the exact-binomial p-value P(Bin(n_served, α) ≤ n_failures) testing H₀: risk(λ) > α. Reject H₀ (certify λ) when p-value ≤ δ.
3. Walk candidates in the fixed-sequence order. Stop at the first qualified candidate (n_served ≥ min_n) whose test fails — subsequent candidates are never certified, even if their empirical risk dips below α again. Candidates below min_n are skipped (not in the test sequence) and do not break the walk.
4. Return the last certified λ (least strict, maximum coverage), or INFINITY (serve nothing) when infeasible.

Why the fixed-sequence walk controls FWER at δ without Bonferroni

A false certification is a false rejection of a true null H₀: risk(λ) > α. In any fixed pre-specified test sequence the family-wise error rate (FWER = P(≥ 1 false rejection)) equals the probability that the first true H₀ encountered in the walk is falsely rejected, which is at most δ by the individual test level. The stopping rule is the key: once a test fails the sequence ends, so accumulating multiple false rejections in one run is impossible. Theorem 2 of Angelopoulos et al. (2021) formalises this for any fixed order, without a Bonferroni correction factor, provided each individual test is valid at level δ. The one-sided exact-binomial test used here is conservative (it never over-rejects under H₀), so the FWER guarantee holds even though all tests share the same calibration set.

Structs

LttDiagnostic

Per-λ diagnostic row emitted by calibrate.

LttResult

Result of LTT threshold calibration.

Functions

calibrate

Calibrate an LTT serving threshold.