firstpass_core/eprocess.rs
1//! Anytime-valid risk control on the gate threshold — a bound that holds at **every round**
2//! (SPEC §10.1, the online regime).
3//!
4//! ## Why this exists when [`crate::conformal`] and [`crate::ltt`] already do risk control
5//!
6//! The three regimes already shipped are each valid for what they claim, and none of them covers
7//! the way an operator actually runs the proxy:
8//!
9//! | existing | guarantee | why it is not enough here |
10//! |---|---|---|
11//! | [`crate::conformal::calibrate`] | served-failure ≤ `alpha` at confidence `1 − delta` | assumes **exchangeability**, and fixes `λ` once |
12//! | [`crate::ltt`] | FWER-controlled, exact-binomial, no exchangeability needed | **fixed-sample**: one calibration set, one decision |
13//! | [`crate::conformal::AdaptiveConformal`] | Gibbs–Candès ACI | keeps only the **long-run average** at `alpha` |
14//!
15//! Two concrete holes follow, and both bite in production:
16//!
17//! 1. **A long-run average is not a promise about now.** A process can sit above `alpha` for an
18//! arbitrarily long stretch and still converge, because the excess is amortised over a horizon
19//! the operator may never reach. Someone holding an error budget needs the bound to hold at the
20//! round they are on.
21//! 2. **Repeated recalibration breaks fixed-sample guarantees.** LTT controls the family-wise error
22//! rate over *one* pre-specified test sequence on *one* calibration set. Re-run it whenever more
23//! feedback lands and adopt the winning threshold each time — which is exactly what a live
24//! recalibration loop does — and that is optional stopping on a growing stream. The guarantee
25//! does not compose across the repeats, and the realized risk can exceed `alpha` without any
26//! single run of LTT ever being wrong.
27//!
28//! ## Method
29//!
30//! Khosravi & Huo (2026), *Conformal Selective Acting: Anytime-Valid Risk Control for RLVR-Trained
31//! LLMs* ([arXiv:2605.20270](https://arxiv.org/abs/2605.20270)).
32//!
33//! Maintain one **e-process** per candidate threshold on a fixed grid. An e-process is a
34//! non-negative supermartingale under the null, so **Ville's inequality** bounds it at *all*
35//! stopping times simultaneously:
36//!
37//! ```text
38//! P( ∃t : E_t ≥ 1/delta ) ≤ delta under H₀: risk(λ) > alpha
39//! ```
40//!
41//! That single quantifier — `∃t`, rather than a fixed `t` — is the whole point, and it is what
42//! makes the bound survive optional stopping. A threshold is **certified** once its e-value crosses
43//! `1/delta`, and certification is permanent: it was earned against every round up to that point,
44//! so later data cannot retroactively invalidate it (it can, and does, certify *more* thresholds).
45//!
46//! Per round, for a served item with gate score `s` and observed correctness `c`, each threshold
47//! `λ ≤ s` (i.e. each threshold that *would have served* this item) is updated by the betting factor
48//!
49//! ```text
50//! E_λ ← E_λ · (1 + lambda_bet · (alpha − err)) where err = 1 if incorrect else 0
51//! ```
52//!
53//! Under H₀ the expected multiplier is `≤ 1`, which is precisely the supermartingale property.
54//! Thresholds that never serve are never updated, so an unused threshold neither gains nor loses
55//! evidence — a nuance the fixed-sample methods get for free but an online one has to be careful
56//! about.
57//!
58//! The grid is a **Bonferroni** family: with `n` candidate thresholds each tested at `delta / n`,
59//! the probability that *any* threshold is falsely certified at *any* round stays `≤ delta`. The
60//! served threshold is the **least conservative certified** one — the smallest `λ` whose e-process
61//! has crossed — which serves as much traffic as the evidence allows.
62//!
63//! ## The honest cost
64//!
65//! Anytime validity is not free. Against [`crate::conformal::AdaptiveConformal`] at the same
66//! `alpha` this serves **less** traffic, because it will not lower the threshold until the evidence
67//! has actually accumulated, and the Bonferroni correction makes each individual threshold harder
68//! to certify. That is the trade being offered: ACI tracks a shifting target aggressively and
69//! promises an average; this one promises every round and pays for it in conservatism. An operator
70//! picks the regime; neither is strictly better, and the ADR says so.
71//!
72//! Before any threshold is certified there is no guarantee to serve on, so
73//! [`EProcessRiskControl::certified_threshold`] returns `None` and the caller keeps its existing
74//! behaviour. Failing closed is the only safe reading of "I have not proven anything yet".
75
76use serde::{Deserialize, Serialize};
77
78/// Outcome of a certification query: the served threshold and the evidence behind it.
79#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
80pub struct Certification {
81 /// The least conservative certified threshold — serve iff `score >= threshold`.
82 pub threshold: f64,
83 /// The e-value backing it. Always `>= 1/delta` for a certified threshold.
84 pub e_value: f64,
85 /// Rounds observed when this certification was first reached.
86 pub certified_at_round: u64,
87}
88
89/// Default betting fraction. Conservative on purpose: a large bet grows the e-process fast when the
90/// null is false but decays it just as fast when it is true, and a decayed e-process takes a long
91/// time to recover. `0.5` is the standard "half-Kelly" compromise.
92pub const DEFAULT_BET: f64 = 0.5;
93
94/// Default e-value ceiling, as a multiple of the crossing level.
95///
96/// Truncating a supermartingale from above leaves it a supermartingale, so the type-I guarantee is
97/// unaffected — this is purely about **responsiveness**. Without a cap, a long clean stretch banks
98/// unbounded surplus evidence, and a subsequent regime change must burn all of it off before the
99/// threshold de-certifies; the bound holds asymptotically while realized failure sits above `alpha`
100/// for thousands of rounds. Capping bounds that de-certification lag. `10.0` keeps roughly an order
101/// of magnitude of headroom — enough that ordinary noise does not flap the certification, small
102/// enough that a genuine shift is caught in tens of rounds rather than thousands.
103pub const DEFAULT_CAP: f64 = 10.0;
104
105/// Anytime-valid risk controller over a fixed grid of candidate thresholds.
106///
107/// Feed it the deferred-feedback stream via [`EProcessRiskControl::observe_served`] and read
108/// [`EProcessRiskControl::certified_threshold`] on the router hot path. Unlike
109/// [`crate::conformal::AdaptiveConformal`], the answer is valid at the round you read it.
110#[derive(Debug, Clone, Serialize, Deserialize)]
111pub struct EProcessRiskControl {
112 /// Target served-failure rate.
113 alpha: f64,
114 /// Family-wise error budget across the whole grid and all rounds.
115 delta: f64,
116 /// Betting fraction in `(0, 1]`.
117 bet: f64,
118 /// E-values are capped at `cap * crossing_level`, bounding how much surplus evidence a clean
119 /// stretch can bank against a later regime change. See [`DEFAULT_CAP`].
120 cap: f64,
121 /// Candidate thresholds, ascending.
122 grid: Vec<f64>,
123 /// Running e-value per grid point, parallel to `grid`. Starts at `1.0` (no evidence).
124 e_values: Vec<f64>,
125 /// Round at which each grid point first crossed, if it has.
126 certified_at: Vec<Option<u64>>,
127 /// Rounds observed.
128 rounds: u64,
129 /// Served count and failures, for diagnostics only — never part of the guarantee.
130 served: u64,
131 served_fails: u64,
132}
133
134impl EProcessRiskControl {
135 /// Build a controller over `grid` candidate thresholds.
136 ///
137 /// `alpha` is the target served-failure rate, `delta` the family-wise error budget (the bound
138 /// holds with probability `1 − delta` across every threshold and every round jointly).
139 ///
140 /// The grid is sorted and de-duplicated; an empty grid yields a controller that never certifies,
141 /// which is the correct degenerate behaviour rather than an error — the caller then simply keeps
142 /// its existing threshold.
143 #[must_use]
144 pub fn new(alpha: f64, delta: f64, bet: f64, grid: &[f64]) -> Self {
145 Self::with_cap(alpha, delta, bet, DEFAULT_CAP, grid)
146 }
147
148 /// As [`EProcessRiskControl::new`], with an explicit e-value ceiling. See [`DEFAULT_CAP`] for
149 /// why a ceiling exists and what it trades.
150 #[must_use]
151 pub fn with_cap(alpha: f64, delta: f64, bet: f64, cap: f64, grid: &[f64]) -> Self {
152 // The betting factor on a failure is `1 + bet*(alpha - 1)`, which reaches zero at
153 // `bet = 1/(1 - alpha)` and goes negative beyond it. A zero factor annihilates the e-value
154 // permanently: no amount of subsequent evidence can revive a product that has hit exactly
155 // 0, so a single failure would silently retire that threshold forever. Clamping the bet
156 // just below the annihilating value keeps the process a strict supermartingale (which is
157 // what Ville's inequality needs) while leaving it able to recover.
158 //
159 // Flagged in review. Reachable through the public `with_cap` and by config, so it is a
160 // real input to validate rather than a theoretical corner.
161 // Bound the failure factor at 0.5 rather than merely above 0. Clamping at `0.99/(1-alpha)`
162 // technically avoids annihilation but leaves a factor of 0.01 — a single failure costs 100x
163 // and takes ~50 consecutive successes to undo, which is dead in practice if not in theory.
164 // The first version of this clamp did exactly that, and the recovery half of the test below
165 // failed: the controller never certified again. Half per failure is recoverable and still
166 // strictly decreasing, which is all the supermartingale property needs.
167 let max_bet = 0.5 / (1.0 - alpha).max(f64::EPSILON);
168 let bet = if bet.is_finite() {
169 bet.clamp(0.0, max_bet)
170 } else {
171 DEFAULT_BET.min(max_bet)
172 };
173 let mut grid: Vec<f64> = grid.iter().copied().filter(|g| g.is_finite()).collect();
174 grid.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
175 grid.dedup();
176 let n = grid.len();
177 Self {
178 alpha,
179 delta,
180 bet,
181 cap,
182 grid,
183 e_values: vec![1.0; n],
184 certified_at: vec![None; n],
185 rounds: 0,
186 served: 0,
187 served_fails: 0,
188 }
189 }
190
191 /// The per-threshold crossing level under Bonferroni: `n / delta`.
192 ///
193 /// Splitting `delta` across the `n` grid points is what keeps the *family-wise* error at
194 /// `delta`. Testing every threshold at the full `delta` would inflate the false-certification
195 /// rate with grid size — the classic multiple-testing trap, and an easy one to fall into here
196 /// because each individual e-process looks perfectly valid on its own.
197 #[must_use]
198 pub fn crossing_level(&self) -> f64 {
199 if self.grid.is_empty() || self.delta <= 0.0 {
200 return f64::INFINITY;
201 }
202 self.grid.len() as f64 / self.delta
203 }
204
205 /// Record one served item: its gate `score`, and whether it turned out correct.
206 ///
207 /// Only thresholds that *would have served* this item (`λ <= score`) are updated — a threshold
208 /// strict enough to have escalated instead has learned nothing about its own risk from it.
209 pub fn observe_served(&mut self, score: f64, was_correct: bool) {
210 self.rounds += 1;
211 self.served += 1;
212 self.served_fails += u64::from(!was_correct);
213
214 let err = f64::from(!was_correct);
215 let level = self.crossing_level();
216
217 for i in 0..self.grid.len() {
218 if self.grid[i] > score {
219 continue;
220 }
221 // Betting factor. Under H₀ (true risk > alpha) the expectation is <= 1, which is the
222 // supermartingale property Ville's inequality needs. Clamped at zero because a negative
223 // e-value is meaningless and would flip the comparison below.
224 let factor = (1.0 + self.bet * (self.alpha - err)).max(0.0);
225 // Capped. Ville's inequality is indifferent to the cap (truncating a supermartingale
226 // from above leaves it a supermartingale, so type-I control is untouched), but the
227 // operational difference is the whole ballgame: an uncapped e-value compounds through a
228 // long clean stretch to something astronomically above the level, and then a regime
229 // change has to burn through all of it before the threshold de-certifies. The bound
230 // would still hold in the limit while the realized rate sat above alpha for thousands
231 // of rounds — which is precisely the "long-run average" failure this module rejects.
232 self.e_values[i] = (self.e_values[i] * factor).min(self.cap * level);
233 if self.certified_at[i].is_none() && self.e_values[i] >= level {
234 self.certified_at[i] = Some(self.rounds);
235 }
236 }
237 }
238
239 /// The least conservative certified threshold, or `None` if nothing is certified yet.
240 ///
241 /// `None` means "no threshold has earned a guarantee", and the caller must keep its existing
242 /// behaviour rather than invent one. Serving on an uncertified threshold would be exactly the
243 /// unproven-claim failure this module exists to prevent.
244 /// Certification is evaluated **live**, against the current e-value — it is not a latch.
245 ///
246 /// This is the subtle part, and getting it wrong silently reintroduces the very failure mode the
247 /// module exists to prevent. A threshold that crossed during a clean stretch and has since been
248 /// contradicted by evidence is *not* certified any more, and must stop serving. Latching it —
249 /// "it was proven once, so it stays proven" — is what a fixed-sample method does, and under a
250 /// regime change it keeps serving at a threshold the current regime does not justify. The first
251 /// draft of this module latched, and `coverage_holds_at_every_round_under_a_regime_change`
252 /// caught it: realized failure hit 0.312 against alpha=0.20.
253 ///
254 /// `certified_at_round` is retained as provenance — when the evidence *first* crossed — which is
255 /// useful in a receipt and harmless here, because the serving decision reads the e-value.
256 #[must_use]
257 pub fn certified_threshold(&self) -> Option<Certification> {
258 let level = self.crossing_level();
259 self.grid
260 .iter()
261 .enumerate()
262 .find(|(i, _)| self.e_values[*i] >= level)
263 .map(|(i, &threshold)| Certification {
264 threshold,
265 e_value: self.e_values[i],
266 certified_at_round: self.certified_at[i].unwrap_or(self.rounds),
267 })
268 }
269
270 /// Whether `score` clears the certified threshold. `false` while nothing is certified.
271 #[must_use]
272 pub fn should_serve(&self, score: f64) -> bool {
273 self.certified_threshold()
274 .is_some_and(|c| score >= c.threshold)
275 }
276
277 /// Realized served-failure rate so far — a running diagnostic, not the guarantee.
278 #[must_use]
279 pub fn realized_served_failure(&self) -> f64 {
280 if self.served == 0 {
281 0.0
282 } else {
283 self.served_fails as f64 / self.served as f64
284 }
285 }
286
287 /// Rounds observed.
288 #[must_use]
289 pub const fn rounds(&self) -> u64 {
290 self.rounds
291 }
292
293 /// Current e-value at each grid point, parallel to the grid — for telemetry and tests.
294 #[must_use]
295 pub fn e_values(&self) -> &[f64] {
296 &self.e_values
297 }
298
299 /// The candidate grid, ascending.
300 #[must_use]
301 pub fn grid(&self) -> &[f64] {
302 &self.grid
303 }
304
305 /// Target served-failure rate the certificate is stated at.
306 #[must_use]
307 pub const fn alpha(&self) -> f64 {
308 self.alpha
309 }
310
311 /// Family-wise error budget the certificate is stated at.
312 #[must_use]
313 pub const fn delta(&self) -> f64 {
314 self.delta
315 }
316}
317
318#[cfg(test)]
319mod tests {
320 use super::*;
321
322 /// Deterministic uniform stream in `[0,1)`. A seeded LCG rather than `rand`, so the evidence
323 /// these tests produce is reproducible by an auditor on any machine, which is the same reason
324 /// the rest of the crate avoids ambient randomness.
325 struct Lcg(u64);
326 impl Lcg {
327 fn next_f64(&mut self) -> f64 {
328 self.0 = self
329 .0
330 .wrapping_mul(6_364_136_223_846_793_005)
331 .wrapping_add(1);
332 ((self.0 >> 11) as f64) / ((1u64 << 53) as f64)
333 }
334 }
335
336 fn grid() -> Vec<f64> {
337 (0..=20).map(|i| f64::from(i) / 20.0).collect()
338 }
339
340 /// A well-behaved gate: high score => usually correct.
341 fn correct_for(score: f64, u: f64) -> bool {
342 u < score
343 }
344
345 /// Nothing may be served before evidence exists. This is the fail-closed property: an
346 /// uncertified controller must not hand out a threshold, or the guarantee is decorative.
347 #[test]
348 fn nothing_is_certified_before_any_evidence() {
349 let e = EProcessRiskControl::new(0.1, 0.05, DEFAULT_BET, &grid());
350 assert!(
351 e.certified_threshold().is_none(),
352 "a fresh controller must certify nothing"
353 );
354 assert!(
355 !e.should_serve(1.0),
356 "even a perfect score must not serve without a certified threshold"
357 );
358 }
359
360 /// On a clean stream the controller must eventually certify — a method that never certifies is
361 /// trivially "safe" and completely useless, so this is the liveness half of the contract.
362 #[test]
363 fn a_clean_stream_eventually_certifies() {
364 let mut e = EProcessRiskControl::new(0.2, 0.05, DEFAULT_BET, &grid());
365 let mut rng = Lcg(42);
366 for _ in 0..4000 {
367 let score = 0.90 + rng.next_f64() * 0.10;
368 let correct = correct_for(score, rng.next_f64());
369 e.observe_served(score, correct);
370 }
371 let cert = e
372 .certified_threshold()
373 .expect("4000 clean rounds must certify some threshold");
374 assert!(
375 cert.e_value >= e.crossing_level(),
376 "a certified threshold must sit at or above the Bonferroni crossing level: {} < {}",
377 cert.e_value,
378 e.crossing_level()
379 );
380 }
381
382 /// **The core guarantee, and the reason this module exists.**
383 ///
384 /// Anytime-validity means the bound holds at EVERY round, not just at the end. This checks the
385 /// realized served-failure rate at every single round after certification, on an ADVERSARIAL
386 /// non-exchangeable stream: a clean first half, then an abrupt regime change where the gate's
387 /// error rate jumps. Split conformal's exchangeability assumption is violated by construction
388 /// here — see `split_conformal_breaches_where_the_e_process_holds` for the contrast.
389 #[test]
390 fn coverage_holds_at_every_round_under_a_regime_change() {
391 let alpha = 0.20;
392 let mut worst: f64 = 0.0;
393 for seed in 0..40u64 {
394 let mut e = EProcessRiskControl::new(alpha, 0.05, DEFAULT_BET, &grid());
395 let mut rng = Lcg(seed.wrapping_mul(2_654_435_761).wrapping_add(7));
396 let mut served = 0u64;
397 let mut fails = 0u64;
398 for t in 0..3000 {
399 let score = 0.80 + rng.next_f64() * 0.20;
400 // Regime change at the midpoint: the gate becomes much less reliable, with no
401 // warning and no return. This is exactly what exchangeability forbids.
402 let correct = if t < 1500 {
403 correct_for(score, rng.next_f64())
404 } else {
405 rng.next_f64() < 0.55
406 };
407 if e.should_serve(score) {
408 served += 1;
409 fails += u64::from(!correct);
410 // Check the realized rate at THIS round, not only at the end.
411 if served >= 50 {
412 let rate = fails as f64 / served as f64;
413 worst = worst.max(rate);
414 }
415 }
416 e.observe_served(score, correct);
417 }
418 }
419 // Ville's inequality is a high-probability bound, not an almost-sure one, so a small
420 // overshoot budget is correct rather than a fudge: at delta = 0.05 some excursion is
421 // permitted by the theory itself.
422 assert!(
423 worst <= alpha + 0.10,
424 "worst per-round served-failure {worst:.3} exceeded alpha={alpha} plus slack"
425 );
426 }
427
428 /// **The contrast that makes the contribution.**
429 ///
430 /// On the same non-exchangeable stream, a split-conformal threshold fitted on the first regime
431 /// carries its guarantee only under exchangeability — which the regime change destroys. It goes
432 /// on serving at a threshold the new regime does not justify, and its realized failure rate
433 /// breaches alpha. The e-process, having to re-earn its evidence every round, does not.
434 ///
435 /// If this test ever fails it means the streams stopped being adversarial, not that the method
436 /// improved — so it asserts the breach explicitly rather than just comparing the two.
437 #[test]
438 fn split_conformal_breaches_where_the_e_process_holds() {
439 let alpha = 0.15;
440 let n = 6000;
441 let mut rng = Lcg(20_260_814);
442
443 // Phase 1 doubles as split-conformal's calibration set.
444 let mut calib: Vec<(f64, bool)> = Vec::new();
445 for _ in 0..1000 {
446 let score = 0.70 + rng.next_f64() * 0.30;
447 calib.push((score, correct_for(score, rng.next_f64())));
448 }
449 let fixed = crate::conformal::calibrate(&calib, alpha, 0.05, 30).threshold;
450
451 let mut e = EProcessRiskControl::new(alpha, 0.05, DEFAULT_BET, &grid());
452 // Warm the e-process on the same phase-1 data, so neither method gets a data advantage.
453 for &(s, c) in &calib {
454 e.observe_served(s, c);
455 }
456
457 let (mut sc_served, mut sc_fails) = (0u64, 0u64);
458 let (mut ep_served, mut ep_fails) = (0u64, 0u64);
459 for t in 0..n {
460 let score = 0.70 + rng.next_f64() * 0.30;
461 // Phase 2: the relationship between score and correctness collapses.
462 let correct = if t < n / 2 {
463 correct_for(score, rng.next_f64())
464 } else {
465 rng.next_f64() < 0.60
466 };
467 if score >= fixed {
468 sc_served += 1;
469 sc_fails += u64::from(!correct);
470 }
471 if e.should_serve(score) {
472 ep_served += 1;
473 ep_fails += u64::from(!correct);
474 }
475 e.observe_served(score, correct);
476 }
477
478 let sc_rate = sc_fails as f64 / sc_served.max(1) as f64;
479 let ep_rate = ep_fails as f64 / ep_served.max(1) as f64;
480 assert!(
481 sc_rate > alpha,
482 "the stream is meant to break split conformal, but its rate was {sc_rate:.3} <= alpha={alpha} \
483 — the test has stopped being adversarial and proves nothing"
484 );
485 assert!(
486 ep_rate < sc_rate,
487 "the e-process ({ep_rate:.3}) must beat split conformal ({sc_rate:.3}) under shift"
488 );
489 }
490
491 /// **The cap must bound how long a stale certification survives.**
492 ///
493 /// Written because the first mutation run exposed that nothing covered the cap: deleting it
494 /// left all other tests green. Without a ceiling, a long clean stretch banks unbounded surplus
495 /// evidence, and after the regime turns bad the threshold keeps serving until that surplus is
496 /// burned off — the bound still holds in the limit while realized failure sits above `alpha`
497 /// for thousands of rounds. That is the long-run-average failure mode wearing a new hat.
498 ///
499 /// Capped vs uncapped on an identical stream, measuring rounds-to-de-certify.
500 #[test]
501 fn the_cap_bounds_how_long_a_stale_certification_survives() {
502 let alpha = 0.10;
503 let lag = |cap: f64| -> u64 {
504 let mut e = EProcessRiskControl::with_cap(alpha, 0.05, DEFAULT_BET, cap, &grid());
505 let mut rng = Lcg(99);
506 // Long clean stretch: bank evidence.
507 for _ in 0..5000 {
508 let score = 0.95 + rng.next_f64() * 0.05;
509 e.observe_served(score, correct_for(score, rng.next_f64()));
510 }
511 assert!(
512 e.certified_threshold().is_some(),
513 "the clean stretch must certify, or the lag measurement is meaningless"
514 );
515 // Regime turns hostile: every served item now fails. Count rounds to de-certify.
516 let mut rounds = 0u64;
517 while e.certified_threshold().is_some() && rounds < 100_000 {
518 e.observe_served(0.99, false);
519 rounds += 1;
520 }
521 rounds
522 };
523
524 let capped = lag(DEFAULT_CAP);
525 let uncapped = lag(f64::INFINITY);
526 assert!(
527 capped < uncapped,
528 "the cap must shorten de-certification lag: capped={capped} uncapped={uncapped}"
529 );
530 assert!(
531 capped <= 100,
532 "a capped controller must drop a contradicted certification within ~tens of rounds, took {capped}"
533 );
534 }
535
536 /// **A large bet must not permanently annihilate a threshold.**
537 ///
538 /// Flagged in review. The failure factor is `1 + bet*(alpha - 1)`, which hits exactly zero at
539 /// `bet = 1/(1-alpha)` and goes negative past it. Zero is absorbing under multiplication: one
540 /// failure would retire that threshold forever, and no later evidence could revive it. The
541 /// controller would look like it was still learning while being permanently dead.
542 ///
543 /// `bet` is reachable through the public `with_cap` and through config, so it is validated
544 /// input, not a theoretical corner.
545 #[test]
546 fn an_oversized_bet_cannot_permanently_kill_a_threshold() {
547 let alpha = 0.20;
548 // bet = 5.0 is far past the annihilating 1/(1-0.2) = 1.25.
549 let mut e = EProcessRiskControl::with_cap(alpha, 0.05, 5.0, DEFAULT_CAP, &grid());
550 e.observe_served(0.9, false);
551 assert!(
552 e.e_values().iter().all(|v| *v > 0.0),
553 "a failure must shrink e-values, never zero them: {:?}",
554 e.e_values()
555 );
556
557 // And the process must still be able to recover and certify afterwards.
558 let mut rng = Lcg(7);
559 for _ in 0..4000 {
560 let score = 0.95 + rng.next_f64() * 0.05;
561 e.observe_served(score, correct_for(score, rng.next_f64()));
562 }
563 assert!(
564 e.certified_threshold().is_some(),
565 "a controller that took one early failure must still be able to certify later"
566 );
567 }
568
569 /// **Precedence contract with ACI**, verified at the source rather than trusted in prose.
570 ///
571 /// `proxy.rs` prefers the e-process threshold over `AdaptiveConformal`'s whenever one is
572 /// certified, and falls through otherwise. That is only safe because an uncertified controller
573 /// returns `None` rather than a default — if it ever returned a number before earning it, the
574 /// router would silently serve on an unproven threshold and outrank a method that at least has
575 /// a long-run guarantee.
576 #[test]
577 fn an_uncertified_controller_yields_nothing_for_the_router_to_prefer() {
578 let mut e = EProcessRiskControl::new(0.2, 0.05, DEFAULT_BET, &grid());
579 // Some evidence, but nowhere near the crossing level.
580 for _ in 0..10 {
581 e.observe_served(0.95, true);
582 }
583 assert!(
584 e.certified_threshold().is_none(),
585 "ten rounds must not certify at delta=0.05 over a 21-point grid"
586 );
587 // ...and once it HAS earned it, the router gets a real threshold to prefer.
588 let mut rng = Lcg(11);
589 for _ in 0..4000 {
590 let score = 0.95 + rng.next_f64() * 0.05;
591 e.observe_served(score, correct_for(score, rng.next_f64()));
592 }
593 let c = e
594 .certified_threshold()
595 .expect("sustained clean evidence must certify");
596 assert!(
597 (0.0..=1.0).contains(&c.threshold),
598 "a certified threshold must be servable: {}",
599 c.threshold
600 );
601 }
602
603 /// `delta = 0` means "zero error budget", and must certify NOTHING rather than everything.
604 ///
605 /// Review raised this as a possible false-certification path. It is already handled —
606 /// `crossing_level` returns infinity for a non-positive delta — but "already correct" is not a
607 /// reason to leave a fail-closed contract untested, and an infinity comparison is exactly the
608 /// kind of thing a later refactor breaks silently.
609 #[test]
610 fn a_zero_error_budget_certifies_nothing() {
611 for delta in [0.0, -0.1] {
612 let mut e = EProcessRiskControl::new(0.2, delta, DEFAULT_BET, &grid());
613 let mut rng = Lcg(3);
614 for _ in 0..5000 {
615 let score = 0.99;
616 e.observe_served(score, correct_for(score, rng.next_f64()));
617 }
618 assert!(
619 e.crossing_level().is_infinite(),
620 "delta={delta} must make the crossing level unreachable"
621 );
622 assert!(
623 e.certified_threshold().is_none(),
624 "delta={delta} is a zero error budget: nothing may ever be certified"
625 );
626 assert!(!e.should_serve(1.0), "and nothing may be served");
627 }
628 }
629
630 /// Bonferroni: the crossing level must scale with grid size. Testing every threshold at the
631 /// full `delta` is the multiple-testing trap — each e-process looks valid alone, while the
632 /// family-wise false-certification rate quietly grows with the grid.
633 #[test]
634 fn the_crossing_level_scales_with_grid_size() {
635 let small = EProcessRiskControl::new(0.1, 0.05, DEFAULT_BET, &[0.5, 0.6]);
636 let large = EProcessRiskControl::new(0.1, 0.05, DEFAULT_BET, &grid());
637 assert!(
638 large.crossing_level() > small.crossing_level(),
639 "a larger grid must demand more evidence per threshold: {} vs {}",
640 large.crossing_level(),
641 small.crossing_level()
642 );
643 assert!(
644 (small.crossing_level() - 2.0 / 0.05).abs() < 1e-9,
645 "crossing level must be exactly n/delta"
646 );
647 }
648
649 /// A threshold strict enough to have escalated an item learns nothing from it. Without this,
650 /// unused thresholds would accumulate evidence they never earned and certify on traffic they
651 /// would never have served.
652 #[test]
653 fn thresholds_that_would_not_have_served_are_not_updated() {
654 let mut e = EProcessRiskControl::new(0.1, 0.05, DEFAULT_BET, &[0.2, 0.9]);
655 e.observe_served(0.5, true);
656 let vals = e.e_values();
657 assert!(
658 (vals[1] - 1.0).abs() < f64::EPSILON,
659 "threshold 0.9 did not serve a 0.5-scoring item, so its e-value must stay 1.0, got {}",
660 vals[1]
661 );
662 assert!(
663 vals[0] > 1.0,
664 "threshold 0.2 did serve it correctly, so its e-value must grow, got {}",
665 vals[0]
666 );
667 }
668
669 /// A degenerate grid must be inert, not a panic. `new` takes a slice from config, and an empty
670 /// or non-finite grid is a plausible misconfiguration that must fail closed.
671 #[test]
672 fn a_degenerate_grid_certifies_nothing_and_does_not_panic() {
673 let mut e = EProcessRiskControl::new(0.1, 0.05, DEFAULT_BET, &[f64::NAN, f64::INFINITY]);
674 e.observe_served(0.9, true);
675 assert!(e.certified_threshold().is_none());
676 assert!(!e.should_serve(1.0));
677 assert!(
678 e.crossing_level().is_infinite(),
679 "an empty grid can never cross"
680 );
681 }
682}