Expand description
§UCCI: calibrated uncertainty for cost-optimal LLM cascade routing
Rust implementation of the method in Varun Kotte, “UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing”, arXiv:2605.18796 (2026), https://arxiv.org/abs/2605.18796. It is a port of the Python reference package in the same repository and is tested against golden vectors that the Python package generates.
A two-model cascade answers every query with a small model and escalates to a large model only when the small model is likely wrong. UCCI builds the routing policy in three steps (Section 4):
signal: a scalar uncertaintyu(x) = 1 - mean_t (p_{t,1} - p_{t,2})from the small model’s top-2 token probabilities (Section 4.1, Eq. 4);calibration: an isotonic mapgfromu(x)to the probability that the small model is wrong,p_hat(x) = g(u(x))(Section 4.2, Eq. 5);policy: escalate whenp_hat(x) > theta, withthetachosen on a validation set as the cheapest threshold whose accuracy reaches a targettau(Section 4.3, Eqs. 6 and 7).
Section 5 (Theorem 1) shows that, under three explicit assumptions, such a
threshold policy on the calibrated probability is cost-optimal among
policies that depend only on u(x). metrics provides the diagnostics
of Section 6 (ECE, reliability table, Brier score, micro-F1 of routed
answers), and router bundles a fitted calibrator and threshold into a
Router. A router fitted by the Python package loads from its JSON file
(feature json, on by default), and RouterBuilder runs the whole
protocol of Section 6.1 in Rust: fit g on the calibration split, choose
theta on the validation split, evaluate on the test split.
Every fallible function returns Result with a UcciError that names
the offending input. Inputs are validated like in the Python package:
NaN and infinities are rejected, probabilities must lie in [0, 1], costs
must be positive.
§Quickstart
use ucci::{default_grid, select_threshold, Costs, IsotonicCalibrator, Router};
// Calibration split: uncertainty u(x) and whether the small model was wrong.
let u_cal = [0.05, 0.10, 0.20, 0.30, 0.40, 0.55, 0.70, 0.90];
let e_cal = [0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0];
let g = IsotonicCalibrator::fit(&u_cal, &e_cal)?;
// Validation split: both models were run, so each query has a score for
// the small and for the large model's answer.
let u_val = [0.05, 0.25, 0.35, 0.60, 0.80, 0.95];
let small = [1.0, 1.0, 0.0, 0.0, 1.0, 0.0];
let large = [1.0, 1.0, 1.0, 1.0, 1.0, 0.0];
let p_val = g.predict_many(&u_val)?;
let costs = Costs::paper(); // c_small = 1.0, c_large = 3.02 (Section 6.1)
let choice = select_threshold(&p_val, &small, &large, 0.8, &costs, &default_grid())?;
assert!(choice.accuracy >= 0.8);
// Route new queries.
let router = Router::new(g, choice.theta, costs)?;
assert!(!router.escalate(0.05)?); // confident: keep the small model's answer
assert!(router.escalate(0.95)?); // uncertain: escalate to the large model§Parity with the Python package
Every computation follows the Python reference implementation operation by
operation, including numpy’s summation order and numpy.interp, and router
files are parsed with exact float round-tripping. The golden test
(tests/golden.rs) replays every vector that tools/make_golden.py
produces with the Python package, requires agreement to 1e-12, and
reports how many values match bit for bit
(cargo test --test golden -- --nocapture); with the current vectors all
of them do. Routers written by this crate are read back by the Python test
suite.
What stays in the Python package: bootstrap confidence intervals (they depend on numpy’s random generator), the baselines of Section 6.1, the command-line tool and the benchmark scripts.
§Features
json(default):Router::from_json,Router::to_json,Router::loadandRouter::savefor the router file format shared with the Python package, serde support forRouterand the result types, andsignal::from_openai_logprobs. It pulls inserdeandserde_json; without it the crate has no dependencies.
The minimum supported Rust version is 1.70.
Re-exports§
pub use calibration::pav;pub use calibration::IsotonicCalibrator;pub use error::Result;pub use error::UcciError;pub use metrics::brier_score;pub use metrics::ece;pub use metrics::micro_f1;pub use metrics::reliability_table;pub use metrics::BinStrategy;pub use metrics::ReliabilityRow;pub use metrics::RoutedMicroF1;pub use policy::default_grid;pub use policy::escalate;pub use policy::escalate_many;pub use policy::evaluate;pub use policy::make_grid;pub use policy::pareto_frontier;pub use policy::policy_accuracy;pub use policy::policy_cost;pub use policy::select_threshold;pub use policy::select_threshold_for_budget;pub use policy::CostModel;pub use policy::Costs;pub use policy::ParetoFrontier;pub use policy::ThresholdChoice;pub use router::CalibratedRouter;pub use router::Route;pub use router::Router;pub use router::RouterBuilder;pub use signal::margins_from_top2;pub use signal::token_margin_uncertainty;pub use signal::top2_from_logprobs;pub use signal::uncertainty_from_logprobs;pub use signal::uncertainty_from_margins;pub use signal::uncertainty_from_probs;pub use signal::uncertainty_from_top_logprobs;pub use signal::MarginAccumulator;
Modules§
- calibration
- Isotonic calibration of u(x) into an error probability (paper Section 4.2).
- error
- The error type shared by every module of the crate.
- metrics
- Forecast-quality and evaluation metrics (Section 6.2, Figure 1).
- policy
- Threshold policy and threshold selection (paper Section 4.3 and Section 5).
- router
- A fitted UCCI router: calibrator plus threshold (Sections 4 and 6.1), and
the
ucci-routerJSON format shared with the Python package. - signal
- Token-margin uncertainty u(x) (paper Section 4.1, Eq. 4).