ucci (Rust)
Rust implementation of UCCI, calibrated uncertainty for cost-optimal LLM cascade routing, from Varun Kotte, "UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing", arXiv:2605.18796 (2026).
A two-model cascade answers each query with a small model and escalates to a large model only when the small model is likely wrong. UCCI makes that decision in three steps (paper Section 4):
- Signal (Section 4.1, Eq. 4):
u(x) = 1 - (1/T) sum_t (p_{t,1} - p_{t,2}), the mean top-1 minus top-2 token-probability margin of the small model's greedy generation, subtracted from 1. - Calibration (Section 4.2): an isotonic map
gfromu(x)to the probability that the small model is wrong,p_hat(x) = g(u(x)). - Threshold (Section 4.3, Eqs. 6 and 7): escalate when
p_hat(x) > theta, withthetathe cheapest threshold whose validation accuracy reaches a targettau. Under the assumptions of Theorem 1 (Section 5) this threshold policy is cost-optimal among policies that depend only onu(x).
This crate is a port of the Python reference package in the same repository
(src/ucci). It computes the same numbers: every function is tested against
golden vectors generated by the Python package, and router files move freely
between the two.
Install
[]
= { = "https://github.com/varunkotte6/ucci" }
The default feature json adds router files and serde support (dependencies:
serde, serde_json). With default-features = false the crate has no
dependencies.
Minimum supported Rust version: 1.70. The crate itself needs nothing newer;
recent serde_derive releases need Rust 1.71, so on 1.70 pin serde with
cargo update -p serde --precise 1.0.228.
Route with a router fitted in Python
Fit and save the router with the Python package:
= # Section 4.2
# Eq. 7
Load it in Rust and route each query from the small model's top-2 log-probabilities:
use uncertainty_from_top_logprobs;
use Router;
cargo run --example route loads examples/router.json, a router saved by
the Python package, and routes a few values of u(x); pass a path and values
to route your own (cargo run --example route -- router.json 0.1 0.4).
With the json feature, ucci::signal::from_openai_logprobs takes the
choice.logprobs.content array of a chat completion directly, and
ucci::signal::MarginAccumulator computes u(x) token by token while a
response streams in.
Fit a router in Rust
The evaluation protocol of Section 6.1 (fit g on the calibration split,
choose theta on the validation split, route the test split end to end with
actual outputs and costs):
use ;
Costs::paper() uses the paper's normalized costs, c_small = 1.0 and
c_large = 3.02 (the measured latency ratio, Section 6.1), under the routing
cost model (an escalated query costs c_large). CostModel::Sequential
charges c_small + c_large instead. choose_threshold_for_budget gives the
matched-budget form used for the bottom block of Table 2, and
policy::pareto_frontier returns the cost and accuracy of every grid
threshold. For a corpus-level metric such as micro-F1 over entities, pass
metrics::RoutedMicroF1::metric() to policy::select_threshold_with_metric.
Modules
| Module | Paper | Contents |
|---|---|---|
signal |
Section 4.1 | u(x) from probability pairs, log-probabilities, OpenAI-style top_logprobs, streaming |
calibration |
Section 4.2 | weighted pool-adjacent-violators, IsotonicCalibrator (scikit-learn semantics, clipped ends) |
policy |
Section 4.3, Theorem 1, Tables 2 and 3 | escalate, policy_cost, select_threshold, budget form, Pareto frontier, evaluate |
metrics |
Section 6 | ECE (uniform or quantile bins), reliability table, Brier score, micro-F1 |
router |
Sections 4 and 6.1 | Router, RouterBuilder, the ucci-router JSON format |
Every fallible function returns Result<_, UcciError>; inputs are validated
exactly like in the Python package (NaN and infinities are rejected,
probabilities must lie in [0, 1], costs must be positive).
Parity with Python
Each computation follows the Python package operation by operation, down to
numpy's summation order, numpy.interp and scikit-learn's tie pooling, and
router files are parsed with exact float round-tripping. The golden test
replays every vector in ../tests/golden (written by
tools/make_golden.py), requires agreement to 1e-12 and reports how many
values match bit for bit; with the current vectors all of them do:
Routers written by this crate are read back by the Python test suite
(tests/test_golden.py). After an intended change of the writer, refresh
that file with UCCI_BLESS=1 cargo test --test golden.
Bootstrap confidence intervals, the baselines of Section 6.1, the command-line tool and the benchmark scripts live in the Python package.
Citation
License
MIT, like the Python package. See LICENSE.