# ucci (Rust)
Rust implementation of **UCCI**, calibrated uncertainty for cost-optimal LLM
cascade routing, from Varun Kotte,
["UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing"](https://arxiv.org/abs/2605.18796),
arXiv:2605.18796 (2026).
A two-model cascade answers each query with a small model and escalates to a
large model only when the small model is likely wrong. UCCI makes that
decision in three steps (paper Section 4):
1. **Signal** (Section 4.1, Eq. 4): `u(x) = 1 - (1/T) sum_t (p_{t,1} - p_{t,2})`,
the mean top-1 minus top-2 token-probability margin of the small model's
greedy generation, subtracted from 1.
2. **Calibration** (Section 4.2): an isotonic map `g` from `u(x)` to the
probability that the small model is wrong, `p_hat(x) = g(u(x))`.
3. **Threshold** (Section 4.3, Eqs. 6 and 7): escalate when `p_hat(x) > theta`,
with `theta` the cheapest threshold whose validation accuracy reaches a
target `tau`. Under the assumptions of Theorem 1 (Section 5) this
threshold policy is cost-optimal among policies that depend only on `u(x)`.
This crate is a port of the Python reference package in the same repository
(`src/ucci`). It computes the same numbers: every function is tested against
golden vectors generated by the Python package, and router files move freely
between the two.
## Install
```toml
[dependencies]
ucci = { git = "https://github.com/varunkotte6/ucci" }
```
The default feature `json` adds router files and serde support (dependencies:
`serde`, `serde_json`). With `default-features = false` the crate has no
dependencies.
Minimum supported Rust version: 1.70. The crate itself needs nothing newer;
recent `serde_derive` releases need Rust 1.71, so on 1.70 pin serde with
`cargo update -p serde --precise 1.0.228`.
## Route with a router fitted in Python
Fit and save the router with the Python package:
```python
from ucci import UCCIRouter
router = UCCIRouter().calibrate(u_cal, e_cal) # Section 4.2
router.choose_threshold(u_val, small_val, large_val, tau=0.91) # Eq. 7
router.save("router.json")
```
Load it in Rust and route each query from the small model's top-2
log-probabilities:
```rust,no_run
use ucci::signal::uncertainty_from_top_logprobs;
use ucci::Router;
fn main() -> Result<(), ucci::UcciError> {
let router = Router::load("router.json")?;
// Candidate log-probabilities per generated content token, as returned by
// an OpenAI-compatible server with top_logprobs = 2 (any order).
let top_logprobs = vec![vec![-0.01, -4.7], vec![-0.9, -0.6], vec![-0.05, -3.2]];
let u = uncertainty_from_top_logprobs(&top_logprobs)?;
let route = router.route(u)?;
println!("p_hat = {:.3}, escalate = {}", route.p_hat, route.escalate);
Ok(())
}
```
`cargo run --example route` loads `examples/router.json`, a router saved by
the Python package, and routes a few values of `u(x)`; pass a path and values
to route your own (`cargo run --example route -- router.json 0.1 0.4`).
With the `json` feature, `ucci::signal::from_openai_logprobs` takes the
`choice.logprobs.content` array of a chat completion directly, and
`ucci::signal::MarginAccumulator` computes `u(x)` token by token while a
response streams in.
## Fit a router in Rust
The evaluation protocol of Section 6.1 (fit `g` on the calibration split,
choose `theta` on the validation split, route the test split end to end with
actual outputs and costs):
```rust
use ucci::{Costs, RouterBuilder};
fn main() -> Result<(), ucci::UcciError> {
// Calibration split: u(x) and whether the small model was wrong.
let u_cal = [0.05, 0.10, 0.20, 0.30, 0.40, 0.55, 0.70, 0.90];
let e_cal = [0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0];
// Validation split: both models were run; per-query scores of their outputs.
let u_val = [0.05, 0.25, 0.35, 0.60, 0.80, 0.95];
let small_val = [1.0, 1.0, 0.0, 0.0, 1.0, 0.0];
let large_val = [1.0, 1.0, 1.0, 1.0, 1.0, 0.0];
// Test split.
let u_test = [0.02, 0.33, 0.51, 0.88];
let small_test = [1.0, 1.0, 0.0, 0.0];
let large_test = [1.0, 1.0, 1.0, 1.0];
let router = RouterBuilder::new(Costs::paper())? // c_small = 1.0, c_large = 3.02
.calibrate(&u_cal, &e_cal)?
.choose_threshold(&u_val, &small_val, &large_val, 0.8)?;
let test = router.evaluate(&u_test, &small_test, &large_test)?;
println!(
"theta = {}, test cost = {:.3}, test accuracy = {:.3}",
router.theta(),
test.cost,
test.accuracy
);
Ok(())
}
```
`Costs::paper()` uses the paper's normalized costs, `c_small = 1.0` and
`c_large = 3.02` (the measured latency ratio, Section 6.1), under the routing
cost model (an escalated query costs `c_large`). `CostModel::Sequential`
charges `c_small + c_large` instead. `choose_threshold_for_budget` gives the
matched-budget form used for the bottom block of Table 2, and
`policy::pareto_frontier` returns the cost and accuracy of every grid
threshold. For a corpus-level metric such as micro-F1 over entities, pass
`metrics::RoutedMicroF1::metric()` to `policy::select_threshold_with_metric`.
## Modules
| `signal` | Section 4.1 | `u(x)` from probability pairs, log-probabilities, OpenAI-style `top_logprobs`, streaming |
| `calibration` | Section 4.2 | weighted pool-adjacent-violators, `IsotonicCalibrator` (scikit-learn semantics, clipped ends) |
| `policy` | Section 4.3, Theorem 1, Tables 2 and 3 | `escalate`, `policy_cost`, `select_threshold`, budget form, Pareto frontier, `evaluate` |
| `metrics` | Section 6 | ECE (uniform or quantile bins), reliability table, Brier score, micro-F1 |
| `router` | Sections 4 and 6.1 | `Router`, `RouterBuilder`, the `ucci-router` JSON format |
Every fallible function returns `Result<_, UcciError>`; inputs are validated
exactly like in the Python package (NaN and infinities are rejected,
probabilities must lie in `[0, 1]`, costs must be positive).
## Parity with Python
Each computation follows the Python package operation by operation, down to
numpy's summation order, `numpy.interp` and scikit-learn's tie pooling, and
router files are parsed with exact float round-tripping. The golden test
replays every vector in `../tests/golden` (written by
`tools/make_golden.py`), requires agreement to `1e-12` and reports how many
values match bit for bit; with the current vectors all of them do:
```sh
cargo test --test golden -- --nocapture
```
Routers written by this crate are read back by the Python test suite
(`tests/test_golden.py`). After an intended change of the writer, refresh
that file with `UCCI_BLESS=1 cargo test --test golden`.
Bootstrap confidence intervals, the baselines of Section 6.1, the command-line
tool and the benchmark scripts live in the Python package.
## Citation
```bibtex
@article{kotte2026ucci,
title = {{UCCI}: Calibrated Uncertainty for Cost-Optimal {LLM} Cascade Routing},
author = {Kotte, Varun},
journal = {arXiv preprint arXiv:2605.18796},
year = {2026}
}
```
## License
MIT, like the Python package. See `LICENSE`.