ucci 0.1.0

UCCI: calibrated uncertainty for cost-optimal LLM cascade routing (token-margin signal, isotonic calibration, constrained threshold selection)
Documentation

ucci (Rust)

Rust implementation of UCCI, calibrated uncertainty for cost-optimal LLM cascade routing, from Varun Kotte, "UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing", arXiv:2605.18796 (2026).

A two-model cascade answers each query with a small model and escalates to a large model only when the small model is likely wrong. UCCI makes that decision in three steps (paper Section 4):

  1. Signal (Section 4.1, Eq. 4): u(x) = 1 - (1/T) sum_t (p_{t,1} - p_{t,2}), the mean top-1 minus top-2 token-probability margin of the small model's greedy generation, subtracted from 1.
  2. Calibration (Section 4.2): an isotonic map g from u(x) to the probability that the small model is wrong, p_hat(x) = g(u(x)).
  3. Threshold (Section 4.3, Eqs. 6 and 7): escalate when p_hat(x) > theta, with theta the cheapest threshold whose validation accuracy reaches a target tau. Under the assumptions of Theorem 1 (Section 5) this threshold policy is cost-optimal among policies that depend only on u(x).

This crate is a port of the Python reference package in the same repository (src/ucci). It computes the same numbers: every function is tested against golden vectors generated by the Python package, and router files move freely between the two.

Install

[dependencies]
ucci = { git = "https://github.com/varunkotte6/ucci" }

The default feature json adds router files and serde support (dependencies: serde, serde_json). With default-features = false the crate has no dependencies.

Minimum supported Rust version: 1.70. The crate itself needs nothing newer; recent serde_derive releases need Rust 1.71, so on 1.70 pin serde with cargo update -p serde --precise 1.0.228.

Route with a router fitted in Python

Fit and save the router with the Python package:

from ucci import UCCIRouter

router = UCCIRouter().calibrate(u_cal, e_cal)          # Section 4.2
router.choose_threshold(u_val, small_val, large_val, tau=0.91)   # Eq. 7
router.save("router.json")

Load it in Rust and route each query from the small model's top-2 log-probabilities:

use ucci::signal::uncertainty_from_top_logprobs;
use ucci::Router;

fn main() -> Result<(), ucci::UcciError> {
    let router = Router::load("router.json")?;

    // Candidate log-probabilities per generated content token, as returned by
    // an OpenAI-compatible server with top_logprobs = 2 (any order).
    let top_logprobs = vec![vec![-0.01, -4.7], vec![-0.9, -0.6], vec![-0.05, -3.2]];
    let u = uncertainty_from_top_logprobs(&top_logprobs)?;

    let route = router.route(u)?;
    println!("p_hat = {:.3}, escalate = {}", route.p_hat, route.escalate);
    Ok(())
}

cargo run --example route loads examples/router.json, a router saved by the Python package, and routes a few values of u(x); pass a path and values to route your own (cargo run --example route -- router.json 0.1 0.4).

With the json feature, ucci::signal::from_openai_logprobs takes the choice.logprobs.content array of a chat completion directly, and ucci::signal::MarginAccumulator computes u(x) token by token while a response streams in.

Fit a router in Rust

The evaluation protocol of Section 6.1 (fit g on the calibration split, choose theta on the validation split, route the test split end to end with actual outputs and costs):

use ucci::{Costs, RouterBuilder};

fn main() -> Result<(), ucci::UcciError> {
    // Calibration split: u(x) and whether the small model was wrong.
    let u_cal = [0.05, 0.10, 0.20, 0.30, 0.40, 0.55, 0.70, 0.90];
    let e_cal = [0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0];
    // Validation split: both models were run; per-query scores of their outputs.
    let u_val = [0.05, 0.25, 0.35, 0.60, 0.80, 0.95];
    let small_val = [1.0, 1.0, 0.0, 0.0, 1.0, 0.0];
    let large_val = [1.0, 1.0, 1.0, 1.0, 1.0, 0.0];
    // Test split.
    let u_test = [0.02, 0.33, 0.51, 0.88];
    let small_test = [1.0, 1.0, 0.0, 0.0];
    let large_test = [1.0, 1.0, 1.0, 1.0];

    let router = RouterBuilder::new(Costs::paper())? // c_small = 1.0, c_large = 3.02
        .calibrate(&u_cal, &e_cal)?
        .choose_threshold(&u_val, &small_val, &large_val, 0.8)?;

    let test = router.evaluate(&u_test, &small_test, &large_test)?;
    println!(
        "theta = {}, test cost = {:.3}, test accuracy = {:.3}",
        router.theta(),
        test.cost,
        test.accuracy
    );
    Ok(())
}

Costs::paper() uses the paper's normalized costs, c_small = 1.0 and c_large = 3.02 (the measured latency ratio, Section 6.1), under the routing cost model (an escalated query costs c_large). CostModel::Sequential charges c_small + c_large instead. choose_threshold_for_budget gives the matched-budget form used for the bottom block of Table 2, and policy::pareto_frontier returns the cost and accuracy of every grid threshold. For a corpus-level metric such as micro-F1 over entities, pass metrics::RoutedMicroF1::metric() to policy::select_threshold_with_metric.

Modules

Module Paper Contents
signal Section 4.1 u(x) from probability pairs, log-probabilities, OpenAI-style top_logprobs, streaming
calibration Section 4.2 weighted pool-adjacent-violators, IsotonicCalibrator (scikit-learn semantics, clipped ends)
policy Section 4.3, Theorem 1, Tables 2 and 3 escalate, policy_cost, select_threshold, budget form, Pareto frontier, evaluate
metrics Section 6 ECE (uniform or quantile bins), reliability table, Brier score, micro-F1
router Sections 4 and 6.1 Router, RouterBuilder, the ucci-router JSON format

Every fallible function returns Result<_, UcciError>; inputs are validated exactly like in the Python package (NaN and infinities are rejected, probabilities must lie in [0, 1], costs must be positive).

Parity with Python

Each computation follows the Python package operation by operation, down to numpy's summation order, numpy.interp and scikit-learn's tie pooling, and router files are parsed with exact float round-tripping. The golden test replays every vector in ../tests/golden (written by tools/make_golden.py), requires agreement to 1e-12 and reports how many values match bit for bit; with the current vectors all of them do:

cargo test --test golden -- --nocapture

Routers written by this crate are read back by the Python test suite (tests/test_golden.py). After an intended change of the writer, refresh that file with UCCI_BLESS=1 cargo test --test golden.

Bootstrap confidence intervals, the baselines of Section 6.1, the command-line tool and the benchmark scripts live in the Python package.

Citation

@article{kotte2026ucci,
  title   = {{UCCI}: Calibrated Uncertainty for Cost-Optimal {LLM} Cascade Routing},
  author  = {Kotte, Varun},
  journal = {arXiv preprint arXiv:2605.18796},
  year    = {2026}
}

License

MIT, like the Python package. See LICENSE.