Struct Router 

pub struct Router { /* private fields */ }

Several providers, tried in order.

Immutable once built, like a provider. No lock, nothing to contend on, safe to share across as many tasks as you like.

Implementations

impl Router

pub fn new(routes: Vec<Route>) -> Self

A router over these routes, tried in this order.

Nothing is retried and nothing is remembered. Say so with Router::retrying and Router::breaking if you want either.

pub fn breaking(self, policy: Breaker) -> Self

Stop trying a route that has been failing, on the terms this policy sets.

Without it, Router::chat starts at route 0 on every call, so a provider that has been answering 503 for ten minutes is tried first, waited on and fallen through for every single request. A router that does not learn is an ordered try list.

With it, a route that fails is skipped for a while, and starts being tried again on its own: nothing has to reopen a circuit, because time does. What opens one and what does not is Breaker’s table, and the short version is that a failure about the provider opens it and a failure about the request does not.

let router = Router::new(routes).breaking(Breaker::default());

A skipped route is never skipped silently. It goes into Routed::fell_through with how much longer it is being left alone and how many requests in a row it has failed. A router that will not say why it avoided something cannot be debugged at three in the morning.

Handed in rather than assumed, for the reason Router::retrying is: not trying a provider is a decision with consequences a library cannot weigh for you.

pub fn within(self, budget: Budget) -> Self

Refuse a call that would take this run over a cap.

crate::Ledger records what a run cost and nothing stopped it. An agentic bot spends money with nobody watching, and a loop that retries or a plan that expands is found out about on the invoice.

let budget = Budget::of(Micros::parse("5.00")?, "USD");
let router = Router::new(routes).within(budget);

The routes have to carry price books for any of this to mean anything. See Route::priced_by, and crate::budget for what can be checked before a call and what cannot.

What it refuses, and how it says so

A route is skipped, with a reason in Routed::fell_through, when the budget is spent, when the reply alone could overrun what is left, when the route has no price for this model, or when its book is in another currency. If no route survives, the answer is Error::OverBudget naming every reason. Nothing was sent, so nothing was billed.

What it cannot promise

The prompt is not priced before it is sent, because that needs a token count and this crate does not count tokens. So a budget caps what a run may start, and the last call can carry it over by whatever the prompt cost. Set ChatRequest::with_max_tokens and the reply’s share of that overshoot is bounded by a rate you can read.

Concurrent calls can overshoot. Two tasks can both see room for one call and both spend it. Holding a reservation across the call would mean holding a lock across an await, which this crate forbids for a worse reason than this one.

A streamed call is not charged automatically. Usage arrives in a crate::Transcript the router never sees, so Router::stream counts one against Spending::unmeasured and Router::charge is how you settle it.

pub fn spending(&self) -> Option<Spending>

What this run has spent, and what is left.

None without Router::within. Read Spending::is_exact before presenting the figure as a bill.

pub fn charge(&self, route: &str, model: &ModelId, usage: &Usage) -> bool

Charges the budget for a call this router could not price itself.

The case is a streamed reply: its usage arrives in a crate::Transcript long after Router::stream returned, so the router counted one call as unmeasured and left the settling to you.

route is the name Routed::route gave you. The usage is priced against that route’s own book, which is the same book the budget would have used, so a streamed call and a whole one are charged identically.

Answers whether it could be priced. false means the route is unknown here, has no book, has no row for the model, is in another currency, or the usage was never reported: the call stays counted as unmeasured, which is the honest record.

pub fn within_deadline(self, deadline: Duration) -> Self

Give up on the whole request after this long, however many routes are left.

A transport can have a timeout and a Retry policy can have delays, and nothing bounded the sum. Three routes and a policy of two attempts each is six timeouts plus the waits between them, and a caller waiting on an agent has no way to say “answer or fail within twenty seconds”.

let router = Router::new(routes).within_deadline(Duration::from_secs(20));

Where it is checked

Before every attempt rather than only at the start, and before every retry wait. A Retry policy that says three attempts is a maximum rather than a promise, so when the two disagree the deadline wins: a wait that would run past it is not taken, and the router stops instead of spending what is left on an attempt that had no time to finish in.

What it cannot do

It does not cut short a call that is already running. Cancelling one needs a timer, which needs a runtime, and this crate does not pick yours. So the deadline bounds when a new attempt is started and how long the router waits between attempts, and the length of any single call is your transport’s timeout. Set both: one call that hangs for ever will hold a request past any deadline set here.

There is no minimum. Any time left at all is enough to try again. A minimum would be this crate guessing how long a call takes, and a guess that stopped an attempt which would have finished is worse than one attempt that overruns.

How it says so

Error::Timeout, carrying how long the whole thing ran, rather than the last error from a route. A call that gave up has to say whether everything failed or time ran out: one of those is a provider to look at and the other is a deadline to raise, and a caller that cannot tell them apart goes looking in the wrong place.

Not through Routed::fell_through, though that is the obvious place. A Routed only exists when a reply came back, and a spent deadline never produces one, so an entry there would be written and never read. What was tried is recorded on the span instead, under the tracing feature.

pub fn ordering(self, order: Order) -> Self

Reach for routes in this order rather than the order they were written in.

The requirement check still comes first, whatever this says: a route that cannot serve the request is never chosen however cheap or healthy it is. What this changes is which of the routes that can serve it is asked first.

let router = Router::new(routes).ordering(Order::Cheapest);

The sort is stable, so routes this ordering cannot tell apart keep the order you gave them, and Order::AsListed is exactly what the router did before this existed.

pub fn resting(&self) -> Vec<(String, Duration)>

The routes currently being skipped, and how much longer for.

Empty when every route is being tried, which is the answer without Router::breaking. Worth putting on a dashboard: a route that is on this list for hours is a provider nobody has noticed is gone.

pub fn retrying(self, policy: Retry) -> Self

Repeat a failed call, on the terms this policy sets.

Applied per route: a route that fails is tried again before the next one is tried at all. A rate limit is usually the same account whichever route you take, so falling through to the next provider on the first 429 spends a second provider’s budget to learn what waiting would have told you for free.

A refusal still stops everything, retries included. It is an answer.

pub fn unusable(&self) -> Vec<String>

Routes that no request can ever select.

A route whose provider does not know its model, by name. Usually a typo, and without this it shows up as a fallback that quietly never fires. Worth calling at startup.

pub fn routes( &self, ) -> impl Iterator<Item = (String, Option<ModelCapabilities>)> + '_

Every route, with what it can do.

pub async fn preflight(&self) -> Vec<(String, Access)>

Asks every route whether it can be reached, once, at startup.

The other half of Router::unusable. That one catches a route whose provider does not know its model, which is a typo. This catches the route whose credential was rejected, whose tool is not installed, or whose account cannot reach the model, which are the same question asked of the outside world.

One entry per route, named as Route::name writes it, in the order the routes were given. No route is dropped: a Access::Denied one stays selectable, exactly as crate::Registry::stale reports a row and does not remove it. Pruning on the strength of a check is a decision somebody should make, and an Access::Unknown is not grounds for it at all.

What it changes, with a breaker

Without Router::breaking this answers and nothing acts on the answer, which is half of a feature: a route that said Access::Denied at startup was still tried first in every request afterwards.

With one, the answers are fed to the same timer a failed request feeds:

• Access::Denied rests the route for Breaker::settled. Denied means settled: a key, an entitlement, a spelling. Asking again in thirty seconds is asking a question that has been answered.
• Access::Unknown does nothing at all. Unknown means “ask again later”, and that is the whole reason the third variant exists. Treating it as denied would take a working provider out of a router for a network blip that had cleared before anybody read the log.
• Access::Ready does nothing. There is nothing to clear at startup, and a Ready is a claim about a moment rather than a promise about the next request.

The route is still not dropped. It is rested, which is a thing that ends by itself, and Routed::fell_through says it was denied at preflight rather than reporting a failure count of zero.

Where to call it

At startup, beside unusable, and nowhere near Router::chat. Validating per request doubles every round trip to learn something that was almost always true, and the answer would be stale by the time the request went out anyway.

Routes are asked one after another rather than together. This runs once, against a handful of providers, and running them concurrently would mean a join primitive and an assumption about the runtime for a saving nobody can measure.

pub async fn chat( &self, request: ChatRequest, needs: Requirements, ) -> Result<Routed>

Sends a request to the first route that can serve it.

The request’s own model is replaced by the route’s, because the router is choosing the model. Everything else about the request is sent as you wrote it.

Errors

Error::Unsupported when no route meets the requirements, naming what each route was missing. That is a configuration problem, and a message that only said “no route available” would leave somebody comparing tables by hand.

When routes were tried and all failed, the last error is returned rather than a summary. It is a real error from a real provider, with its own retry advice, and a wrapper around it would lose that.

pub async fn stream( &self, request: ChatRequest, needs: Requirements, ) -> Result<(EventStream<'_>, Routed<()>)>

Starts a stream on the first route that can serve it.

The streaming half of Router::chat, and the same request rewriting applies: the route’s model replaces the request’s, because the router is choosing the model.

Falling through stops at the first event

A route is only replaceable before the caller has seen anything. Once a chunk has reached you, moving to another provider would mean handing half a sentence to a second model and asking it to continue: text nobody wrote, in one voice, with nothing downstream able to detect it.

So a provider that fails while the stream is being opened falls through normally, and a provider that fails after that does not. The second kind arrives as an Err item inside the stream and stays there, which is what crate::Transcript::drain is for: what already arrived is still yours.

The seam is real rather than assumed. HttpTransport::send_streaming checks the status before handing over any bytes, so a 429 or a 503 is an Err from this method and is fallen through, not an error mid-answer.

What comes back

The stream, and a Routed<()> carrying which route won and what fell through, exactly as Router::chat does. They are two values rather than one because an EventStream is neither Debug nor Clone.

There is no usage yet when this returns. A streamed call reports its tokens in its final frame, so what you want is crate::Transcript, and it is the transcript that knows what the call consumed.

let request = ChatRequest::new("gpt-5", vec![Message::user("Hello")]);
let needs = Requirements::of(&request).streaming();

let mut transcript = Transcript::new(request.model.clone());
let (events, routed) = router.stream(request, needs).await?;
let outcome = transcript.drain(events).await;

println!("{} answered", routed.route);
let reply = transcript.finish();
if let Err(cut_short) = outcome {
    eprintln!("{} arrived before {cut_short}", reply.text().len());
}

A provider that does not really stream

crate::Provider::stream has a default that calls chat and replays the finished reply as one burst. Routing to one of those is not an error and this method does not prevent it, for the reason the trait does not: it is a real answer with the same text and the same usage, just all at once.

It does mean the window in which this can fall through covers the whole call rather than the first byte, which is more forgiving and not less. Say Requirements::streaming when a person is watching a screen, and a pairing that only pretends is skipped before it is asked.

Errors

The same as Router::chat: Error::Unsupported when no route meets the requirements, and otherwise the last error from the routes that were tried.