Module router 

Cost-aware routing and cross-provider fallback chain for LLMClient. Cost-aware routing and fallback chain for LLMClient.

RouterClient holds an ordered set of Backends and routes each request according to a RoutingStrategy. On a transient error it falls through to the next backend; a permanent client error short-circuits. This is the orchestration layer for cross-provider / cross-model resilience.

Separation of concerns

The llm module composes resilience from focused decorators rather than folding every behavior into one client:

Concern	Decorator
rate-limit (429) / 5xx backoff	RetryClient
routing / cross-provider fallback	RouterClient
request/response observation	MiddlewareClient
response caching	CacheClient

Because every decorator implements LLMClient, they compose freely. A resilient, observed, cached multi-backend stack reads inside-out:

use llm_kernel::llm::{
    CacheClient, LLMClient, MiddlewareClient, NoopMiddleware, OpenAIClient,
    RetryClient, RouterClient, RoutingStrategy, Backend, RetryConfig,
};

let cheap = Backend::new(OpenAIClient::from_key("gpt-4o-mini", "sk-...")?, Some((0.15, 0.60)));
let powerful = Backend::new(OpenAIClient::from_key("gpt-4o", "sk-...")?, Some((2.50, 10.00)));

// Route across backends; retry each transiently; observe; cache.
let stack = CacheClient::new(
    MiddlewareClient::new(
        RouterClient::new(
            vec![cheap, powerful]
                .into_iter()
                .map(|b| Backend { client: RetryClient::new(b.client, RetryConfig::default()), ..b })
                .collect(),
            RoutingStrategy::LowestCost,
        )?,
        NoopMiddleware,
    ),
    store,
);

Streaming cannot fall through once a stream is established. So RouterClient::stream_complete tries each backend in order and returns the first stream it can establish; an error raised before the stream starts (connection, 403) falls through to the next backend just like LLMClient::complete. Once a stream is returned, errors raised during streaming are not retried — wrap individual backends in RetryClient for transient resilience.

Structs

Backend

A single backend in a RouterClient chain.

RouterClient

A LLMClient that routes requests across multiple backends with fallback.

Enums

RoutingStrategy

How a RouterClient orders and falls through its backends.