Skip to main content

Crate llm_fallback_router

Crate llm_fallback_router 

Source
Expand description

§llm-fallback-router

Multi-provider failover for LLM calls.

You want Anthropic. You also don’t want a 503 from Anthropic to break a production agent. Router takes an ordered list of providers and falls through on retryable errors.

use llm_fallback_router::{Router, Provider, RetryHint};

#[derive(Debug)]
enum MyError {
    RateLimit,
    BadAuth,
}

impl std::fmt::Display for MyError {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        write!(f, "{self:?}")
    }
}
impl std::error::Error for MyError {}
impl RetryHint for MyError {
    fn is_retryable(&self) -> bool {
        matches!(self, MyError::RateLimit)
    }
}

fn anthropic_call(_req: &String) -> Result<String, MyError> { Err(MyError::RateLimit) }
fn openai_call(req: &String) -> Result<String, MyError> { Ok(format!("ok: {req}")) }

let router = Router::<String, String, MyError>::new(vec![
    Provider::new("anthropic", anthropic_call),
    Provider::new("openai", openai_call),
]).unwrap();

let result = router.complete(&"hi".to_string()).unwrap();
assert_eq!(result.provider, "openai");
assert_eq!(result.tries, 2);

§Design: trait vs closure for is_retryable

Rust does not have Python-style runtime exception class inspection. To keep default_is_retryable ergonomic without forcing every caller to plug in a closure, this crate exposes a RetryHint trait that user error types implement. default_is_retryable dispatches through it.

If you’d rather not implement a trait (e.g. you’re wrapping a third-party error type), supply a closure via Router::with_retry_predicate or Provider::with_retry_predicate; the closure wins over the default.

default_is_retryable_status and default_is_retryable_by_name are exposed as standalone helpers so a custom RetryHint impl (or a closure) can reuse the same status-code list and class-name keywords as the Python sibling.

Structs§

Attempt
Record of a single provider call, success or failure.
Provider
A single provider in the fallback chain.
RouteResult
Result of a successful crate::Router::complete call.
Router
Try providers in order, fall through on retryable errors.

Enums§

RouteError
Error returned by Router::complete.

Traits§

RetryHint
Hint a user error type gives the router about whether it should be retried.

Functions§

default_is_retryable
Reasonable default retryable predicate.
default_is_retryable_by_name
Class-name keyword check used by the default retryable predicate.
default_is_retryable_status
Status codes that are considered retryable by default.