llmkit

Unified multi-provider async LLM client for Rust — OpenAI, Anthropic, and local Ollama behind one LlmProvider trait, one streaming API, and a Tower middleware stack (retry, rate limiting, cost tracking, tracing, fallback).

use llmkit::prelude::*;

#[tokio::main]
async fn main() -> LlmResult<()> {
    let client = LlmClientBuilder::new()
        .provider(AnthropicProvider::from_env()?.model("claude-opus-4-8"))
        .fallback(OpenAiProvider::from_env()?.model("gpt-4o-mini"))
        .layer(RetryLayer::exponential(3, std::time::Duration::from_millis(200)))
        .build()?;

    let resp = client.chat(ChatRequest::builder().user("Hello!").build()).await?;
    println!("{}", resp.text().unwrap_or_default());
    Ok(())
}

See the repository for the full guide, examples, and the per-provider crates.

Cargo features

• openai, anthropic, ollama — provider adapters (all enabled by default)

License

Apache-2.0 OR MIT.