llm-universal

One async Rust client for many LLM APIs. Write your chat and tool-calling code once and switch providers by switching the constructor.

Features:

• Tool calling with one normalized format for every provider
• Errors modeled on the official SDKs (BadRequest, Authentication, RateLimit, …, request IDs, retry rules)
• Client-side validation before any network call
• Safe defaults:
  • connect/request timeouts and a 32 MiB response cap
  • no redirects, so the key never goes to another host
  • keys are refused over plain http:// to non-local hosts
  • keys are redacted from Debug and marked sensitive in headers
  • URLs are stripped from connection errors

Install

[dependencies]
llm-universal = "0.2"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Quick start

use llm_universal::{AnthropicProvider, ChatRequest, LlmProvider, Message, OpenAiProvider};

#[tokio::main]
async fn main() -> llm_universal::Result<()> {
    let provider: Box<dyn LlmProvider> = match std::env::var("ANTHROPIC_API_KEY") {
        Ok(key) => Box::new(AnthropicProvider::new(key)),
        Err(_) => Box::new(OpenAiProvider::new(std::env::var("OPENAI_API_KEY").unwrap_or_default())),
    };

    let req = ChatRequest::new("claude-opus-5-5")
        .system("Be concise.")
        .max_tokens(512)
        .message(Message::user("Hello!"));

    let resp = provider.chat(&req).await?;
    println!("{}", resp.text());
    Ok(())
}

Local OpenAI-compatible server:

# use llm_universal::OpenAiProvider;
let ollama = OpenAiProvider::new("").with_base_url("http://localhost:11434/v1");

Tool calling

# use llm_universal::*;
# use serde_json::json;
# async fn run(provider: &dyn LlmProvider) -> Result<()> {
let weather = Tool::new(
    "get_weather",
    "Get the current weather in a city",
    json!({ "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"] }),
);
let mut req = ChatRequest::new("claude-opus-5-5")
    .tool(weather)
    .message(Message::user("What's the weather in Berlin?"));

let resp = provider.chat(&req).await?;
if resp.stop_reason == StopReason::ToolUse {
    let calls = resp.tool_calls();
    req.messages.push(resp.message);
    let results = calls
        .into_iter()
        .map(|(id, _name, _args)| Part::ToolResult { call_id: id, content: "18°C, cloudy".into(), is_error: false })
        .collect();
    req.messages.push(Message::new(Role::Tool, results));
    let final_resp = provider.chat(&req).await?;
    println!("{}", final_resp.text());
}
# Ok(()) }

A complete loop is in examples/tools.rs: cargo run --example tools.

Errors

ApiError carries status, kind (ApiErrorKind::RateLimit, …), message, error_type, code, param, request_id and the JSON body. Its message format matches the SDKs:

401 invalid x-api-key
429 Rate limit reached for requests
invalid args: model must not be empty

# use llm_universal::*;
# async fn run(provider: &dyn LlmProvider, req: &ChatRequest) {
match provider.chat(req).await {
    Ok(resp) => println!("{}", resp.text()),
    Err(err) if err.is_retryable() => eprintln!("retry later: {err} (request id: {:?})", err.request_id()),
    Err(err) => eprintln!("{err}"),
}
# }

is_retryable() follows the SDK retry rules: 408, 409, 429, >= 500, connection errors and timeouts.

Notes

• Anthropic newer models accept only temperature = 1.0; leave temperature unset unless you need it.
• Extended-thinking blocks are not part of the normalized message; if you enable thinking, keep the raw response to send signed thinking blocks back unchanged.
• Gemini tool calls without an id get a local id (llm-universal-call-N) that is never sent back to the API.
• Amazon Bedrock and Vertex AI need cloud-signed requests and are not supported yet.
• Streaming is not supported yet.

License

MIT-0 (MIT No Attribution): use, copy, modify and sell it for any purpose, commercial included, with no attribution required.