llm-universal

One async Rust client for many LLM APIs. Write your chat, streaming and tool-calling code once and switch providers by switching the constructor.

Features

Safe defaults:

• Connect timeout, idle read timeout, a 32 MiB response cap and a 4 MiB cap per stream event.
• No redirects, so the key never goes to another host.
• Keys are refused over plain http:// to non-local hosts.
• Keys are redacted from Debug and marked sensitive in headers.
• URLs are stripped from connection errors.
• Model, project and region are checked before they go into a URL.

Install

[dependencies]
llm-universal = "0.3"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Quick start

use llm_universal::{AnthropicProvider, ChatRequest, LlmProvider, Message, OpenAiProvider};

#[tokio::main]
async fn main() -> llm_universal::Result<()> {
    let provider: Box<dyn LlmProvider> = match std::env::var("ANTHROPIC_API_KEY") {
        Ok(key) => Box::new(AnthropicProvider::new(key)),
        Err(_) => Box::new(OpenAiProvider::new(std::env::var("OPENAI_API_KEY").unwrap_or_default())),
    };

    let req = ChatRequest::new("claude-opus-5-5")
        .system("Be concise.")
        .max_tokens(512)
        .message(Message::user("Hello!"));

    let resp = provider.chat(&req).await?;
    println!("{}", resp.text());
    Ok(())
}

Streaming

# use llm_universal::*;
# async fn run(provider: &dyn LlmProvider, req: &ChatRequest) -> Result<()> {
let mut stream = provider.chat_stream(req).await?;
while let Some(event) = stream.next().await {
    match event? {
        StreamEvent::TextDelta(text) => print!("{text}"),
        StreamEvent::Done(response) => println!("\n{} tokens", response.usage.output_tokens),
        _ => {}
    }
}
# Ok(()) }

Tool calling

run_tools sends the request, runs your handler for every tool call, sends the results back and repeats until the model answers:

# use llm_universal::*;
# use serde_json::json;
# async fn run(provider: &dyn LlmProvider) -> Result<()> {
let weather = Tool::new(
    "get_weather",
    "Get the current weather in a city",
    json!({ "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"] }),
);
let mut req = ChatRequest::new("claude-opus-5-5")
    .tool(weather)
    .message(Message::user("What's the weather in Berlin?"));

let answer = run_tools(provider, &mut req, 5, |call| async move {
    match call.name.as_str() {
        "get_weather" => Ok(format!("18°C, cloudy in {}", call.arguments["city"])),
        other => Err(format!("unknown tool {other}")),
    }
})
.await?;
println!("{}", answer.text());
# Ok(()) }

The manual loop: check response.tool_calls(), push response.message to req.messages, then push Message::new(Role::Tool, results) built with Part::tool_result(id, content, is_error).

Images

# use llm_universal::*;
# async fn run(provider: &dyn LlmProvider, png_base64: String) -> Result<()> {
let req = ChatRequest::new("claude-opus-5-5")
    .message(Message::user_with_image("What is in this picture?", Part::image_base64("image/png", png_base64)));
println!("{}", provider.chat(&req).await?.text());
# Ok(()) }

Part::image_url(url) works with OpenAI, Anthropic and Cohere; Gemini needs base64 data.

Structured output

# use llm_universal::*;
# use serde_json::json;
# async fn run(provider: &dyn LlmProvider) -> Result<()> {
#[derive(serde::Deserialize)]
struct City { name: String, population: u64 }

let req = ChatRequest::new("gpt-4o")
    .json_schema("city", json!({
        "type": "object",
        "properties": { "name": { "type": "string" }, "population": { "type": "integer" } },
        "required": ["name", "population"],
        "additionalProperties": false
    }))
    .message(Message::user("Largest city in Germany?"));
let city: City = provider.chat(&req).await?.json()?;
println!("{} — {}", city.name, city.population);
# Ok(()) }

Errors

ApiError carries status, kind (ApiErrorKind::RateLimit, …), message, error_type, code, param, request_id, retry_after and the JSON body. The message format matches the SDKs:

401 invalid x-api-key
429 Rate limit reached for requests
invalid args: model must not be empty

Retries follow the SDKs: 2 retries by default on connection errors, timeouts, 408, 409, 429 and

= 500. retry-after / retry-after-ms (up to 60 s) and x-should-retry are honored; otherwise the delay is 0.5 s doubling up to 8 s, with jitter. err.is_retryable() tells you whether a final error is worth retrying later.

Notes

• Anthropic's newer models accept only temperature = 1.0; leave temperature unset unless you need it.
• Reasoning blocks (Part::Reasoning) and Gemini's thoughtSignature are kept in the message and sent back unchanged on the next turn, as Anthropic and Gemini require. Keep response.message in the history.
• Gemini tool calls without an id get a local id (llm-universal-call-N) that is never sent back to the API.
• Vertex AI needs an OAuth access token (for example gcloud auth print-access-token); tokens expire after about an hour. Model lists and Gemini embeddings are not available on Vertex in this crate.
• Bedrock uses the OpenAI-compatible endpoint with a Bedrock API key; SigV4 signing is not implemented.

License

MIT-0 (MIT No Attribution): use, copy, modify and sell it for any purpose, commercial included, with no attribution required.