llm-universal 0.2.0

# llm-universal

One async Rust client for many LLM APIs. Write your chat and tool-calling code once and switch
providers by switching the constructor.

| Provider | Constructor | API |
|---|---|---|
| Anthropic (Claude) | `AnthropicProvider::new(key)` | Messages API |
| OpenAI | `OpenAiProvider::new(key)` | Chat Completions |
| Google Gemini | `GeminiProvider::new(key)` | `generateContent` |
| Cohere | `CohereProvider::new(key)` | Chat API v2 |
| Azure OpenAI | `OpenAiProvider::azure(resource, key)` | Azure OpenAI v1 (`api-key` header) |
| Groq | `OpenAiProvider::groq(key)` | OpenAI-compatible |
| Mistral | `OpenAiProvider::mistral(key)` | OpenAI-compatible |
| DeepSeek | `OpenAiProvider::deepseek(key)` | OpenAI-compatible |
| xAI (Grok) | `OpenAiProvider::xai(key)` | OpenAI-compatible |
| OpenRouter | `OpenAiProvider::openrouter(key)` | OpenAI-compatible |
| Together AI | `OpenAiProvider::together(key)` | OpenAI-compatible |
| Fireworks AI | `OpenAiProvider::fireworks(key)` | OpenAI-compatible |
| Perplexity | `OpenAiProvider::perplexity(key)` | OpenAI-compatible |
| Ollama (local) | `OpenAiProvider::ollama()` | OpenAI-compatible |
| LM Studio (local) | `OpenAiProvider::lm_studio()` | OpenAI-compatible |
| Anything else OpenAI-compatible (vLLM, LiteLLM, …) | `OpenAiProvider::new(key).with_base_url(url)` | OpenAI-compatible |

Features:

- Tool calling with one normalized format for every provider
- Errors modeled on the official SDKs (`BadRequest`, `Authentication`, `RateLimit`, …, request IDs, retry rules)
- Client-side validation before any network call
- Safe defaults:
  - connect/request timeouts and a 32 MiB response cap
  - no redirects, so the key never goes to another host
  - keys are refused over plain `http://` to non-local hosts
  - keys are redacted from `Debug` and marked sensitive in headers
  - URLs are stripped from connection errors

## Install

```toml
[dependencies]
llm-universal = "0.2"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
```

## Quick start

```rust,no_run
use llm_universal::{AnthropicProvider, ChatRequest, LlmProvider, Message, OpenAiProvider};

#[tokio::main]
async fn main() -> llm_universal::Result<()> {
    let provider: Box<dyn LlmProvider> = match std::env::var("ANTHROPIC_API_KEY") {
        Ok(key) => Box::new(AnthropicProvider::new(key)),
        Err(_) => Box::new(OpenAiProvider::new(std::env::var("OPENAI_API_KEY").unwrap_or_default())),
    };

    let req = ChatRequest::new("claude-opus-5-5")
        .system("Be concise.")
        .max_tokens(512)
        .message(Message::user("Hello!"));

    let resp = provider.chat(&req).await?;
    println!("{}", resp.text());
    Ok(())
}
```

Local OpenAI-compatible server:

```rust,no_run
# use llm_universal::OpenAiProvider;
let ollama = OpenAiProvider::new("").with_base_url("http://localhost:11434/v1");
```

## Tool calling

```rust,no_run
# use llm_universal::*;
# use serde_json::json;
# async fn run(provider: &dyn LlmProvider) -> Result<()> {
let weather = Tool::new(
    "get_weather",
    "Get the current weather in a city",
    json!({ "type": "object", "properties": { "city": { "type": "string" } }, "required": ["city"] }),
);
let mut req = ChatRequest::new("claude-opus-5-5")
    .tool(weather)
    .message(Message::user("What's the weather in Berlin?"));

let resp = provider.chat(&req).await?;
if resp.stop_reason == StopReason::ToolUse {
    let calls = resp.tool_calls();
    req.messages.push(resp.message);
    let results = calls
        .into_iter()
        .map(|(id, _name, _args)| Part::ToolResult { call_id: id, content: "18°C, cloudy".into(), is_error: false })
        .collect();
    req.messages.push(Message::new(Role::Tool, results));
    let final_resp = provider.chat(&req).await?;
    println!("{}", final_resp.text());
}
# Ok(()) }
```

A complete loop is in [`examples/tools.rs`](examples/tools.rs): `cargo run --example tools`.

## Errors

| `LlmError` variant | When | SDK equivalent |
|---|---|---|
| `Api(Box<ApiError>)` | non-2xx response | `APIStatusError` and subclasses |
| `Connection` | DNS, TLS, dropped connection | `APIConnectionError` |
| `Timeout` | request timed out | `APIConnectionTimeoutError` |
| `Deserialize` | body is not JSON (`content` keeps the start of it; not printed) | `JSONDeserialize` (async-openai) |
| `InvalidResponse` | JSON is missing expected fields | — |
| `InvalidArgument` | client-side validation failed | `InvalidArgument` (async-openai) |

`ApiError` carries `status`, `kind` (`ApiErrorKind::RateLimit`, …), `message`, `error_type`,
`code`, `param`, `request_id` and the JSON `body`. Its message format matches the SDKs:

```text
401 invalid x-api-key
429 Rate limit reached for requests
invalid args: model must not be empty
```

```rust,no_run
# use llm_universal::*;
# async fn run(provider: &dyn LlmProvider, req: &ChatRequest) {
match provider.chat(req).await {
    Ok(resp) => println!("{}", resp.text()),
    Err(err) if err.is_retryable() => eprintln!("retry later: {err} (request id: {:?})", err.request_id()),
    Err(err) => eprintln!("{err}"),
}
# }
```

`is_retryable()` follows the SDK retry rules: 408, 409, 429, >= 500, connection errors and timeouts.

## Notes

- Anthropic newer models accept only `temperature = 1.0`; leave `temperature` unset unless you need it.
- Extended-thinking blocks are not part of the normalized message; if you enable thinking, keep the raw
  response to send signed thinking blocks back unchanged.
- Gemini tool calls without an id get a local id (`llm-universal-call-N`) that is never sent back to the API.
- Amazon Bedrock and Vertex AI need cloud-signed requests and are not supported yet.
- Streaming is not supported yet.

## License

[MIT-0](LICENSE) (MIT No Attribution): use, copy, modify and sell it for any purpose, commercial included, with no attribution required.