Crate llm_universal 

One async client for many LLM APIs: Anthropic, OpenAI, Google Gemini, Cohere, Azure OpenAI, Amazon Bedrock, Vertex AI and every OpenAI-compatible server.

The same code (ChatRequest -> ChatResponse) runs against any provider; switching providers means switching the constructor:

let provider: Box<dyn LlmProvider> = Box::new(AnthropicProvider::new("sk-ant-..."));
// or: Box::new(OpenAiProvider::new("sk-...")),
// or: Box::new(GeminiProvider::new("AIza...")),
// or: Box::new(OpenAiProvider::ollama()),
let req = ChatRequest::new("claude-opus-5-5")
    .system("Be concise.")
    .max_tokens(512)
    .message(Message::user("Hello!"));
let resp = provider.chat(&req).await?;
println!("{}", resp.text());

// Streaming:
let mut stream = provider.chat_stream(&req).await?;
while let Some(event) = stream.next().await {
    if let StreamEvent::TextDelta(text) = event? {
        print!("{text}");
    }
}

Errors follow the official SDKs: see LlmError and ApiError. Failed requests are retried like the SDKs do (2 retries by default; see with_max_retries).

Structs

AnthropicProvider

Client for the Anthropic Messages API (POST {base_url}/v1/messages), or for Claude on Google Cloud Vertex AI via vertex.

ApiError

A response with a non-2xx status (APIStatusError in the SDKs).

ChatRequest

Provider-neutral chat request.

ChatResponse

CohereProvider

Client for the Cohere Chat API v2 (POST {base_url}/chat).

Embeddings

Result of an embeddings call: one vector per input, in input order.

GeminiProvider

Client for the Google Gemini API (POST {base_url}/models/{model}:generateContent, key in x-goog-api-key), or for Gemini on Vertex AI via vertex.

Message

ModelInfo

A model available to the key.

OpenAiProvider

Client for the OpenAI Chat Completions API (POST {base_url}/chat/completions).

ResponseFormat

Asks the model to answer with JSON matching schema.

Tool

A function the model may call. parameters is a JSON Schema with "type": "object".

ToolCall

A tool call taken out of a response.

Usage

Enums

ApiErrorKind

Error class by HTTP status, matching the APIError subclasses in the official SDKs.

ImageSource

Where an image comes from.

LlmError

Every error this crate returns.

Part

One piece of a message.

ReasoningEffort

How hard the model should think, on models that support reasoning.

Role

Author of a message.

StopReason

Why generation stopped, normalized across providers.

StreamEvent

What a stream yields. The last item is always StreamEvent::Done (or an error).

Constants

DEFAULT_MAX_RETRIES

Retries by default, like the official SDKs.

DEFAULT_MAX_TOKENS

max_tokens sent to Anthropic when the request does not set one.

Traits

EmbeddingProvider

Providers that can turn text into vectors (OpenAI and compatible servers, Gemini, Cohere).

LlmProvider

Common contract implemented by every provider.

StreamExt

An extension trait for Streams that provides a variety of convenient combinator functions.

Functions

run_tools

Runs the tool loop until the model gives a final answer.

Type Aliases

ChatStream

A stream of StreamEvents. Read it with StreamExt::next (re-exported by this crate).

Result