Crate multi_llm 

multi-llm

Unified multi-provider LLM client with support for OpenAI, Anthropic, Ollama, and LMStudio.

Key Features

• Multiple Providers: Seamless switching between LLM providers
• Unified Messages: Provider-agnostic message architecture with caching hints
• Prompt Caching: Native support for Anthropic prompt caching
• Tool Calling: First-class function/tool calling support
• Resilience: Built-in retry logic, rate limiting, and error handling

Example

use multi_llm::{UnifiedLLMClient, LLMConfig, OpenAIConfig, DefaultLLMParams, UnifiedMessage};

let config = LLMConfig {
    provider: Box::new(OpenAIConfig {
        api_key: Some("your-api-key".to_string()),
        base_url: "https://api.openai.com".to_string(),
        default_model: "gpt-4".to_string(),
        max_context_tokens: 128_000,
        retry_policy: Default::default(),
    }),
    default_params: DefaultLLMParams::default(),
};

let client = UnifiedLLMClient::from_config(config)?;
let messages = vec![UnifiedMessage::user("Hello, how are you?")];
// Use client.execute_llm(...) for actual requests

Re-exports

pub use client::UnifiedLLMClient;

pub use config::AnthropicConfig;

pub use config::DefaultLLMParams;

pub use config::LLMConfig;

pub use config::LMStudioConfig;

pub use config::OllamaConfig;

pub use config::OpenAIConfig;

pub use config::ProviderConfig;

pub use error::LlmError;

pub use error::LlmResult;

pub use messages::CacheType;

pub use messages::MessageAttributes;

pub use messages::MessageCategory;

pub use messages::MessageContent;

pub use messages::MessageRole;

pub use messages::UnifiedLLMRequest;

pub use messages::UnifiedMessage;

pub use provider::LlmProvider;

pub use provider::RequestConfig;

pub use provider::Response;

pub use provider::ResponseFormat;

pub use provider::TokenUsage;

pub use provider::Tool;

pub use provider::ToolCall;

pub use provider::ToolCallingRound;

pub use provider::ToolChoice;

pub use provider::ToolResult;

pub use providers::AnthropicProvider;

pub use providers::LMStudioProvider;

pub use providers::OllamaProvider;

pub use providers::OpenAIProvider;

Modules

client

Unified LLM client for multi-provider operations.

config

Configuration types for LLM providers.

error

Error types for LLM operations.

messages

Unified message architecture for LLM interactions.

provider

Provider trait and types for LLM abstraction.

providers

LLM Provider implementations

Macros

unwrap_response

Extract the Response from execute_llm results (non-events version).

Structs

AnthropicTokenCounter

Token counter for Anthropic Claude models.

OpenAITokenCounter

Token counter for OpenAI GPT models using tiktoken.

RetryPolicy

Retry policy configuration for LLM requests

TokenCounterFactory

Factory for creating token counters for different providers.

Traits

TokenCounter

Trait for counting tokens in text and messages.