Crate litellm_rs

Expand description

§LiteLLM-RS

A Rust implementation of Python LiteLLM - call 100+ LLM APIs using OpenAI format. High-performance AI Gateway with unified interface for multiple providers.

§Features

Python LiteLLM Compatible: Drop-in replacement with same API design
OpenAI Compatible: Full compatibility with OpenAI API format
Multi-Provider: Support for 100+ AI providers (OpenAI, Anthropic, Azure, Google, etc.)
Unified Interface: Call any LLM using the same function signature
High Performance: Built with Rust and Tokio for maximum throughput
Intelligent Routing: Smart load balancing and failover across providers
Cost Optimization: Automatic cost tracking and provider selection
Streaming Support: Real-time response streaming

§Quick Start - Python LiteLLM Style

use litellm_rs::{completion, user_message, system_message};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Call OpenAI (default provider for gpt-* models)
    let response = completion(
        "gpt-4",
        vec![
            system_message("You are a helpful assistant."),
            user_message("Hello, how are you?"),
        ],
        None,
    ).await?;
     
    if let Some(content) = &response.choices[0].message.content {
        println!("Response: {}", content);
    }

    // Call Anthropic with explicit provider
    let response = completion(
        "anthropic/claude-3-sonnet-20240229",
        vec![user_message("What is the capital of France?")],
        None,
    ).await?;
     
    if let Some(content) = &response.choices[0].message.content {
        println!("Claude says: {}", content);
    }
     
    Ok(())
}

§Gateway Mode

Requires the gateway feature (enabled by default via storage):

use litellm_rs::{Gateway, Config};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = Config::from_file("config/gateway.yaml").await?;
    let gateway = Gateway::new(config).await?;
    gateway.run().await?;
    Ok(())
}

Re-exports§

pub use config::Config;
pub use utils::error::gateway_error::GatewayError;
pub use utils::error::gateway_error::Result;
pub use version::BuildInfo;
pub use version::GIT_HASH;
pub use version::VERSION;
pub use version::build_info;
pub use version::full_version;
pub use core::completion::Choice;
pub use core::completion::CompletionOptions;
pub use core::completion::CompletionResponse;
pub use core::completion::ContentPart;
pub use core::completion::LiteLLMError;
pub use core::completion::Message;
pub use core::completion::Router;
pub use core::completion::Usage;
pub use core::completion::acompletion;
pub use core::completion::assistant_message;
pub use core::completion::completion;
pub use core::completion::completion_stream;
pub use core::completion::system_message;
pub use core::completion::user_message;
pub use core::embedding::EmbeddingInput;
pub use core::embedding::EmbeddingOptions;
pub use core::embedding::EmbeddingResponse;
pub use core::embedding::aembedding;
pub use core::embedding::cosine_similarity;
pub use core::embedding::dot_product;
pub use core::embedding::embed_text;
pub use core::embedding::embed_texts;
pub use core::embedding::embed_texts_with_options;
pub use core::embedding::embedding;
pub use core::embedding::euclidean_distance;
pub use core::embedding::normalize;
pub use core::streaming::types::ChatCompletionChunk;
pub use core::streaming::types::ChatCompletionChunkChoice;
pub use core::streaming::types::ChatCompletionDelta;
pub use core::types::message::MessageContent;
pub use core::types::message::MessageRole;
pub use core::models::RequestContext;
pub use core::models::openai::AudioContent;
pub use core::models::openai::AudioDelta;
pub use core::models::openai::AudioParams;
pub use core::models::openai::CacheControl;
pub use core::models::openai::ChatChoice;
pub use core::models::openai::ChatChoiceDelta;
pub use core::models::openai::ChatCompletionChoice;
pub use core::models::openai::ChatCompletionRequest;
pub use core::models::openai::ChatCompletionResponse;
pub use core::models::openai::ChatMessage;
pub use core::models::openai::ChatMessageDelta;
pub use core::models::openai::CompletionChoice;
pub use core::models::openai::CompletionRequest;
pub use core::models::openai::CompletionTokensDetails;
pub use core::models::openai::ContentLogprob;
pub use core::models::openai::DocumentSource;
pub use core::models::openai::EmbeddingObject;
pub use core::models::openai::EmbeddingRequest;
pub use core::models::openai::EmbeddingUsage;
pub use core::models::openai::Function;
pub use core::models::openai::FunctionCall;
pub use core::models::openai::FunctionCallDelta;
pub use core::models::openai::ImageGenerationRequest;
pub use core::models::openai::ImageGenerationResponse;
pub use core::models::openai::ImageObject;
pub use core::models::openai::ImageSource;
pub use core::models::openai::ImageUrl;
pub use core::models::openai::Logprobs;
pub use core::models::openai::Model;
pub use core::models::openai::ModelListResponse;
pub use core::models::openai::PromptTokensDetails;
pub use core::models::openai::ResponseFormat;
pub use core::models::openai::StreamOptions;
pub use core::models::openai::Tool;
pub use core::models::openai::ToolCall;
pub use core::models::openai::ToolCallDelta;
pub use core::models::openai::ToolChoice;
pub use core::models::openai::ToolChoiceFunction;
pub use core::models::openai::ToolChoiceFunctionSpec;
pub use core::models::openai::TopLogprob;
pub use core::providers::Provider;
pub use core::providers::ProviderError;
pub use core::providers::ProviderRegistry;
pub use core::providers::ProviderType;
pub use core::router::CooldownReason;
pub use core::router::Deployment;
pub use core::router::DeploymentConfig;
pub use core::router::FallbackConfig;
pub use core::router::FallbackType;
pub use core::router::RouterConfig;
pub use core::router::RouterError;
pub use core::router::UnifiedRouter;
pub use core::router::UnifiedRoutingStrategy as RoutingStrategy;

Modules§

config: Configuration management for the Gateway
core: Core functionality for the Gateway
monitoring: Monitoring and observability system
sdk: Unified LLM Provider SDK
server: HTTP server implementation
services: Services module
storage: Storage layer for the Gateway
utils: Utility modules for the LiteLLM Gateway
version: Build and version information

Macros§

build_request: Macro to implement standard HTTP request builder
define_extended_error_mapper: Generate an extended ErrorMapper implementation for a provider.
define_http_provider_with_hooks: Macro to define HTTP-based providers with custom hooks for request/response handling.
define_openai_compatible_provider: Macro to define OpenAI-compatible providers with shared boilerplate.
define_pooled_http_provider_with_hooks: Macro to define pooled HTTP providers that use GlobalPoolManager with custom hooks.
define_provider_config: Configuration
define_provider_error_helpers: Generate standard provider error helper functions.
define_standalone_provider_config: Macro for standalone provider configs that don’t use BaseConfig.
define_standard_error_mapper: Generate a standard ErrorMapper implementation for a provider.
extract_usage: Macro to implement usage extraction from response
global_shared: Macro to create a global shared resource
impl_error_conversion: Macro to implement error conversion for all provider errors This eliminates the 15 repetitive From implementations
impl_from_reqwest_error: Implement From<reqwest::Error> for an error type with timeout/connect/other branching.
impl_from_serde_error: Implement From<serde_json::Error> for an error type.
impl_health_check: Macro to implement health check using a simple API call
impl_provider_basics: Macro to implement common provider methods
impl_provider_error_helpers: Generate standard provider error methods on ProviderError.
impl_streaming: Macro to implement streaming response handler This eliminates the repetitive 20-line streaming handler pattern
model_list: Macro to generate model list
not_implemented: Macro to implement not-implemented methods
provider_config: Macro to generate provider configuration struct
require_config: Macro to extract required configuration value with provider context
safe_unwrap: Macro for safe unwrapping with context
safe_unwrap_option: Macro for safe option unwrapping with context
standard_provider: Macro to create standard provider implementation
validate_response: Macro to validate required fields in a response
with_retry: Macro to implement retry logic

Structs§

Gateway: A minimal LiteLLM Gateway implementation

Constants§

DESCRIPTION: Description of the crate
NAME: Name of the crate

Crate litellm_rs

Crate litellm_rs Copy item path

§LiteLLM-RS

§Features

§Quick Start - Python LiteLLM Style

§Gateway Mode

Re-exports§

Modules§

Macros§

Structs§

Constants§

Crate litellm_rs