litellm-rust

Minimal Rust SDK port of LiteLLM (library only). Provides a unified interface for chat, embeddings, images, and video across multiple LLM providers.

Note: This project is under active development. APIs may change.

Features

• Unified client for OpenAI-compatible, Anthropic, Gemini, xAI, and Z.AI providers
• Chat completions with streaming (SSE) support
• Tool calling — model-requested function calls surfaced on ChatResponse.tool_calls in OpenAI shape (Anthropic tool_use and Gemini functionCall normalized through the same shape)
• Text embeddings
• Image generation (OpenAI DALL-E / GPT Image)
• Video generation (Gemini Veo, OpenAI Sora)
• Model routing with provider/model format
• Automatic retry with exponential backoff
• Cost tracking via response headers
• Embedded model registry with pricing and context window data
• Daily model registry refreshes from LiteLLM's canonical catalog

Supported Providers

Installation

Prefer crates.io. The published crate is litellm-rust 0.4.0.

cargo add litellm-rust

Or add it to your Cargo.toml:

[dependencies]
litellm-rust = "0.4.0"

This is a library crate (no CLI). There is no cargo install binary.

To depend on unreleased main instead of the crates.io release:

[dependencies]
litellm-rust = { git = "https://github.com/avivsinai/litellm-rust" }

Quick Start

Astra Responses transport

Tool-enabled Astra requests use the opt-in OpenAIResponses provider kind. Keep the existing OpenAICompatible kind for Chat Completions and for cheaper or image models. Responses requests use max_completion_tokens (mapped to max_output_tokens); max_tokens is rejected. Streaming function-call items are exposed in each chunk's raw field so callers can accumulate argument deltas and send the resulting ChatRequest tool output with its call_id.

let client = LiteLLM::new()?.with_provider(
    "astra",
    ProviderConfig::default()
        .with_kind(ProviderKind::OpenAIResponses)
        .with_base_url("https://api.openai.com/v1")
        .with_api_key_env("OPENAI_API_KEY"),
);

let response = client
    .completion(ChatRequest::new("astra/gpt-6-astra").message("user", "hello"))
    .await?;

use litellm_rust::{
    LiteLLM, ProviderConfig, ProviderKind,
    ChatRequest, EmbeddingRequest, ImageRequest, VideoRequest,
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = LiteLLM::new()?
        .with_provider(
            "openai",
            ProviderConfig::default()
                .with_kind(ProviderKind::OpenAICompatible)
                .with_api_key_env("OPENAI_API_KEY"),
        )
        .with_provider(
            "gemini",
            ProviderConfig::default()
                .with_kind(ProviderKind::Gemini)
                .with_api_key_env("GEMINI_API_KEY"),
        );

    let resp = client
        .completion(ChatRequest::new("openai/gpt-4o").message("user", "hello"))
        .await?;
    println!("{}", resp.content);

    let embed = client
        .embedding(EmbeddingRequest {
            model: "openai/text-embedding-3-small".to_string(),
            input: serde_json::json!("hello"),
            extra_body: None,
        })
        .await?;
    println!("{} vectors", embed.vectors.len());

    let images = client
        .image_generation(ImageRequest {
            model: "openai/gpt-image-1.5".to_string(),
            prompt: "A cozy cabin in snow".to_string(),
            n: Some(1),
            size: None,
            quality: None,
            background: None,
            extra_body: None,
        })
        .await?;
    println!("{} images", images.images.len());

    let video = client
        .video_generation(VideoRequest {
            model: "openai/sora-2-pro".to_string(),
            prompt: "Drone flyover".to_string(),
            seconds: Some(5),
            size: None,
        })
        .await?;
    println!("video url: {:?}", video.video_url);

    Ok(())
}

Streaming

use futures_util::StreamExt;
use litellm_rust::{LiteLLM, ChatRequest};

# async fn run() -> anyhow::Result<()> {
let client = LiteLLM::new()?;
let mut stream = client
    .stream_completion(ChatRequest::new("openai/gpt-4o").message("user", "hello"))
    .await?;
while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    print!("{}", chunk.content);
}
# Ok(())
# }

Provider Configuration

Set API keys via environment variables:

Model routing uses provider/model format (e.g., openai/gpt-4o, openrouter/anthropic/claude-sonnet-4-5).

Minimum Supported Rust Version

The MSRV is Rust 1.88. This is verified in CI.

Notes

• xAI uses OpenAI-compatible endpoints. Configure provider xai with base URL https://api.x.ai/v1 and XAI_API_KEY.
• Z.AI uses OpenAI-compatible endpoints. The built-in zai route uses https://api.z.ai/api/paas/v4 and ZAI_API_KEY.
• Model IDs are passed through to providers rather than allow-listed. The embedded registry is used for pricing and context metadata and is refreshed daily by an automated pull request that merges itself once validation CI passes; failed refreshes surface as a GitHub issue and retry the next day.
• This crate intentionally excludes LiteLLM proxy/server features.

Contributing

See CONTRIBUTING.md for development setup and guidelines.

License

MIT - This project is a Rust port of LiteLLM by Berri AI (also MIT licensed).