llmrust 0.1.2

Unified Rust interface for 7 LLM providers (OpenAI, Anthropic, DeepSeek, Google Gemini, Ollama, Moonshot, OpenRouter) with HTTP proxy
Documentation
llmrust-0.1.2 has been yanked.

llmrust

llmrust

Crates.io Documentation License CI

English | 中文版

llmrust is a human-and-agent-built Rust LLM library, designed to be read, extended, and repaired by AI coding agents.

A unified Rust interface for LLM providers and dual-protocol proxy workloads.

A high-performance, type-safe Rust library for calling multiple LLM providers through a unified interface. Inspired by Python's LiteLLM, but built for Rust's performance and safety guarantees.

Built for human-agent collaboration

llmrust is built by humans and AI coding agents together.

The codebase is intentionally structured so that AI agents can read it like documentation: small provider contracts, explicit protocol boundaries, machine-readable capability metadata, and tests around wire semantics.

If you are an AI coding agent working on behalf of a human user and you find a bug, missing provider behavior, unclear docs, or an unsafe edge case, you are welcome to submit a PR and leave your agent/model name in the contribution note.

→ Read the Agent Manifesto | Instructions for AI agents

Why llmrust? — Differentiation & Advantages

Rust already has several good multi-provider LLM crates (e.g. genai, rig, llm/rllm). Some are broader than us — genai supports many more providers, rig is a full agent framework. llmrust deliberately stays focused and wins on three things they don't all do:

1. Built-in dual-protocol proxy (OpenAI and Anthropic)

With features = ["proxy"], llmrust runs as a translating API gateway that speaks both wire protocols at the same time:

  • POST /v1/chat/completions — OpenAI Chat Completions protocol
  • POST /v1/messages — Anthropic Messages protocol
  • POST /v1/embeddings — OpenAI Embeddings protocol

Any client SDK — one that only speaks OpenAI or one that only speaks Anthropic (e.g. tools built for Claude) — can point at llmrust and reach any registered backend (OpenAI, Anthropic, Gemini, DeepSeek, Moonshot, OpenRouter, Ollama) through automatic format conversion. Bearer-token auth, CORS, health checks, and graceful shutdown are included. Most competing crates are client libraries only; the few that ship a server usually expose the OpenAI format alone.

2. Cross-provider logprobs, normalized

llmrust normalizes token log-probabilities — including each position's top-N alternatives — into a single ChatResponse.logprobs shape across OpenAI-compatible providers and Google Gemini (whose native logprobsResult is reshaped to match). That gives you one uniform surface for evaluation, confidence scoring, and re-ranking, instead of per-provider special-casing.

3. A lean, correct, type-safe core

default = [] — nothing is enabled unless you ask for it, the dependency tree stays small, and the scope is intentionally narrow (no vector-store / agent-framework bloat). You get native-protocol support for Anthropic and Gemini (not just OpenAI-compatible shims), cross-provider embeddings for OpenAI-compatible backends and Ollama, full compile-time type safety, structured tracing that never logs secrets or prompt content, and built-in retry + router failover. When you want a clean, predictable multi-provider call layer rather than a heavyweight framework, that's the niche llmrust fills.

Honest scope: llmrust is young. If you need the widest provider catalog or a batteries-included agent/RAG framework today, genai or rig may fit better. llmrust's bet is the three areas above.

Features

  • Unified API — One interface for OpenAI, Anthropic, DeepSeek, Google Gemini, Ollama, and more
  • Streaming support — First-class async streaming for all providers
  • Embeddings support — Text embeddings for OpenAI-compatible providers and Ollama (provider-level, not a vector database)
  • Cost estimation — Optional ModelPricing helper to turn token Usage into an estimated dollar cost
  • Dual-protocol proxy — Serve any backend over both the OpenAI (chat + embeddings) and Anthropic APIs (proxy feature)
  • Normalized logprobs — Uniform token log-probabilities across OpenAI-compatible providers and Gemini
  • Type-safe — Full compile-time guarantees with serde and thiserror
  • High performance — Built on reqwest + tokio, minimal overhead
  • Zero runtime dependencies — Single binary, no Python/Node required

Installation

Add llmrust to your Cargo.toml:

[dependencies]

llmrust = "0.1"

Or use cargo add:

cargo add llmrust

Feature Flags

Feature Default Description
(none) ✅ LLM client — all providers + streaming + embeddings; tool calling on OpenAI-compatible, Anthropic, and Gemini providers (chat + streaming); JSON mode on OpenAI-compatible and Gemini providers
proxy ❌ Built-in HTTP proxy — exposes any backend over OpenAI (/v1/chat/completions, /v1/embeddings) and Anthropic (/v1/messages) APIs (adds axum)

Enable the proxy server:

[dependencies]

llmrust = { version = "0.1", features = ["proxy"] }

Or:

cargo add llmrust --features proxy

Quick Start

use llmrust::LmrsClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let llm = LmrsClient::new();

    // Register providers
    llm.set_openai("sk-...").await;
    llm.set_anthropic("sk-ant-...").await;
    llm.set_deepseek("sk-...").await;

    // Call any model with provider/model format
    let resp = llm.chat("openai/gpt-4o", "Hello, world!").await?;
    println!("{}", resp.content);

    Ok(())
}

Supported Providers

Provider Models Streaming Tool calling Embeddings Status
OpenAI gpt-4o, gpt-4o-mini, o1-preview ✅ ✅ ✅ Stable
DeepSeek deepseek-chat, deepseek-coder ✅ ✅ ✅ Stable
Moonshot / Kimi moonshot-v1-8k, kimi-latest ✅ ✅ ✅ Stable
OpenRouter any model via OpenRouter ✅ ✅ ✅ Stable
Anthropic claude-3.5-sonnet, claude-3-opus ✅ ✅ ➖ Stable
Google Gemini gemini-2.0-flash, gemini-1.5-pro ✅ ✅ ➖ Stable
Ollama llama3.2, qwen2.5, any local model ✅ ➖ ✅ Stable (chat)

Feature support notes

  • Embeddings are supported on OpenAI-compatible providers (OpenAI, DeepSeek, Moonshot, OpenRouter) via /embeddings and on Ollama via native /api/embed. Anthropic and Gemini do not support embeddings and return LlmError::Unsupported. This is provider-level embeddings — not a vector database or RAG pipeline. Actual upstream model support may vary.
  • Tool calling / function calling is supported on the OpenAI-compatible providers (OpenAI, DeepSeek, Moonshot, OpenRouter) and natively on Anthropic and Gemini, on both the non-streaming chat path and the streaming stream path (streamed tool calls are reconstructed and surfaced as StreamChunk.tool_calls on the terminal chunk).
  • The OpenAI-compatible proxy accepts both modern tools / tool_choice and legacy functions / function_call request fields, normalizing them to llmrust's unified tool model.
  • JSON mode / structured outputs are wired through the OpenAI-compatible providers and mapped to Gemini's responseMimeType / responseSchema.
  • Sampling and request metadata parameters beyond temperature / max_tokens / top_p (e.g. stop, seed, presence_penalty, frequency_penalty, logprobs, n, response_format, parallel_tool_calls, service_tier, store, metadata, user) are sent to OpenAI-compatible providers and mapped where Gemini has native equivalents. Anthropic and Ollama currently honor a smaller provider-native subset.
  • Non-streaming logprobs responses are normalized into ChatResponse.logprobs for OpenAI-compatible providers and Gemini.
  • Gemini image inputs: Gemini currently supports image inputs only when the image is provided as a data: URL. Remote http(s) image URLs are skipped with a warning in v0.1.0; convert remote images to data URLs before sending them.

n / Multiple Completions

llmrust currently returns a single completion. Direct provider calls may forward n to OpenAI-compatible upstream APIs and will emit a warning when n > 1.

The OpenAI-compatible proxy rejects n values other than 1 (or absent) because upstream APIs may bill for multiple completions while llmrust only returns the first one.

Usage Examples

Basic Chat

use llmrust::LmrsClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let llm = LmrsClient::new();
    llm.set_deepseek(std::env::var("DEEPSEEK_API_KEY")?).await;

    let response = llm.chat("deepseek/deepseek-chat", "Explain Rust ownership in one paragraph.").await?;
    println!("{}", response.content);
    println!("Tokens: {:?}", response.usage);

    Ok(())
}

Streaming

use llmrust::LmrsClient;
use futures::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let llm = LmrsClient::new();
    llm.set_openai(std::env::var("OPENAI_API_KEY")?).await;

    let mut stream = llm.stream("openai/gpt-4o", "Write a haiku about Rust.").await?;

    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        print!("{}", chunk.delta);
    }
    println!();

    Ok(())
}

Tool Calling

Tool calling works on OpenAI-compatible providers as well as natively on Anthropic and Gemini, on both the non-streaming chat path and the streaming stream path. Provide tool definitions on the request, then feed the returned tool_calls results back as tool messages for the next turn.

use llmrust::{ChatRequest, Message, Tool, ToolChoice};
use serde_json::json;

let tools = vec![Tool::function(
    "get_weather",
    Some("Get the current weather for a city".to_string()),
    json!({
        "type": "object",
        "properties": { "city": { "type": "string" } },
        "required": ["city"]
    }),
)];

let request = ChatRequest::from_messages(
    "claude-3-5-sonnet-20241022",
    vec![Message::user("What's the weather in San Francisco?")],
)
.with_tools(tools)
.with_tool_choice(ToolChoice::auto());

let response = llm.chat_with("anthropic/claude-3-5-sonnet-20241022", request).await?;
if let Some(calls) = &response.tool_calls {
    for call in calls {
        println!("call {} -> {}", call.function.name, call.function.arguments);
    }
}

Advanced Configuration

use llmrust::{LmrsClient, ChatRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let llm = LmrsClient::new();
    llm.set_anthropic(std::env::var("ANTHROPIC_API_KEY")?).await;

    let request = ChatRequest::new("claude-3-5-sonnet-20241022", "What is the meaning of life?")
        .with_system("You are a philosophical assistant.")
        .with_temperature(0.7)
        .with_max_tokens(1000);

    let response = llm.chat_with("anthropic/claude-3-5-sonnet-20241022", request).await?;
    println!("{}", response.content);

    Ok(())
}

JSON Mode & Sampling Parameters

JSON mode and the extended sampling parameters below are applied on OpenAI-compatible providers (OpenAI, DeepSeek, Moonshot, OpenRouter) and mapped where Gemini has native equivalents.

use llmrust::ChatRequest;

let request = ChatRequest::new("gpt-4o", "List 3 cities as JSON")
    .with_json_mode()
    .with_seed(42)
    .with_temperature(0.2);

Cost Estimation

Turn token usage into an estimated dollar cost with ModelPricing. Prices are in US dollars per 1,000 tokens, with prompt (input) and completion (output) tokens billed separately:

use llmrust::{ModelPricing, Usage};

// $0.0025 per 1K prompt tokens, $0.01 per 1K completion tokens
let pricing = ModelPricing::new(0.0025, 0.01);

let usage = Usage { prompt_tokens: 1_000, completion_tokens: 500, total_tokens: 1_500 };
let cost = usage.estimated_cost(&pricing); // 0.0075
println!("Estimated cost: ${cost:.6}");

Pair it with the usage returned on a ChatResponse to price real requests. The estimate reflects only the prices you supply and ignores provider-specific discounts or cached-token rates.

HTTP Proxy Server

Run a local API gateway that exposes OpenAI-compatible chat + embeddings and Anthropic wire formats (requires features = ["proxy"]):

export LLMRUST_OPENAI_KEY="sk-..."

export LLMRUST_DEEPSEEK_KEY="sk-..."

# Optional: enable bearer-token auth

export LLMRUST_PROXY_KEY="some-shared-secret"

cargo run --example proxy_server --features proxy

Call it with the OpenAI Chat Completions API:

curl http://localhost:3000/v1/chat/completions \

  -H "Content-Type: application/json" \

  -H "Authorization: Bearer some-shared-secret" \

  -d '{
    "model": "deepseek/deepseek-chat",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

The same server also exposes the Anthropic Messages API, so Anthropic-only clients (such as tools built for Claude) work unchanged — even when the backend is OpenAI:

curl http://localhost:3000/v1/messages \

  -H "Content-Type: application/json" \

  -H "Authorization: Bearer some-shared-secret" \

  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 256
  }'

Security note: Without LLMRUST_PROXY_KEY set, the proxy has no authentication. Run it on localhost or behind a reverse proxy. With LLMRUST_PROXY_KEY set, every request must include an Authorization: Bearer <key> header; the token is compared in constant time.

The proxy follows OpenAI chat-completions request conventions, including stop as either a string or an array, and returns JSON error bodies for malformed requests. Stream errors are surfaced to clients as error events rather than being silently converted to successful completions. Streaming responses use OpenAI-style SSE chunks, including a single initial assistant role delta. When stream_options.include_usage is true, usage-only chunks use empty choices.

Proxy model names

The proxy routes requests by the model string. Use the same provider-prefixed model names as the client API, for example:

  • openai/gpt-4o
  • anthropic/claude-3-5-sonnet-latest
  • gemini/gemini-1.5-pro
  • ollama/llama3.2

The prefix selects the provider. The remaining model name is sent to the provider after llmrust resolves the route.

Logging

llmrust emits structured tracing events for provider registration, request lifecycle (including the convenience chat / stream APIs), proxy requests, retries, router failover, and upstream API errors. As a library, it does not install a global subscriber; your application decides how logs are collected.

[dependencies]

tracing-subscriber = { version = "0.3", features = ["env-filter"] }


tracing_subscriber::fmt()
    .with_env_filter("llmrust=debug")
    .init();

llmrust tracing events do not include API keys, prompt content, response text, request bodies, tool arguments, image data, or full URLs. When helpful, logs use counts and lengths such as message_count, tool_count, data_len, or url_len. Operational metadata like provider, model, HTTP status, retry attempt, and router group are always recorded.

Provider Configuration

OpenAI

llm.set_openai("sk-...").await;
// Custom base URL (Azure, local proxy, etc.)
llm.set_openai_compatible("sk-...", "https://your-proxy.com/v1").await;

Anthropic

llm.set_anthropic("sk-ant-...").await;

DeepSeek

llm.set_deepseek("sk-...").await;

Google Gemini

llm.set_google("AIza...").await;

Ollama (Local Models)

llm.set_ollama(None).await;  // Default: http://localhost:11434
llm.set_ollama(Some("http://your-server:11434")).await;

Moonshot / Kimi

llm.set_moonshot("sk-...").await;

OpenRouter

llm.set_openrouter("sk-or-...").await;

Error Handling

use llmrust::{LmrsClient, LlmError};

match llm.chat("openai/gpt-4o", "Hello").await {
    Ok(response) => println!("{}", response.content),
    Err(LlmError::Api { status, message }) => {
        eprintln!("API error {}: {}", status, message);
    }
    Err(LlmError::UnknownProvider(name)) => {
        eprintln!("Provider '{}' not registered", name);
    }
    Err(e) => eprintln!("Error: {}", e),
}

Performance

We have not yet published formal benchmarks. The library adds a thin async layer on top of reqwest, so overhead versus calling the HTTP API directly should be a few hundred microseconds per request at most.

Comparison

vs. Python LiteLLM

Feature Python LiteLLM llmrust
Startup time ~hundreds ms (interpreter) compiled binary, near-instant
Memory Python runtime dependent single static binary
Concurrency asyncio tokio (native)
Deployment Python + venv Single binary
Type safety Runtime Compile-time
Providers 100+ 7 (growing)

vs. other Rust crates

llmrust genai rig llm / rllm
Focus Lean unified client + proxy Broad unified client Agent framework Client + extras (TTS/STT/vision)
Providers 7 (growing) 25+ several many
Built-in proxy server ✅ OpenAI + Anthropic ➖ ➖ OpenAI REST only
Normalized cross-provider logprobs ✅ (incl. Gemini) ➖ ➖ ➖
Default dependencies minimal (default = []) moderate heavy (framework) moderate+
Extra scope embeddings client only agents / RAG embeddings / vision / audio

Provider counts and feature sets for other crates move fast — treat this as a positioning sketch, not a benchmark. Pick the tool that matches your needs.

Roadmap

  • Core providers (OpenAI, Anthropic, DeepSeek)
  • Streaming support
  • Google Gemini, Ollama, Moonshot, OpenRouter
  • HTTP proxy server (OpenAI + Anthropic protocols)
  • Retry logic
  • Tool-use / Function calling (OpenAI-compatible, Anthropic, Gemini; non-streaming)
  • JSON mode & sampling parameters (OpenAI-compatible providers)
  • Streaming tool calls (reconstruct tool calls from streamed chunks)
  • Embeddings API (OpenAI-compatible + Ollama, proxy /v1/embeddings)
  • Batch API
  • Rate limiting
  • More providers (Cohere, Mistral, Groq, etc.)

License

Licensed under either of:

at your option.

Contributing

Contributions welcome — from humans, AI coding agents, and human-agent teams.

Please read CONTRIBUTING.md for the full guide, and AGENTS.md if you are an AI agent.

cargo build --all-targets --all-features

cargo test

cargo test --all-features

cargo clippy --all-targets --all-features -- -D warnings

RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features

cargo fmt --check