<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-dark.svg">
<img alt="Xberg" width="420" src="https://cdn.jsdelivr.net/gh/xberg-io/assets@v1/banner/readme-banner-light.svg">
</picture>
</p>
# Rust
<div align="center" style="display: flex; flex-wrap: wrap; gap: 8px; justify-content: center; margin: 20px 0">
<a href="https://github.com/xberg-io/alef">
<img src="https://img.shields.io/badge/built%20with-alef%20%D7%90-007ec6" alt="Built with alef" />
</a>
<a href="https://crates.io/crates/liter-llm">
<img src="https://img.shields.io/crates/v/liter-llm?label=Rust&color=007ec6" alt="Rust" />
</a>
<a href="https://pypi.org/project/liter-llm/">
<img src="https://img.shields.io/pypi/v/liter-llm?label=Python&color=007ec6" alt="Python" />
</a>
<a href="https://www.npmjs.com/package/@xberg-io/liter-llm">
<img src="https://img.shields.io/npm/v/@xberg-io/liter-llm?label=Node.js&color=007ec6" alt="Node.js" />
</a>
<a href="https://www.npmjs.com/package/@xberg-io/liter-llm-wasm">
<img src="https://img.shields.io/npm/v/@xberg-io/liter-llm-wasm?label=WASM&color=007ec6" alt="WASM" />
</a>
<a href="https://central.sonatype.com/artifact/io.xberg.literllm/liter-llm">
<img src="https://img.shields.io/maven-central/v/io.xberg.literllm/liter-llm?label=Java&color=007ec6" alt="Java" />
</a>
<a href="https://github.com/xberg-io/liter-llm/tree/main/packages/go">
<img src="https://img.shields.io/github/v/tag/xberg-io/liter-llm?label=Go&color=007ec6" alt="Go" />
</a>
<a href="https://www.nuget.org/packages/XbergIo.LiterLlm">
<img src="https://img.shields.io/nuget/v/XbergIo.LiterLlm?label=C%23&color=007ec6" alt="C#" />
</a>
<a href="https://packagist.org/packages/xberg-io/liter-llm">
<img src="https://img.shields.io/packagist/v/xberg-io/liter-llm?label=PHP&color=007ec6" alt="PHP" />
</a>
<a href="https://rubygems.org/gems/liter_llm">
<img src="https://img.shields.io/gem/v/liter_llm?label=Ruby&color=007ec6" alt="Ruby" />
</a>
<a href="https://hex.pm/packages/liter_llm">
<img src="https://img.shields.io/hexpm/v/liter_llm?label=Elixir&color=007ec6" alt="Elixir" />
</a>
<a href="https://github.com/xberg-io/liter-llm/pkgs/container/liter-llm">
<img src="https://img.shields.io/badge/Docker-007ec6?logo=docker&logoColor=white" alt="Docker" />
</a>
<a href="https://github.com/xberg-io/homebrew-tap/blob/main/Formula/liter-llm.rb">
<img src="https://img.shields.io/badge/Homebrew-007ec6?logo=homebrew&logoColor=white" alt="Homebrew" />
</a>
<a href="https://github.com/xberg-io/scoop-bucket/blob/main/bucket/liter-llm.json">
<img src="https://img.shields.io/badge/Scoop-007ec6" alt="Scoop" />
</a>
<a href="https://github.com/xberg-io/liter-llm/tree/main/crates/liter-llm-ffi">
<img src="https://img.shields.io/badge/C-FFI-007ec6" alt="C FFI" />
</a>
<a href="https://github.com/xberg-io/liter-llm/blob/main/LICENSE">
<img src="https://img.shields.io/badge/License-MIT-007ec6" alt="License" />
</a>
<a href="https://docs.liter-llm.xberg.io">
<img src="https://img.shields.io/badge/Docs-liter--llm-007ec6" alt="Docs" />
</a>
</div>
<div align="center" style="margin: 24px 0 0">
<a href="https://xberg.io">
<img
alt="xberg.io"
src="https://github.com/user-attachments/assets/1b6c6ad7-3b6d-4171-b1c9-f2026cc9deb8"
/>
</a>
</div>
<div align="center" style="display: flex; flex-wrap: wrap; gap: 12px; justify-content: center; margin: 28px 0 24px">
<a href="https://discord.gg/xt9WY3GnKR">
<img
height="22"
src="https://img.shields.io/badge/Discord-Chat-007ec6?logo=discord&logoColor=white"
alt="Join Discord"
/>
</a>
</div>
Universal LLM API client for Rust. Access 165 LLM providers through a single unified interface. Async/await with Tokio, streaming via BoxStream, composable Tower middleware stack, and compile-time type safety.
## What This Package Provides
- **One provider surface** — chat, streaming, embeddings, images, audio, search, OCR, tools, and structured output across the provider registry.
- **Provider/model routing** — call models with the `provider/model` convention and keep provider-specific request code out of application paths.
- **Production controls** — retries, fallback, rate limits, cache layers, budgets, health checks, OpenTelemetry spans, and redacted secrets.
- **Same core as every binding** — Rust, Python, Node.js, Go, Java, PHP, Ruby, .NET, Elixir, WASM, Kotlin Android, Swift, Dart, Zig, and C FFI use the same Rust implementation.
- **Rust crate** — canonical async client with Tower middleware and Tokio integration.
## Installation
### Package Installation
Add the crate to your project:
```bash
cargo add liter-llm
```
Or add it to your `Cargo.toml`:
```toml
[dependencies]
liter-llm = "2.0.0"
```
### System Requirements
- API keys via environment variables (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
## Quick Start
### Basic Chat
Send a message to any provider using the `provider/model` prefix:
```rust
use liter_llm::{
ChatCompletionRequest, ClientConfigBuilder, DefaultClient, LlmClient,
Message, UserContent, UserMessage,
};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let config = ClientConfigBuilder::new(std::env::var("OPENAI_API_KEY")?)
.build();
let client = DefaultClient::new(config, Some("openai/gpt-4o"))?;
let request = ChatCompletionRequest {
model: "openai/gpt-4o".into(),
messages: vec![Message::User(UserMessage {
content: UserContent::Text("Hello!".into()),
name: None,
})],
..Default::default()
};
let response = client.chat(request).await?;
if let Some(choice) = response.choices.first() {
let text = choice.message.content.as_ref().and_then(|content| content.as_text());
println!("{}", text.unwrap_or_default());
}
Ok(())
}
```
### Common Use Cases
#### Streaming Responses
Stream tokens in real time:
```rust
use futures::StreamExt;
use liter_llm::{
ChatCompletionRequest, ClientConfigBuilder, DefaultClient, LlmClient,
Message, UserContent, UserMessage,
};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let config = ClientConfigBuilder::new(std::env::var("OPENAI_API_KEY")?)
.build();
let client = DefaultClient::new(config, Some("openai/gpt-4o"))?;
let request = ChatCompletionRequest {
model: "openai/gpt-4o".into(),
messages: vec![Message::User(UserMessage {
content: UserContent::Text("Tell me a story".into()),
name: None,
})],
..Default::default()
};
let mut stream = client.chat_stream(request).await?;
while let Some(chunk) = stream.next().await {
let chunk = chunk?;
if let Some(choice) = chunk.choices.first() {
if let Some(content) = &choice.delta.content {
print!("{content}");
}
}
}
println!();
Ok(())
}
```
### Next Steps
- **[Provider Registry](https://github.com/xberg-io/liter-llm/blob/main/schemas/providers.json)** - Full list of supported providers
- **[GitHub Repository](https://github.com/xberg-io/liter-llm)** - Source, issues, and discussions
## Features
### Supported Providers (165)
Route to any provider using the `provider/model` prefix convention:
| **OpenAI** | `openai/gpt-4o`, `openai/gpt-4o-mini` |
| **Anthropic** | `anthropic/claude-3-5-sonnet-20241022` |
| **Groq** | `groq/llama-3.1-70b-versatile` |
| **Mistral** | `mistral/mistral-large-latest` |
| **Cohere** | `cohere/command-r-plus` |
| **Together AI** | `together/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo` |
| **Fireworks** | `fireworks/accounts/fireworks/models/llama-v3p1-70b-instruct` |
| **Google Vertex** | `vertexai/gemini-1.5-pro` |
| **Amazon Bedrock** | `bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0` |
**[Complete Provider List](https://github.com/xberg-io/liter-llm/blob/main/schemas/providers.json)**
### Key Capabilities
- **Provider Routing** -- Single client for 165 LLM providers via `provider/model` prefix
- **Local LLMs** — Connect to locally-hosted models via Ollama, LM Studio, vLLM, llama.cpp, and other local inference servers
- **Unified API** -- Consistent `chat`, `chat_stream`, `embeddings`, `list_models` interface
- **Streaming** -- Real-time token streaming via `chat_stream`
- **Tool Calling** -- Function calling and tool use across all supporting providers
- **Type Safe** -- Schema-driven types compiled from JSON schemas
- **Secure** -- API keys never logged or serialized, managed via environment variables
- **Observability** -- Built-in [OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/gen-ai/) with GenAI semantic conventions
- **Error Handling** -- Structured errors with provider context and retry hints
### Performance
Built on a compiled Rust core for speed and safety:
- **Provider resolution** at client construction -- zero per-request overhead
- **Configurable timeouts** and connection pooling
- **Zero-copy streaming** with SSE and AWS EventStream support
- **API keys** wrapped in secure memory, zeroed on drop
## Provider Routing
Route to 165 providers using the `provider/model` prefix convention:
```text
openai/gpt-4o
anthropic/claude-3-5-sonnet-20241022
groq/llama-3.1-70b-versatile
mistral/mistral-large-latest
```
See the [provider registry](https://github.com/xberg-io/liter-llm/blob/main/schemas/providers.json) for the full list.
## Proxy, MCP Server & Plugin
<details>
<summary><strong>Run the OpenAI-compatible proxy or the MCP server</strong></summary>
Beyond the SDK, the `liter-llm` CLI ships an OpenAI-compatible proxy and a Model Context Protocol (MCP) server:
```bash
brew install xberg-io/tap/liter-llm # or: cargo install liter-llm-cli
# Windows: scoop bucket add xberg https://github.com/xberg-io/scoop-bucket && scoop install liter-llm
liter-llm api --config liter-llm-proxy.toml # OpenAI-compatible proxy
liter-llm mcp --transport stdio # MCP tool server
# or run the proxy without installing:
docker run -p 4000:4000 -e LITER_LLM_MASTER_KEY=sk-your-key ghcr.io/xberg-io/liter-llm
```
To use the MCP server inside a coding agent, install the **liter-llm plugin** from the [`xberg-io/plugins`](https://github.com/xberg-io/plugins) marketplace — it auto-registers the server. See the [MCP server](https://docs.liter-llm.xberg.io/server/mcp-server/) and [proxy server](https://docs.liter-llm.xberg.io/server/proxy-server/) guides for configuration, CLI usage, and agent integration.
</details>
## Documentation
- **[Documentation](https://docs.liter-llm.xberg.io)** -- Full docs and API reference
- **[GitHub Repository](https://github.com/xberg-io/liter-llm)** -- Source, issues, and discussions
- **[Provider Registry](https://github.com/xberg-io/liter-llm/blob/main/schemas/providers.json)** -- 165 supported providers
## Part of Xberg.io
- [Xberg](https://github.com/xberg-io/xberg) — document intelligence: text, tables, metadata from 101 formats with optional OCR.
- [Xberg Enterprise](https://github.com/xberg-io/xberg-enterprise) — managed extraction API with SDKs, dashboards, and observability.
- [crawlberg](https://github.com/xberg-io/crawlberg) — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
- [html-to-markdown](https://github.com/xberg-io/html-to-markdown) — fast, lossless HTML→Markdown engine.
- [liter-llm](https://github.com/xberg-io/liter-llm) — universal LLM API client with native bindings for 14 languages and 165 providers.
- [tree-sitter-language-pack](https://github.com/xberg-io/tree-sitter-language-pack) — tree-sitter grammars and code-intelligence primitives.
- [alef](https://github.com/xberg-io/alef) — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
- [Discord](https://discord.gg/xt9WY3GnKR) — community, roadmap, announcements.
## Contributing
Contributions are welcome! See [CONTRIBUTING.md](https://github.com/xberg-io/liter-llm/blob/main/CONTRIBUTING.md) for guidelines.
Join our [Discord community](https://discord.gg/xt9WY3GnKR) for questions and discussion.
## License
MIT -- see [LICENSE](https://github.com/xberg-io/liter-llm/blob/main/LICENSE) for details.