# llmshim
A blazing-fast LLM API translation layer written in **pure Rust**. One request format, every provider — OpenAI, Anthropic, Google Gemini, and xAI.
Send an OpenAI-style request, pick any model, and llmshim translates it to that provider's native API (and translates the response back). Switch providers by changing one string.
**Three ways to use it:**
| **Rust crate** | Rust apps that call LLMs directly, in-process | `cargo add llmshim` |
| **CLI** | Interactive chat + a local proxy, from your terminal | `brew install sanjay920/tap/llmshim` |
| **HTTP proxy** | Any language (Python, JS, Go, …) over HTTP | run `llmshim proxy`, or `pip install llmshim` |
The Rust crate is the engine. The CLI and proxy wrap it. The Python package is a thin client that bundles the Rust binary, starts the proxy for you, and talks to it over HTTP — so you get the Rust engine behind a Python API.
## Benchmarks
Median of 5 runs, each p50 over 20 warm requests. Same prompt, same models, same machine.
| Anthropic (p50) | 999ms | 981ms | **973ms** |
| OpenAI (p50) | **511ms** | 613ms | 602ms |
| Streaming TTFT | 1,023ms | **906ms** | 1,249ms |
| Memory (RSS) | **12 MB** | 281 MB | 281 MB |
| Transform overhead | **1.5µs** | — | — |
All three libraries hit the same APIs (Responses API for OpenAI, Messages API for Anthropic), so latency is dominated by the network round-trip — llmshim's own translation work is ~1.5µs, roughly a millionth of the request time.[^bench] The differences between libraries on any single latency row are within network noise and reshuffle run-to-run; the durable wins are memory footprint (~24× leaner than the Python stacks) and near-zero startup/overhead.
Run it yourself:
```bash
cargo run --release --example bench # Rust (llmshim)
uv run --with litellm --with langchain-anthropic --with langchain-openai \
python benchmarks/bench_python.py # Python (litellm + langchain)
```
## Configure API keys
llmshim reads keys from environment variables or `~/.llmshim/config.toml`. Precedence: **env vars > config file**.
```bash
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=AIza...
export XAI_API_KEY=xai-...
```
Or persist them to the config file (used by all three surfaces):
```bash
llmshim configure # interactive prompt
```
---
## Use it from Rust
```bash
cargo add llmshim tokio serde_json
```
```rust
use serde_json::json;
#[tokio::main]
async fn main() {
// Router::from_env() picks up the *_API_KEY env vars.
let router = llmshim::router::Router::from_env();
let request = json!({
"model": "claude-sonnet-4-6",
"messages": [{"role": "user", "content": "What is Rust?"}],
"max_tokens": 500,
});
// Responses come back in OpenAI Chat Completions format.
let resp = llmshim::completion(&router, &request).await.unwrap();
println!("{}", resp["choices"][0]["message"]["content"]);
}
```
Switch providers by changing the `"model"` string — everything else stays the same.
**Streaming:**
```rust
use futures::StreamExt;
use serde_json::json;
let router = llmshim::router::Router::from_env();
let request = json!({
"model": "gpt-5.5",
"messages": [{"role": "user", "content": "Write a haiku about Rust."}],
"max_tokens": 128,
});
let mut stream = llmshim::stream(&router, &request).await.unwrap();
while let Some(Ok(chunk)) = stream.next().await {
let parsed: serde_json::Value = serde_json::from_str(&chunk).unwrap_or_default();
if let Some(text) = parsed.pointer("/choices/0/delta/content").and_then(|c| c.as_str()) {
print!("{text}");
}
}
```
See [`examples/chat.rs`](examples/chat.rs) and [`examples/stream.rs`](examples/stream.rs) for runnable programs (`cargo run --example chat`).
---
## Use it from the CLI
```bash
brew install sanjay920/tap/llmshim # macOS
cargo install llmshim --features proxy # from source (any platform)
```
```bash
llmshim # show help
llmshim chat # interactive multi-model chat (streaming, /model to switch)
llmshim configure # set API keys
llmshim set <key> <value> # set a config value
llmshim list # show configured keys
llmshim models # list available models
llmshim proxy # start the HTTP proxy (see below)
```
---
## Use it from any language (HTTP proxy)
Run llmshim as a local HTTP server and call it from any language. It has its own compact API (not OpenAI-shaped).
```bash
llmshim proxy
# Listening on http://localhost:3000
```
```bash
curl http://localhost:3000/v1/chat \
-H "Content-Type: application/json" \
-d '{"model":"claude-sonnet-4-6","messages":[{"role":"user","content":"Hi"}],"config":{"max_tokens":100}}'
```
| `POST` | `/v1/chat` | Chat completion (or streaming with `stream: true`) |
| `POST` | `/v1/chat/stream` | Always-streaming SSE with typed events |
| `GET` | `/v1/models` | List available models |
| `GET` | `/health` | Health check |
Full API spec: [`api/openapi.yaml`](api/openapi.yaml).
### Python client
`pip install llmshim` gives you a Python wrapper that bundles the Rust binary, starts the proxy on first use, and stops it on exit — no server to manage.
```bash
pip install llmshim
```
```python
import llmshim
# Keys can also come from env vars or `llmshim configure`.
llmshim.configure(anthropic="sk-ant-...", openai="sk-...")
resp = llmshim.chat("claude-sonnet-4-6", "Hello!", max_tokens=500)
print(resp["message"]["content"])
```
**Streaming:**
```python
for event in llmshim.stream("claude-sonnet-4-6", "Write a poem"):
if event["type"] == "content":
print(event["text"], end="", flush=True)
elif event["type"] == "usage":
print(f"\n[↑{event['input_tokens']} ↓{event['output_tokens']}]")
```
**Multi-model conversation** — switch providers mid-chat, history carries over:
```python
messages = [{"role": "user", "content": "What is a closure?"}]
r1 = llmshim.chat("claude-sonnet-4-6", messages, max_tokens=500)
print(f"Claude: {r1['message']['content']}")
messages.append({"role": "assistant", "content": r1["message"]["content"]})
messages.append({"role": "user", "content": "Now explain it differently."})
r2 = llmshim.chat("gpt-5.5", messages, max_tokens=500)
print(f"GPT: {r2['message']['content']}")
```
**Tool use** — pass tools in OpenAI Chat Completions format; llmshim translates to each provider's native format:
```python
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}]
resp = llmshim.chat("claude-sonnet-4-6", "Weather in Tokyo?", max_tokens=500, tools=tools)
for tc in resp["message"].get("tool_calls", []):
print(f"{tc['function']['name']}({tc['function']['arguments']})")
```
**Reasoning / thinking:**
```python
resp = llmshim.chat(
"claude-sonnet-4-6",
"Solve: x^2 - 5x + 6 = 0",
max_tokens=4000,
reasoning_effort="high",
)
print(resp["reasoning"]) # thinking content
print(resp["message"]["content"]) # answer
```
**Fallback chains** — automatic failover across providers:
```python
resp = llmshim.chat(
"anthropic/claude-sonnet-4-6",
"Hello",
max_tokens=100,
fallback=["openai/gpt-5.5", "gemini/gemini-3.5-flash"],
)
```
> These capabilities (streaming, multi-model, tools, reasoning, fallback) are all provided by the Rust core, so they work identically from the Rust crate and the proxy — the Python snippets above are just the most concise way to show them.
---
## Supported models
| **OpenAI** | `gpt-5.5`, `gpt-5.5-pro`, `gpt-5.4`, `gpt-5.4-pro`, `gpt-5.4-mini`, `gpt-5.4-nano` | Yes (summaries) |
| **Anthropic** | `claude-opus-4-8`, `claude-sonnet-5`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001` | Yes (full thinking) |
| **Google Gemini** | `gemini-3.5-flash`, `gemini-3.1-pro-preview`, `gemini-3.1-flash-lite-preview`, `gemini-3-flash-preview` | Yes (thought summaries) |
| **xAI** | `grok-4.3`, `grok-4.20-multi-agent-beta-0309`, `grok-4.20-beta-0309-reasoning`, `grok-4.20-beta-0309-non-reasoning`, `grok-4-1-fast-reasoning`, `grok-4-1-fast-non-reasoning` | No (hidden) |
Use a bare model name (auto-detected by prefix) or an explicit `provider/model` string.
## Docker
```bash
llmshim docker build
llmshim docker start
llmshim docker status
llmshim docker logs
llmshim docker stop
```
## How it works
No canonical struct. Requests flow as `serde_json::Value` — each provider maps only what it understands. Adding a provider = implementing one trait with three methods.
```
llmshim::completion(router, request)
→ router.resolve("anthropic/claude-sonnet-4-6")
→ provider.transform_request(model, &value)
→ HTTP
→ provider.transform_response(model, body)
```
## Key features
- **Multi-model conversations** — switch providers mid-chat, history carries over
- **Reasoning/thinking** — visible chain-of-thought from OpenAI, Anthropic, and Gemini
- **Streaming** — token-by-token, with thinking surfaced separately
- **Tool use** — Chat Completions format auto-translated to each provider
- **Vision/images** — send images in any format, auto-translated between providers
- **Fallback chains** — automatic failover across providers with exponential backoff
- **Cross-provider translation** — system messages, tool calls, and provider-specific fields all handled
## Build & test
```bash
cargo build # dev build
cargo build --release --features proxy # release build (~6MB binary)
cargo test --features proxy --tests # unit tests (~370)
cargo test --features proxy -- --ignored # integration tests (needs API keys)
```