llmshim 0.7.2

Blazing fast LLM API translation layer in pure Rust
Documentation

llmshim

A blazing-fast LLM API translation layer written in pure Rust. One request format, every provider — OpenAI, ChatGPT subscriptions, Anthropic, Google Gemini, xAI, OpenRouter, and self-hosted vLLM / SGLang.

Send an OpenAI-style request, pick any model, and llmshim translates it to that provider's native API (and translates the response back). Switch providers by changing one string.

Three ways to use it:

Surface For Install
Rust crate Rust apps that call LLMs directly, in-process cargo add llmshim
CLI Interactive chat + a local proxy, from your terminal brew install sanjay920/tap/llmshim
HTTP proxy Any language (Python, JS, Go, …) over HTTP run llmshim proxy, or pip install llmshim

The Rust crate is the engine. The CLI and proxy wrap it. The Python package is a thin client that bundles the Rust binary, starts the proxy for you, and talks to it over HTTP — so you get the Rust engine behind a Python API.

Benchmarks

Measured on 2026-09-18 UTC (September 17 Pacific), Apple M5 Pro, macOS 26.4.1, Rust 1.97.0, llmshim 0.4.0 in release mode. These are measurements of the current Rust implementation.

The API run used claude-sonnet-4-6 and gpt-5.4, the prompt Say 'benchmark' and nothing else., a 50-token output cap, no tools, and reasoning_effort: none. Each warm row summarizes 20 complete responses from one run; p50 uses the upper median. The streaming row is one completed stream.

Metric Measured result
Anthropic first request, after connection warmup 1426.84 ms
Anthropic warm p50 / mean 934.64 / 1111.58 ms
OpenAI warm p50 / mean 731.79 / 790.89 ms
Anthropic streaming time to first visible text 872.01 ms
Process RSS after API calls, before the tool sweep 85.48 MiB

A separate local benchmark measures request transformation with distinct tool parameter schemas reused across 10,000 iterations. It includes request copying, replay/cache policy, schema handling, and native translation. HTTP serialization, dispatch, and the higher-level capability plan are outside this measurement. The full synthetic fixture is in benchmarks/bench.rs.

Tools Schema cache disabled Warm schema cache
0 6.48 µs 5.88 µs
1 17.09 µs 10.01 µs
5 61.44 µs 25.55 µs
10 116.74 µs 45.03 µs
25 277.37 µs 106.01 µs
50 559.19 µs 210.39 µs

CPU values are means from separate runs; small differences, including the zero-tool row, reflect run-to-run variation.

At 50 tools, memoization makes this full transform 2.66× faster. The remaining 210.39 µs includes hashing and copying; caching removes repeated schema walking and validation compilation. For scale, that is about 0.023% of the measured one-word, zero-tool Anthropic response time above. End-to-end latency includes network and model work, and varies with payload, provider load, region, and time of day.

Run it yourself:

# Reads ANTHROPIC_API_KEY and OPENAI_API_KEY from env or ~/.llmshim/config.toml.
# Makes 43 logical model requests; the normal client retry policy applies.
cargo run --release --example bench

# Local CPU measurements; no keys or HTTP requests.
cargo run --release --example bench -- --transforms-only
LLMSHIM_NO_SCHEMA_CACHE=1 cargo run --release --example bench -- --transforms-only

Configure API keys

llmshim reads keys from environment variables or ~/.llmshim/config.toml. Precedence: env vars > config file.

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=AIza...
export XAI_API_KEY=xai-...
export OPENROUTER_API_KEY=sk-or-...

Reach any model through OpenRouter by addressing it as openrouter/<vendor>/<model> (e.g. openrouter/anthropic/claude-sonnet-5). OpenRouter is OpenAI Chat Completions-compatible, so tools, vision, streaming, and reasoning_effort all pass through; OpenRouter-only controls (provider routing, model fallbacks, transforms) go under an x-openrouter key.

Point at a self-hosted vLLM or SGLang server (local or remote) by setting its base URL — no key needed unless the server was launched with one:

export SGLANG_BASE_URL=http://localhost:30000/v1   # or https://your-host/v1
export VLLM_BASE_URL=http://localhost:8000/v1

Then address the served model as sglang/<served-model> or vllm/<served-model> (e.g. sglang/Qwen/Qwen3.6-35B-A3B-FP8). Server-specific knobs go under x-vllm / x-sglang.

Or persist them to the config file (used by all three surfaces):

llmshim configure          # interactive prompt

ChatGPT subscription (OAuth)

Sign in once with a ChatGPT account, then use chatgpt/<model> from Rust, the CLI, or any proxy client. No OPENAI_API_KEY is needed for this route. The device-code flow follows LiteLLM's ChatGPT provider.

llmshim login chatgpt             # open the printed URL and enter the code
llmshim login chatgpt --status    # inspect the saved login without network access
llmshim chat                     # select a ChatGPT model
# or: llmshim proxy

If needed, enable device-code login in your ChatGPT security settings or workspace permissions (OpenAI authentication docs).

{
  "model": "chatgpt/gpt-6-astra",
  "messages": [{"role": "user", "content": "Hello!"}]
}

Tokens live in ~/.llmshim/chatgpt/auth.json; expired access tokens refresh automatically, with file locking and atomic saves. CHATGPT_TOKEN_DIR and CHATGPT_AUTH_FILE select a different cache (LiteLLM's flat auth-file format is supported). This cache is independent of Codex's login. Run llmshim logout chatgpt to remove the selected local cache; this does not revoke the session at OpenAI. Login is explicit, never started inside a proxy request. Restart an existing proxy after the first login so it registers the provider. For containers, mount the token directory writable and set CHATGPT_TOKEN_DIR to its container path.

Both completion and streaming calls use the subscription Responses backend. Non-streaming calls collect the upstream stream into one normal response. Tools, images, and reasoning use the existing Responses translation; x-chatgpt supplies supported native fields (under provider_config in proxy requests). The backend requires store: false and stream: true and rejects token limits, sampling fields, and metadata, so these constraints also apply to native overrides. Bare gpt-* names still route to API-key OpenAI. The ChatGPT route supports only chatgpt/gpt-6-astra, chatgpt/gpt-5.6-sol, chatgpt/gpt-5.6-terra, and chatgpt/gpt-5.6-luna. Older and unlisted model IDs return a local error before authentication or an upstream request. Access to these models and usage limits depend on the ChatGPT account.

See provider configuration for endpoint and header overrides.

Endpoint redirects

The shared HTTP client does not follow redirects. Configure the final API URL directly: a 3xx response is returned as a provider error instead of forwarding the prompt and provider-specific credential headers to another endpoint. This applies to both streaming and non-streaming requests.

Use it from Rust

Non-streaming OpenAI Responses, xAI Responses, Gemini, and Anthropic results require supported terminal metadata. Missing, malformed, or unsupported statuses return a ProviderError (502) with a fixed diagnostic instead of being interpreted as successful completion. Known completion, output-limit, filtering and Anthropic tool-call mappings remain available. Additional native reasons require an explicit mapping; streaming transforms are unchanged. Provider mocks should include the native terminal status/stop reason.

cargo add llmshim tokio serde_json

use serde_json::json;

#[tokio::main]
async fn main() {
    // Router::from_env() picks up the *_API_KEY env vars.
    let router = llmshim::router::Router::from_env();

    let request = json!({
        "model": "claude-sonnet-5",
        "messages": [{"role": "user", "content": "What is Rust?"}],
        "max_tokens": 500,
    });

    // Responses come back in OpenAI Chat Completions format.
    let resp = llmshim::completion(&router, &request).await.unwrap();
    println!("{}", resp["choices"][0]["message"]["content"]);
}

Switch providers by changing the "model" string — everything else stays the same.

Streaming:

use futures::StreamExt;
use serde_json::json;

let router = llmshim::router::Router::from_env();
let request = json!({
    "model": "gpt-5.6-sol",
    "messages": [{"role": "user", "content": "Write a haiku about Rust."}],
    "max_tokens": 128,
});

let mut stream = llmshim::stream(&router, &request).await.unwrap();
while let Some(Ok(chunk)) = stream.next().await {
    let parsed: serde_json::Value = serde_json::from_str(&chunk).unwrap_or_default();
    if let Some(text) = parsed.pointer("/choices/0/delta/content").and_then(|c| c.as_str()) {
        print!("{text}");
    }
}

See examples/chat.rs and examples/stream.rs for runnable programs (cargo run --example chat).

Use it from the CLI

brew install sanjay920/tap/llmshim        # macOS
cargo install llmshim --features proxy    # from source (any platform)

llmshim                     # show help
llmshim chat                # interactive multi-model chat (streaming, /model to switch)
llmshim configure           # set API keys
llmshim login chatgpt        # sign in with a ChatGPT subscription
llmshim set <key> <value>   # set a config value
llmshim list                # show configured keys
llmshim models              # list available models
llmshim proxy               # start the HTTP proxy (see below)

Use it from any language (HTTP proxy)

Run llmshim as a local HTTP server and call it from any language. It has its own compact API (not OpenAI-shaped).

llmshim proxy
# Listening on http://localhost:3000

curl http://localhost:3000/v1/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-sonnet-5","messages":[{"role":"user","content":"Hi"}],"config":{"max_tokens":100}}'
Method Path Description
POST /v1/chat Chat completion (or streaming with stream: true)
POST /v1/chat/stream Always-streaming SSE with typed events
GET /v1/models List available models
GET /health Health check

Full API spec: api/openapi.yaml.

Scaling the proxy

The proxy is built to run as a horizontally-scaled fleet (Cloud Run, ECS, Kubernetes) without hammering provider rate limits. Two layers protect you:

  • Reactive retry (always on): on an upstream 429/5xx it honors the provider's Retry-After header and reset hints, falling back to full-jitter exponential backoff.
  • Proactive shedding (this layer): a per-provider token bucket rejects excess load before dispatching, and a per-instance concurrency cap sheds with 503 instead of running out of memory. Rejections carry a Retry-After header so clients back off cleanly.

All configuration is via env vars — everything optional with safe defaults. With no RPM/TPM limits set, only the concurrency cap applies.

Var Default Meaning
LLMSHIM_MAX_CONCURRENCY 256 Max in-flight upstream requests per instance.
LLMSHIM_QUEUE_TIMEOUT_MS 5000 Max wait for a concurrency slot before returning 503.
LLMSHIM_RATE_LIMIT_RPM unset Global requests-per-minute limit (per provider).
LLMSHIM_RATE_LIMIT_TPM unset Global tokens-per-minute limit.
LLMSHIM_OPENAI_RPM, LLMSHIM_ANTHROPIC_TPM, … unset Per-provider overrides (LLMSHIM_<PROVIDER>_RPM/_TPM).
LLMSHIM_REDIS_URL unset Enable distributed coordination (see below).

Two deployment modes:

  1. Sidecar / zero-infra (default). Each replica limits itself with an in-memory token bucket — no extra services. Running N replicas? Set each instance's limit to provider_limit / N.
  2. Redis-coordinated fleet. Build with the redis-coordination feature and set LLMSHIM_REDIS_URL; all replicas share one global token bucket in Redis, so you can set the true provider limit once regardless of replica count. It fails open (keeps serving) if Redis is briefly unreachable.
# Zero-infra: cap each instance
LLMSHIM_MAX_CONCURRENCY=512 LLMSHIM_OPENAI_RPM=1000 llmshim proxy

# Redis-coordinated fleet (build with the feature once)
cargo build --release --features redis-coordination
LLMSHIM_REDIS_URL=redis://my-redis:6379 LLMSHIM_OPENAI_RPM=10000 llmshim proxy

Verify the shedding behavior yourself — the load-test harness drives the real proxy against a mock upstream (no provider calls, $0) and asserts the concurrency-cap, RPM-shed, and overload paths all shed correctly with Retry-After:

cargo run --release --features proxy --example loadtest

Python client

pip install llmshim gives you a Python wrapper that bundles the Rust binary, starts the proxy on first use, and stops it on exit — no server to manage.

pip install llmshim

import llmshim

# Keys can also come from env vars or `llmshim configure`.
llmshim.configure(anthropic="sk-ant-...", openai="sk-...")

resp = llmshim.chat("claude-sonnet-5", "Hello!", max_tokens=500)
print(resp["message"]["content"])

Streaming:

for event in llmshim.stream("claude-sonnet-5", "Write a poem"):
    if event["type"] == "content":
        print(event["text"], end="", flush=True)
    elif event["type"] == "usage":
        print(f"\n[↑{event['input_tokens']} ↓{event['output_tokens']}]")

Multi-model conversation — switch providers mid-chat, history carries over:

messages = [{"role": "user", "content": "What is a closure?"}]

r1 = llmshim.chat("claude-sonnet-5", messages, max_tokens=500)
print(f"Claude: {r1['message']['content']}")

messages.append({"role": "assistant", "content": r1["message"]["content"]})
messages.append({"role": "user", "content": "Now explain it differently."})

r2 = llmshim.chat("gpt-5.6-sol", messages, max_tokens=500)
print(f"GPT: {r2['message']['content']}")

Tool use — pass tools in OpenAI Chat Completions format; llmshim translates to each provider's native format:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

resp = llmshim.chat("claude-sonnet-5", "Weather in Tokyo?", max_tokens=500, tools=tools)
for tc in resp["message"].get("tool_calls", []):
    print(f"{tc['function']['name']}({tc['function']['arguments']})")

Reasoning / thinking — one vocabulary across every provider:

resp = llmshim.chat(
    "claude-sonnet-5",
    "Solve: x^2 - 5x + 6 = 0",
    max_tokens=4000,
    reasoning_effort="high",   # none | low | medium | high | xhigh | max
    reasoning_mode="pro",      # standard (default) | pro — much more model work
)
print(resp["reasoning"])          # thinking content
print(resp["message"]["content"]) # answer

llmshim maps these to each provider's native control (OpenAI reasoning.effort/mode, Anthropic adaptive thinking, Gemini thinkingLevel, xAI reasoning.effort), clamping to the nearest tier the target model actually supports — so reasoning_effort="max" works everywhere even though only some models have a native max. Full verified mapping tables: the reasoning guide. Prefer a provider's exact native dialect? Pass it via provider_config (x-openai.reasoning, x-anthropic.thinking, x-gemini.thinkingConfig) and llmshim won't touch it.

Fallback chains — automatic failover across providers:

resp = llmshim.chat(
    "anthropic/claude-sonnet-5",
    "Hello",
    max_tokens=100,
    fallback=["openai/gpt-5.6-sol", "gemini/gemini-3.8-flash"],
)

These capabilities (streaming, multi-model, tools, reasoning, fallback) are all provided by the Rust core, so they work identically from the Rust crate and the proxy — the Python snippets above are just the most concise way to show them.

TypeScript / JavaScript client

npm install llmshim bundles a prebuilt proxy binary for your platform (same idea as the Python package) and auto-starts it on first use — nothing to run yourself. Pass an explicit baseUrl instead to connect to a proxy you're already running.

npm install llmshim

import { Client } from "llmshim";

const client = new Client(); // no baseUrl -> auto-starts the bundled proxy
const res = await client.chat({
  model: "anthropic/claude-sonnet-5",
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(res.message.content);

Full docs: clients/typescript/README.md.

Go client

go get github.com/sanjay920/llmshim/clients/go

client := llmshim.New() // defaults to http://localhost:3000
resp, err := client.Chat(ctx, llmshim.ChatRequest{
    Model:    "anthropic/claude-sonnet-5",
    Messages: []llmshim.Message{{Role: "user", Content: "Hello!"}},
})

Standard library only. Full docs: clients/go/README.md.

Ruby client

gem install llmshim

require "llmshim"

resp = Llmshim.chat(model: "anthropic/claude-sonnet-5", messages: [{role: "user", content: "Hello!"}])
puts resp.message.content

Standard library only. Full docs: clients/ruby/README.md.

Advertised models

Provider Models Reasoning visible
OpenAI gpt-6-astra, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna Yes (summaries)
Anthropic claude-fable-5-1, claude-opus-5, claude-sonnet-5, claude-haiku-4-5-20251001 Yes (thinking summaries)
Google Gemini gemini-3.8-flash, gemini-3.5-flash-lite Yes (thought summaries)
xAI grok-4.6 No (hidden)

The CLI and server advertise these current tiers. ChatGPT subscription access uses the same four OpenAI models under chatgpt/. OpenRouter and self-hosted providers accept caller-selected IDs without a fixed advertised list.

Use a bare model name (auto-detected by prefix) or an explicit provider/model string. Older explicit IDs retain their provider routing and metadata; the ChatGPT route continues to enforce its four-model allowlist.

Docker

llmshim docker build
llmshim docker start
llmshim docker status
llmshim docker logs
llmshim docker stop

How it works

No canonical struct. Requests flow as serde_json::Value — each provider maps only what it understands. Adding a provider = implementing one trait with three methods.

llmshim::completion(router, request)
  → router.resolve("anthropic/claude-sonnet-5")
  → provider.transform_request(model, &value)
  → HTTP
  → provider.transform_response(model, body)

Key features

  • Multi-model conversations — switch providers mid-chat, history carries over
  • Reasoning/thinking — visible chain-of-thought from OpenAI, Anthropic, and Gemini
  • Streaming — token-by-token, with thinking surfaced separately
  • Tool use — Chat Completions format auto-translated to each provider
  • Vision/images — send images in any format, auto-translated between providers
  • Fallback chains — automatic failover across providers with exponential backoff
  • Cross-provider translation — system messages, tool calls, and provider-specific fields all handled

Build & test

cargo build                                    # dev build
cargo build --release --features proxy         # release build (~6MB binary)
cargo test --features proxy --tests            # unit tests (~370)
cargo test --features proxy -- --ignored       # integration tests (needs API keys)

Contributing

Contributions are welcome — see CONTRIBUTING.md for the development setup, the CI gates, and the rules that protect the public API. Report suspected vulnerabilities privately per SECURITY.md, and be excellent to each other (Code of Conduct).

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option. llmshim is provided "AS IS", without warranty of any kind; the authors and contributors accept no liability for its use. See NOTICE. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in llmshim by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.