llmshim 0.2.1

Blazing fast LLM API translation layer in pure Rust
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115

# Rust quickstart

Use the crate when the application making LLM requests is already written in
Rust. The translation engine runs in-process; there is no proxy to start.

> **Availability:** Rust: direct engine API ยท CLI/HTTP/clients: not used in this path

## 1. Add the dependencies

```bash
cargo add llmshim tokio serde_json
```

`tokio` and `serde_json` are direct dependencies of your application. llmshim
does not re-export them.

Set at least one provider key before running the program:

```bash
export ANTHROPIC_API_KEY=sk-ant-...
```

## 2. Send a completion

```rust
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let router = llmshim::router::Router::from_env();

    // The public request contract is a serde_json::Value.
    let request = json!({
        "model": "openai/gpt-5.6-sol",
        "messages": [
            {"role": "user", "content": "What is Rust in one sentence?"}
        ],
        "max_tokens": 128
    });

    let response = llmshim::completion(&router, &request).await?;
    let text = response["choices"][0]["message"]["content"]
        .as_str()
        .unwrap_or("");

    println!("{text}");
    Ok(())
}
```

Run it with `cargo run`. `Router::from_env()` registers the providers whose
environment variables are present. Change only the `model` address to route
the same conversation elsewhere.

The result uses an OpenAI Chat Completions-style shape, regardless of which
provider answered. The assistant text is at
`response["choices"][0]["message"]["content"]`.

## 3. Stream content

Streaming uses the `StreamExt` trait, so add `futures` as a direct dependency:

```bash
cargo add futures
```

This complete example prints text deltas as they arrive:

```rust
use futures::StreamExt;
use serde_json::json;
use std::io::{self, Write};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let router = llmshim::router::Router::from_env();
    let request = json!({
        "model": "anthropic/claude-sonnet-5",
        "messages": [
            {"role": "user", "content": "Write a haiku about Rust."}
        ],
        "max_tokens": 128
    });

    let mut stream = llmshim::stream(&router, &request).await?;

    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        let parsed: serde_json::Value = serde_json::from_str(&chunk)?;

        if let Some(text) = parsed
            .pointer("/choices/0/delta/content")
            .and_then(|value| value.as_str())
        {
            print!("{text}");
            io::stdout().flush()?;
        }
    }

    println!();
    Ok(())
}
```

Each item is a JSON string in the normalized Chat Completions delta shape. A
chunk can carry something other than text, such as reasoning, usage, or a tool
call, so production code should inspect the fields it needs.

For config-file loading, aliases, and model inference, see
[Models and the Router](../concepts/routing.md). Runnable repository examples
are available in
[`examples/chat.rs`](https://github.com/sanjay920/llmshim/blob/main/examples/chat.rs)
and
[`examples/stream.rs`](https://github.com/sanjay920/llmshim/blob/main/examples/stream.rs).