llmshim 0.2.0

Blazing fast LLM API translation layer in pure Rust
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118

# Streaming

Streaming exposes the same kinds of incremental output on every surface, but
the Rust crate and the proxy encode them differently.

> **Availability:** Rust: normalized JSON chunks · CLI: rendered live · Proxy: typed SSE · Clients: language-native stream iterators

The useful distinction is: **same semantic events, two encodings.**

| Signal | Rust chunk | Proxy/client event |
|---|---|---|
| Answer text | `choices[0].delta.content` | `content` |
| Provider-returned reasoning | `choices[0].delta.reasoning_content` | `reasoning` |
| Tool call | `choices[0].delta.tool_calls` | `tool_call` |
| Token counts | top-level `usage` | `usage` |
| Completion | `finish_reason` | `done` |
| Stream failure | Rust `Err` | `error` |

## Rust: normalized Chat Completions chunks

`llmshim::stream` returns a stream whose items are JSON strings. Each provider
adapter has already translated its native event into an OpenAI Chat
Completions-style delta.

```rust
use futures::StreamExt;
use std::io::{self, Write};

let mut stream = llmshim::stream(&router, &request).await?;

while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    let parsed: serde_json::Value = serde_json::from_str(&chunk)?;

    if let Some(reasoning) = parsed
        .pointer("/choices/0/delta/reasoning_content")
        .and_then(|value| value.as_str())
    {
        eprint!("{reasoning}");
    }

    if let Some(text) = parsed
        .pointer("/choices/0/delta/content")
        .and_then(|value| value.as_str())
    {
        print!("{text}");
        io::stdout().flush()?;
    }
}
```

Not every chunk contains text. Inspect only the fields your application needs,
and keep handling `Err` items until the stream ends.

The [Rust quickstart](../start/rust.md#3-stream-content) contains a complete
runnable program.

## Proxy: typed SSE events

Send the compact request to the always-streaming endpoint:

```bash
curl -N http://localhost:3000/v1/chat/stream \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [{"role": "user", "content": "Write a haiku."}],
    "config": {"max_tokens": 128}
  }'
```

The response is an SSE stream. The SSE `event` name and the JSON `type` agree:

```text
event: reasoning
data: {"type":"reasoning","text":"..."}

event: content
data: {"type":"content","text":"Rust shapes silent thought"}

event: usage
data: {"type":"usage","input_tokens":12,"output_tokens":7,"total_tokens":19}

event: done
data: {"type":"done"}
```

The six event types are:

| Type | Payload |
|---|---|
| `content` | `text` |
| `reasoning` | `text` |
| `tool_call` | `id`, `name`, JSON-encoded `arguments` |
| `usage` | input, output, optional reasoning, and total token counts |
| `done` | no additional fields |
| `error` | `message` |

Setting `"stream": true` on `POST /v1/chat` produces the same SSE encoding.
An admission error can still arrive as an HTTP error before streaming begins;
an error after the stream begins arrives as an `error` event.

## Client ergonomics

The language clients decode the typed SSE stream without changing its event
vocabulary:

- Python returns an iterator of event dictionaries.
- TypeScript returns an async iterator of typed events.
- Go returns a channel of typed events.
- Ruby yields typed events to a block.

See the canonical client guides for complete loops:
[Python](https://github.com/sanjay920/llmshim/blob/main/clients/python/README.md#streaming),
[TypeScript](https://github.com/sanjay920/llmshim/blob/main/clients/typescript/README.md#30-second-quickstart),
[Go](https://github.com/sanjay920/llmshim/blob/main/clients/go/README.md#streaming), and
[Ruby](https://github.com/sanjay920/llmshim/blob/main/clients/ruby/README.md#streaming).