llmshim 0.4.0

Blazing fast LLM API translation layer in pure Rust
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165

# Model discovery

Runtime discovery is the canonical answer to “what can this configured
installation offer?”

```bash
llmshim models
curl http://localhost:3000/v1/models
```

Both commands filter the built-in registry to providers with configured API
keys or a saved ChatGPT login. The proxy returns `id`, `provider`, and unprefixed `name`; the CLI prints
the ID and display label.

The standalone `llmshim-catalog` crate supplies broader metadata without
changing this curated discovery list:

```bash
llmshim models --all --json
llmshim models --refresh
LLMSHIM_CATALOG_OFFLINE=1 llmshim models --all
```

Its owned `ModelInfo` adds model family, pricing, reasoning effort/budget
options, modalities, dates, and field-level sources. `llmshim::catalog::resolve`
reads the resident catalog; `llmshim::models::spec` retains the borrowed,
compile-time API for curated and historical facts.

Startup reads the vendored floor, cached data, and local overrides without
waiting for network. Refresh uses ETags and a 24-hour TTL; errors keep the old
snapshot available. Offline mode uses only vendored and local layers. User
overrides live in `~/.config/llmshim/models.toml`, project overrides in
`.llmshim/models.toml`, and cached data in `~/.cache/llmshim/models.dev.json`.
Local overrides win; verified builtin assertions win over models.dev data.
Provider discovery is explicit, accepts capability facts, and never supplies
pricing. Catalog entries do not imply credentials or account entitlement.

## Advertised catalog

The CLI picker, `llmshim models`, and `/v1/models` share this curated set of
15 routes. It keeps the current model in each retained tier. Google entries
are stable releases only. Credentials determine which providers are listed.

### OpenAI

| ID | Display name |
|---|---|
| `openai/gpt-6-astra` | GPT-6 Astra |
| `openai/gpt-5.6-sol` | GPT-5.6 Sol |
| `openai/gpt-5.6-terra` | GPT-5.6 Terra |
| `openai/gpt-5.6-luna` | GPT-5.6 Luna |

### Anthropic

| ID | Display name |
|---|---|
| `anthropic/claude-fable-5-1` | Claude Fable 5.1 |
| `anthropic/claude-opus-5` | Claude Opus 5 |
| `anthropic/claude-sonnet-5` | Claude Sonnet 5 |
| `anthropic/claude-haiku-4-5-20251001` | Claude Haiku 4.5 |

### Google Gemini

| ID | Display name |
|---|---|
| `gemini/gemini-3.8-flash` | Gemini 3.8 Flash |
| `gemini/gemini-3.5-flash-lite` | Gemini 3.5 Flash Lite |

### xAI

| ID | Display name |
|---|---|
| `xai/grok-4.6` | Grok 4.6 |

### ChatGPT subscription

| ID | Display name |
|---|---|
| `chatgpt/gpt-6-astra` | GPT-6 Astra (ChatGPT) |
| `chatgpt/gpt-5.6-sol` | GPT-5.6 Sol (ChatGPT) |
| `chatgpt/gpt-5.6-terra` | GPT-5.6 Terra (ChatGPT) |
| `chatgpt/gpt-5.6-luna` | GPT-5.6 Luna (ChatGPT) |

The ChatGPT provider accepts only these four model IDs. Older and unlisted
IDs are rejected locally before authentication or network calls, including
when selected through a Router alias. The list is not an account entitlement
check. Context and output limits remain unspecified because they depend on
the account/backend. Bare GPT names still route to API-key OpenAI.

## Spec metadata

Each registry entry can also carry **spec metadata** beyond its identity, so a
consumer can read a model's facts from one place instead of maintaining its own
parallel table:

| Field | Type | Meaning |
|---|---|---|
| `context_window_tokens` | `Option<u32>` | Total context window (input + output), if published |
| `max_output_tokens` | `Option<u32>` | Maximum output tokens per response, if published |
| `capabilities.tools` | `Support` | Function/tool calling |
| `capabilities.streaming` | `Support` | Streaming responses |
| `capabilities.images` | `Support` | Image input |
| `capabilities.prompt_cache` | `Support` | Provider-side prompt caching |
| `capabilities.structured_output` | `Support` | JSON-schema / structured responses |
| `capabilities.parallel_tool_calls` | `Support` | Multiple tool calls per turn |
| `capabilities.reasoning` | `Support` | Accepts a reasoning-effort control |

`Support` is tri-state: `Supported`, `Unsupported`, or `Unknown`. **`Unknown` is
honest, not a bug** — llmshim never guesses a spec to fill a cell. Context
window, output ceiling, and capabilities are versioned snapshots populated
from official provider docs (platform.claude.com, developers.openai.com,
ai.google.dev, docs.x.ai), and `reasoning` is cross-checked against the provider
clamp logic. What's deliberately left `Unknown`/`None`:

- `parallel_tool_calls` for most models (providers rarely document it per model);
- xAI `max_output_tokens` (not published) and per-model streaming.

Note also that Gemini publishes an input limit rather than a combined total, so
`context_window_tokens` is the input window with `max_output_tokens` separate.

`reasoning` is deliberately a single flag — "does this model accept a reasoning
control at all." The detailed per-tier mapping is not duplicated here; it lives
in the [reasoning guide](../guides/reasoning.md) and the provider transforms.

Look up one model's full spec from the Rust crate:

```rust
if let Some(m) = llmshim::models::spec("openai/gpt-5.6-sol") {
    println!("{}: reasoning = {:?}", m.label, m.capabilities.reasoning);
}
```

`spec()` accepts a full id (`"openai/gpt-5.6-sol"`) or a bare name
(`"gpt-5.6-sol"`). It also retains historical metadata for explicit lookups;
those models are absent from the advertised list. `None` means no metadata
is recorded. Specs remain a point-in-time snapshot pinned by the crate version.

## Routing beyond the catalog

Pruning the advertised list does not remove provider adapters or their
compatibility behavior. Older explicit IDs still reach their provider, subject
to upstream availability. Known historical IDs also remain selectable by
exact ID in the CLI, and `spec()` keeps their metadata.

The Router does not check explicit model names against this registry. If a
provider is registered, `provider/arbitrary-model-id` is routed to that
provider with `arbitrary-model-id` unchanged. A bare, unregistered model name
works only when its prefix identifies a provider (`gpt`, `o1`, `o3`, `o4`,
`claude`, `gemini`, or `grok`).

The ChatGPT provider is the exception: it validates requests against the four
supported subscription models above.

**OpenRouter** is intentionally not enumerated above — its catalog is large and
dynamic. Any `openrouter/<vendor>/<model>` slug routes through (e.g.
`openrouter/anthropic/claude-sonnet-5`, `openrouter/meta-llama/llama-3.1-70b-instruct:nitro`);
the slug's internal slash and `:variant` suffix are preserved. Because its
slugs collide with other providers' prefixes, OpenRouter has no bare-model
inference — always address it explicitly as `openrouter/…`.

**Self-hosted vLLM / SGLang** are likewise not enumerated. Set `VLLM_BASE_URL` or `SGLANG_BASE_URL` (with an optional `*_API_KEY`) and address the served model as `vllm/<served-model>` or `sglang/<served-model>` — e.g. `sglang/Qwen/Qwen3.6-35B-A3B-FP8`. Local vs remote is just the base-URL value.

Rust applications can also define one-level Router aliases with
`Router::alias`. Those aliases are not part of the static registry and are not
configured by the stock CLI or proxy. See [Models and the Router](../concepts/routing.md).