---
title: What rtb-chat does not do
description: The unsupported combinations, the known defects and the deliberate omissions in rtb-chat — with the behaviour you actually get in each case.
tags: [explanation, limitations]
---
# What rtb-chat does not do
Documentation that only describes what software does leaves you to infer the rest
from silence. This page states the gaps.
Two kinds of gap are mixed here on purpose, but they are labelled: **deliberate**
omissions, which are not going to change soon, and **defects**, where the crate
does not do what its own API implies. The behaviour described is what v0.7.2
actually does either way.
## A custom base_url is ignored on every genai-backed provider
**Defect.**
`Config::base_url` is validated for `Provider::OpenAi`,
`Provider::OpenAiCompatible`, `Provider::Gemini` and `Provider::Ollama`, and then
never used. `rtb-chat` builds a `genai::Client::default()` and passes it no
endpoint, so genai resolves its own.
Which means the headline use case for `Provider::OpenAiCompatible` — pointing at
Together, Fireworks, vLLM or an in-house gateway — does not work. The request goes
to whatever endpoint genai picks from the model name, and if the name matches
nothing genai recognises, that endpoint is `http://localhost:11434/` because
Ollama is its fallback adapter.
Configured with a custom base URL and the model `llama-3.1-70b-instruct`, the
request lands on localhost:
```text
POST /api/chat HTTP/1.1 ← received by a listener on 127.0.0.1:11434
```
If something is listening there, there is no error and no warning — the only
signal is the endpoint the request turns up at. If nothing is, the call fails
against localhost rather than against the endpoint you configured:
```text
provider error: Web call failed for model 'llama-3.1-70b-instruct (adapter: Ollama)'.
Cause: Reqwest error: error sending request for url (http://localhost:11434/api/chat)
```
**What to do instead.** Use `Provider::AnthropicLocal` with `base_url` set if your
gateway speaks the Anthropic Messages format — that path honours the override. If
it speaks OpenAI's format, there is no way to reach it through `rtb-chat` today.
## Config::timeout is ignored on every genai-backed provider
**Defect**, and the same cause. The timeout is applied to the `reqwest` client
that the Anthropic-direct backend builds. genai builds its own HTTP client and is
given no timeout, so `Config::timeout` has no effect on four of the six providers.
## temperature and max_tokens are silently dropped on genai providers
**Defect.** `ChatRequest::temperature` and `ChatRequest::max_tokens` are read only
by the Anthropic-direct request builder. The genai request builder copies across
the system prompt and the messages, and nothing else.
Set both on an OpenAI-compatible call and the wire body contains neither:
```json
{"messages":[{"content":"hi","role":"user"}],"model":"llama-3.1-70b-instruct","stream":false}
```
The call succeeds, at the provider's default temperature and length limit. If
determinism matters to you — a classifier, a structured extraction — that
`temperature: Some(0.0)` is doing nothing on those providers.
## Streaming token counts are always zero
**Defect**, on both backends, for two separate reasons.
On the **Anthropic-direct** path, `ChatStreamEvent::Done` reads usage from the
`message_stop` SSE event, which does not carry any. The Messages API reports input
tokens on `message_start` and output tokens on `message_delta`, both of which
`rtb-chat` parses and discards.
On the **genai** path, `Done` reads genai's `captured_usage`, which genai only
populates when `capture_usage` is switched on in its chat options. `rtb-chat`
passes no chat options, and genai's default is off.
Either way, a streamed call ends with `Done(Usage::default())` — four zeros —
however many tokens were really used. Use the non-streaming `chat` call when you
need to account for usage.
## Only text content is supported
**Deliberate, for now.** `ContentBlock` has one variant, `Text`. Images, audio,
tool use and function calling have no representation in the request, and any
non-text block in a response is dropped during parsing rather than surfaced.
`ContentBlock` and `Provider` are both `#[non_exhaustive]` so that adding those
variants later is not a breaking change.
## No tool use, no function calling, no agent loop
**Deliberate.** `rtb-chat` sends a request and returns a reply. It does not
execute tools, does not loop, and does not manage conversation state — the
`messages` vector you pass is the whole history, and keeping it is your job.
genai's tool-call stream chunks are received and thrown away, which is what makes
this a hard boundary rather than a partial implementation.
## No retries, no backoff, no circuit breaker
**Deliberate.** A 429 becomes `AiError::RateLimited` carrying `retry_after` when
the provider sent a parseable `Retry-After` header, and that is the extent of it.
Nothing sleeps and nothing retries.
Retry policy belongs to the caller because the right policy depends on whether a
human is waiting. Note that `AiError::RateLimited` never occurs on a genai
provider — see below — so a retry loop keyed on that variant covers only the two
Anthropic providers.
## genai providers report every failure as a provider error
**Deliberate consequence of the two-backend split, worth knowing about.** The
status-code mapping that produces `AiError::Auth` and `AiError::RateLimited` lives
in `rtb-chat`'s own Anthropic path. genai surfaces failures through its own error
type, and all of them are wrapped as `AiError::Provider`.
A bad OpenAI key does not give you `AiError::Auth`. It gives you
`AiError::Provider` with genai's description inside.
## The genai backend puts your key in the process environment
**Deliberate, and the crate's one `unsafe` block.** genai resolves provider
credentials from environment variables. To hand it the key from `Config`,
`AiClient::new` calls `std::env::set_var` — `OPENAI_API_KEY` for
`Provider::OpenAi` and `Provider::OpenAiCompatible`, `GEMINI_API_KEY` for
`Provider::Gemini`.
Three consequences follow, none of them obvious from the type signature:
- **The key outlives the client.** It stays in the environment for the life of the
process and is readable by anything in it, including code that dumps the
environment into a crash report.
- **The last client wins.** Build a second `AiClient` for the same provider with a
different key and the first client starts using the second key, because both
read the same variable at call time.
- **Constructing clients concurrently is a data race.** `std::env::set_var` is
`unsafe` on modern Rust editions for exactly this reason. The crate assumes the
common pattern of one client built once per process; building clients from
several threads while other threads read the environment is undefined behaviour.
The Anthropic-direct path does not do this — it sends the key as a header and
touches no environment variable. Neither does `Provider::Ollama`, which sets
nothing.
## An API key is required even when the provider does not need one
**Deliberate, and awkward.** `AiClient::new` rejects an empty `api_key` for every
provider, `Provider::Ollama` included. A local Ollama needs no credential, so you
have to invent one:
```rust
api_key: SecretString::from("unused".to_string()),
```
The check is one rule applied uniformly rather than a per-provider table. It costs
a placeholder string; leaving it out would let a genuine missing-key mistake reach
the provider as a confusing 401.
## Config cannot be deserialised
**Deliberate.** `Config` derives `Debug` and `Clone` and nothing else. There is no
`Deserialize`, so a config file cannot be mapped onto it directly — the tool layer
reads its own settings type and constructs a `Config` in code.
That is what keeps `allow_insecure_base_url` out of reach of anything but Rust
source, which is the property that makes the HTTPS enforcement worth having. See
[Why the endpoint is validated](02-endpoint-validation.md).
## One model per client, and no fallback
**Deliberate.** `Config::model` is fixed at construction. There is no per-request
override, no fallback chain, and no "try the cheap model first" behaviour. Build a
second `AiClient` for a second model — bearing in mind the environment-variable
caveat above if both are genai-backed.
## anthropic-version cannot be changed
**Deliberate.** The Anthropic-direct path sends `anthropic-version: 2023-06-01`
from a compile-time constant. A Messages API feature gated behind a newer version
string needs a crate release, not a config change.
## Structured output is one attempt, with no repair
**Deliberate.** `chat_structured` asks for bare JSON, parses what comes back,
validates it against the schema and deserialises it. If any of those fails, you
get the error — there is no retry with a corrective prompt, no fence-stripping, no
partial extraction.
The most common failure is a reply truncated by the 1024-token default
`max_tokens`, which arrives as `AiError::Deserialize` rather than anything that
mentions length.
## Where the defects are recorded
The four defects above are documented here because they are the crate's current
behaviour and a reader needs to know about them. They are not accepted design.
They were found while writing these pages and are reported against
[rust/chat#1](https://gitlab.com/phpboyscout/rust/chat/-/issues/1); this page
should shrink as they are fixed.