Module llm 

An LLM proxy (18.9.0, #41): one endpoint in front of OpenAI-compatible and Anthropic providers that counts tokens, enforces budgets and fails over between providers.

use rust_web_server::llm::{Api, LlmProxy, Provider};

let proxy = LlmProxy::new()
    .provider(Provider::new("openai", Api::OpenAi, "https://api.openai.com").unwrap().api_key("sk-…"))
    .provider(Provider::new("azure", Api::OpenAi, "https://example.openai.azure.com").unwrap().api_key("…"))
    .provider(Provider::new("anthropic", Api::Anthropic, "https://api.anthropic.com").unwrap().api_key("sk-ant-…"))
    .virtual_key("vk-alice", "alice")
    .budget("alice", 2_000_000, 86_400);
// `proxy` is an `Application`: serve it, or mount it on a route.

• Which API. A request to a path ending in /messages is Anthropic’s Messages API; anything else (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/models, …) is OpenAI’s. It goes to the providers of that API, in the order added; the path is sent as is, after the provider URL’s own path.
• Virtual keys. Clients authenticate with a key the proxy issued (Authorization: Bearer <key> or x-api-key: <key>, as each SDK sends it); it names an identity and never reaches a provider, which gets its own key instead (Authorization: Bearer for OpenAI, x-api-key and a default anthropic-version for Anthropic). Unknown keys get 401.
• Fallback. A provider that doesn’t answer, or answers 429 or 5xx, is skipped for the next one of the same API — before anything was sent to the client, so a stream never switches provider midway.
• Usage. Tokens come from the provider’s response, JSON or event stream (usage). For OpenAI streams the proxy adds "stream_options":{"include_usage":true} when the request has no stream_options, so the stream ends with one extra chunk with empty choices that carries the usage.
• Budgets. Tokens per identity per fixed window, in memory or shared between replicas through Redis (LlmProxy::redis). Checked before a request and charged after it with what the provider reported, so the request that crosses the budget completes; the next one gets 429. Redis errors are logged and let requests through (fail open).
• Metrics. rws_llm_requests_total, rws_llm_tokens_total, rws_llm_time_to_first_token_seconds, rws_llm_fallbacks_total, rws_llm_budget_rejections_total on /metrics; with LlmProxy::audit_log one JSON line per request in the access log.

Modules

usage

Token usage out of provider responses, as they stream past.

Structs

LlmProxy

The LLM proxy, an Application. See the module docs.

Provider

One upstream provider.

Usage

Tokens a response reported.

UsageParser

Usage from a response body fed in pieces. See the module docs.

Enums

Api

The API a provider speaks.