libsy-llm-client
An HTTP client that speaks Switchyard's neutral IR directly. You hand it a
switchyard_protocol::Request and a ModelId; it looks up the configured backend,
encodes the request to that backend's wire format, adds auth and forwards your
headers, makes the call with a shared reqwest::Client, and decodes the reply
back into a switchyard_protocol::Response — buffered or streamed.
It also pairs the client with a libsy algorithm: [run] drives
[Algorithm::run_stream], serves routing-time calls, and consumes the terminal routing outcome.
When routing has not already produced the answer, run makes the terminal call and owns backend
retries plus ordered candidate fallback.
It depends on switchyard-libsy, switchyard-protocol, and
switchyard-translation; no server, no provider SDK.
Concepts
- Model configs. A client is built from
ModelConfigvalues, each keyed by aModelId. Each model has a defaultBackendand can have additional backends for other wire formats.TranslatingLlmClient::call_rewrite_modeluses the request's metadata wire format when set, otherwise the model's default backend. - Model ids. A
ModelIdis a newtype overStringthat behaves like one — it derefs tostr, compares against string literals, and converts from&strorStringwith.into(). Anywhere below that takesimpl Into<ModelId>accepts a bare literal; the borrowed positions need aModelIdvalue to reference. - Backends. A
Backendis one ofOpenAiChat,OpenAiResponses, orAnthropic, each wrapping anHttpBackendConfig(base_url,api_key, staticextra_headers, defaultextra_bodyfields, andmax_retries). The variant fixes the URL path and auth scheme (Bearer vsx-api-key+anthropic-version). - Model rewrite. The resolved
ModelIdis both the map key and the model id sent upstream — it overwrites whatevermodelthe request arrived with. - Streaming is chosen by the encoded request body. If the body has
stream: trueafterextra_bodyis applied, you getLlmResponse::Stream; otherwiseLlmResponse::Agg. OpenAI Chat streaming requests defaultstream_options.include_usagetotrue; an explicit caller value is preserved.
Add the dependency
Within this workspace:
[]
= "0.3"
= { = "../libsy" }
= { = "../libsy-llm-client" }
= { = "../protocol" }
= { = "../switchyard-translation" } # for WireFormat
Quickstart
Build a client
use BTreeMap;
use ;
Buffered call
use ;
use ;
async
Streaming call
Set stream on the IR request and drive the returned chunk stream:
use StreamExt;
use TranslatingLlmClient;
use ;
async
Routing an algorithm
[run] takes a libsy algorithm and a [ClientRouter], and returns the algorithm-selected
ModelId plus the final response. Routing-time CallModels are served while the algorithm
runs. The terminal outcome either already contains the answer or supplies the selected model and
ordered fallbacks for the client to try. ClientRouter::single is the single-provider case:
use Arc;
use ;
use ;
use Request;
async
When targets are served by different clients — a judge on one provider and the serving
models on another, say — build the router from ModelId -> client with
[ClientRouter::new] instead. A model missing from that map fails with
LlmClientError::Configuration rather than silently going to another provider.
use HashMap;
use Arc;
use ClientRouter;
use ;
A router is not itself a client: [ClientRouter::route] hands back a
[switchyard_protocol::RoutedLlmClient] and the caller makes the call.
Cross-format translation
The request/response are translated through the neutral IR, so the inbound shape
you build and the backend's wire format are independent. Pointing a
WireFormat::AnthropicMessages backend at an Anthropic endpoint while building
requests with the same helpers works the same way. Set request.metadata.wire_format
to select a non-default backend before calling call_rewrite_model. Register several
formats under one model to serve it over more than one upstream API:
use ;
Headers & auth
- The
Backendvariant sets auth: OpenAI formats sendAuthorization: Bearer <key>; Anthropic sendsx-api-key: <key>plusanthropic-version. - By default, only
x-request-idfromrequest.metadata.http_headersis forwarded. Withforward_auth, application headers are forwarded, while connection, destination, body, and the selected provider's owned headers are removed or rebuilt. HttpBackendConfig::forward_authuses the caller's credential instead of the backend's configured key and forwards other application headers. OpenAI backends normalizeauthorization,chatgpt-account-id, andx-openai-fedramp. Anthropic backends forwardauthorizationorx-api-key; they also keepoauth-*values fromanthropic-betaand remove other caller-supplied beta values. All backends reachable through a forwarding route must use the same provider. Headers owned by other providers are preserved as application headers.- Per-backend custom headers go in
HttpBackendConfig::extra_headers. Set credentials withapi_key. OpenAI backends rejectAuthorization; Anthropic backends rejectx-api-keyandanthropic-version. Header names are case-insensitive. - Per-target top-level request defaults go in
HttpBackendConfig::extra_body. The merge is shallow and fields already present in the request take precedence. HttpBackendConfig::max_retriescontrols additional attempts after retryable transport failures, timeouts, HTTP 408/429, and 5xx responses. Buffered body transport failures are retried; streaming body failures are not replayed after the response has been returned.HttpBackendConfig::timeoutbounds one complete response, including retries, retry delays, and every stream read. Expiry returnsLlmClientError::Timeout, either from the call or from the returned stream, which then ends.Noneleaves the wait unbounded.
run and decide collect streams used during routing, including answers that an
algorithm must inspect, before returning them to the algorithm. They retain the
provider events for replay. After the configured retries, a client failure stops
run or decide before the algorithm can choose another routing candidate. This
also applies when timeout is None or an advisor has fail_open = true.
libsy and custom hosts that drive it directly are unchanged.
Retries replay the same upstream request to the same model. After routing completes, non-timeout failures may try another completion candidate. A timeout stops the call. A transport failure can duplicate a request that the provider processed but did not finish returning. Each attempt is counted once; expiry during a retry delay or after a stream has started does not count another attempt.
Errors
call_rewrite_model returns LlmClientError:
| Variant | When |
|---|---|
InvalidRequest { message } |
the request does not identify a model |
Configuration { message } |
the model or requested wire format has no configured backend |
RequestTranslation(msg) |
decoding the inbound request failed in the translation engine |
RequestEncoding(msg) |
re-encoding an already-decoded request to the wire format failed (internal fault) |
ResponseTranslation(msg) |
response decoding or encoding failed in the translation engine |
Timeout { source } |
request or response body read exceeded its timeout |
Transport { source } |
non-timeout connection or transport failure |
ContextWindowExceeded { model, message } |
upstream 400 detected as a context overflow (checked before UpstreamHttp, so callers can evict-and-retry) |
UpstreamHttp { status, body } |
any other non-2xx upstream response |
InvalidResponse { source } |
the upstream response could not be decoded |
Other(source) |
a client-specific failure outside the shared categories |