Skip to main content

Crate switchyard_llm_client

Crate switchyard_llm_client 

Source
Expand description

HTTP LLM client that speaks Switchyard’s neutral IR directly.

TranslatingLlmClient maps a model name (and the wire format resolved from the request) to a Backend, encodes a switchyard_protocol::Request to that backend’s wire format via switchyard-translation, applies auth and forwards caller headers, makes the HTTP call with a shared reqwest::Client, and decodes the wire response back to a switchyard_protocol::Response — supporting both buffered and streamed responses.

run() pairs the client with a libsy algorithm: it drives switchyard_libsy::Algorithm::run_stream, serves routing-time calls, and makes the terminal answer call from the routing outcome when needed. A host that just wants the answer does not have to drive the step stream itself.

Re-exports§

pub use backend::Backend;
pub use backend::DEFAULT_MAX_RETRIES;
pub use backend::HttpBackendConfig;
pub use client::AuxiliaryOperation;
pub use client::ModelConfig;
pub use client::TranslatingLlmClient;
pub use error::Result;
pub use raw::RawResponse;
pub use run::ClientRouter;
pub use run::decide;
pub use run::run;

Modules§

backend
Per-provider backend configuration: wire format, upstream URL, and auth.
client
TranslatingLlmClient — the crate’s single public entry point: encode a neutral request, call the configured backend over HTTP, decode the neutral response.
error
Canonical client error re-export and shared context-window-overflow detection.
metrics
Metric labelling inherited from Python
raw
The raw wire result of a call.
run
Drive a libsy algorithm to completion, making the model calls it offloads.

Structs§

LlmCallObservation
One completed model call observed while serving an algorithm run.

Enums§

LlmClientError
Failures a routed LLM client can surface to its caller.
RunObservation
Events emitted inline while [crate::run] serves a routing request.

Functions§

initialize_metrics
Registers process-wide compatibility gauges with the global meter provider.

Type Aliases§

RawEventStream
A stream of wire-format event objects in one format — the unframed body of an SSE response. The serving layer frames each Value (e.g. as an SSE data:/event: block).
RunObserver
Request-scoped callback for algorithm-run observations.