libsy-llm-client

An HTTP client that speaks Switchyard's neutral IR directly. You hand it a switchyard_protocol::Request and a model name; it looks up the configured backend, encodes the request to that backend's wire format, adds auth and forwards your headers, makes the call with a shared reqwest::Client, and decodes the reply back into a switchyard_protocol::Response — buffered or streamed.

It depends only on switchyard-protocol and switchyard-translation; no server, no provider SDK.

Concepts

• Model configs. A client is built from ModelConfig values. Each model has a default Backend and can have additional backends for other wire formats. TranslatingLlmClient::call_rewrite_model uses the request's metadata wire format when set, otherwise the model's default backend.
• Backends. A Backend is one of OpenAiChat, OpenAiResponses, or Anthropic, each wrapping an HttpBackendConfig (base_url, api_key, static extra_headers, default extra_body fields, and max_retries). The variant fixes the URL path and auth scheme (Bearer vs x-api-key + anthropic-version).
• Model rewrite. The resolved model name is both the map key and the model id sent upstream — it overwrites whatever model the request arrived with.
• Streaming is chosen by the request. If the encoded body has stream: true (i.e. request.llm_request.stream), you get LlmResponse::Stream; otherwise LlmResponse::Agg. OpenAI Chat streaming requests default stream_options.include_usage to true; an explicit caller value is preserved.

Add the dependency

Within this workspace:

[dependencies]
switchyard-llm-client = { path = "../libsy-llm-client" }
switchyard-protocol = { path = "../libsy-protocol" }
switchyard-translation = { path = "../switchyard-translation" }   # for WireFormat

Quickstart

Build a client

use std::collections::BTreeMap;
use switchyard_llm_client::{
    Backend, HttpBackendConfig, ModelConfig, TranslatingLlmClient,
};

fn build_client() -> switchyard_llm_client::Result<TranslatingLlmClient> {
    let openai = HttpBackendConfig {
        base_url: "https://api.openai.com/v1".to_string(),
        api_key: std::env::var("OPENAI_API_KEY").ok(),
        extra_headers: BTreeMap::new(),
        extra_body: BTreeMap::new(),
        max_retries: 2,
    };

    let models = [ModelConfig::new(
        "gpt-4o-mini",
        Backend::OpenAiChat(openai),
        None,
    )];

    TranslatingLlmClient::new(&models)
}

Buffered call

use switchyard_llm_client::{LlmClientError, TranslatingLlmClient};
use switchyard_protocol::{completion_text, text_request, Context, LlmResponse, Request};

async fn ask(client: &TranslatingLlmClient) -> switchyard_llm_client::Result<String> {
    let request = Request {
        llm_request: text_request(None, "Say hello in five words."),
        raw_request: None,
        metadata: None,
    };

    // model_name wins over request.llm_request.model; it is also sent upstream.
    let response = client
        .call_rewrite_model(Context::default(), request, Some("gpt-4o-mini"))
        .await?;

    match response.llm_response {
        LlmResponse::Agg(agg) => Ok(completion_text(&agg)),
        LlmResponse::Stream(_) => Err(LlmClientError::InvalidResponse {
            source: "expected a buffered response".into(),
        }),
    }
}

Streaming call

Set stream on the IR request and drive the returned chunk stream:

use futures_util::StreamExt;
use switchyard_llm_client::TranslatingLlmClient;
use switchyard_protocol::{text_request, Context, LlmResponse, LlmResponseChunk, Request};

async fn stream(
    client: &TranslatingLlmClient,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let mut llm_request = text_request(None, "Count to five.");
    llm_request.stream = true;
    let request = Request { llm_request, raw_request: None, metadata: None };

    let response = client
        .call_rewrite_model(Context::default(), request, Some("gpt-4o-mini"))
        .await?;

    if let LlmResponse::Stream(mut chunks) = response.llm_response {
        while let Some(item) = chunks.next().await {
            match item {
                Ok(LlmResponseChunk::TextDelta { text, .. }) => print!("{text}"),
                Ok(_) => {}                       // usage, tool-call deltas, message start/stop
                Err(error) => return Err(error),
            }
        }
    }
    Ok(())
}

Cross-format translation

The request/response are translated through the neutral IR, so the inbound shape you build and the backend's wire format are independent. Pointing a WireFormat::AnthropicMessages backend at an Anthropic endpoint while building requests with the same helpers works the same way. Set request.metadata.wire_format to select a non-default backend before calling call_rewrite_model. Register several formats under one model to serve it over more than one upstream API:

use switchyard_llm_client::{
    Backend, HttpBackendConfig, ModelConfig, TranslatingLlmClient,
};

fn build_multi_format_client(
    openai_chat: HttpBackendConfig,
    openai_responses: HttpBackendConfig,
    anthropic: HttpBackendConfig,
) -> switchyard_llm_client::Result<TranslatingLlmClient> {
    let models = [ModelConfig::new(
        "my-model",
        Backend::OpenAiChat(openai_chat),
        Some(vec![
            Backend::OpenAiResponses(openai_responses),
            Backend::Anthropic(anthropic),
        ]),
    )];

    TranslatingLlmClient::new(&models)
}

Headers & auth

• The Backend variant sets auth: OpenAI formats send Authorization: Bearer <key>; Anthropic sends x-api-key: <key> plus anthropic-version.
• request.metadata.http_headers are forwarded upstream, except reserved ones: host, content-length, connection, and the backend-owned authorization / x-api-key / anthropic-version / content-type. So a caller's placeholder credential never overrides the backend's real key.
• Per-backend static headers go in HttpBackendConfig::extra_headers.
• Per-target top-level request defaults go in HttpBackendConfig::extra_body. The merge is shallow and fields already present in the request take precedence.
• HttpBackendConfig::max_retries controls additional attempts after retryable transport failures, timeouts, HTTP 408/429, and 5xx responses. Buffered body transport failures are retried; streaming body failures are not replayed after the response has been returned.

Retries replay the same upstream request. A transport failure can therefore duplicate a request that the provider processed but did not finish returning, and the retry budget plus capped Retry-After delays determines total latency.

Errors

call_rewrite_model returns LlmClientError: