Struct TranslatingLlmClient 

pub struct TranslatingLlmClient { /* private fields */ }

A client that dispatches neutral-IR requests to per-model HTTP backends.

Construct it with a list of ModelConfigs — one per model, each naming a default Backend and any additional per-format backends. Each call resolves the model and wire format, encodes the request to that backend’s wire format, applies auth and forwarded headers, sends the HTTP request with a shared reqwest::Client, and decodes the response back to the neutral IR (buffered or streamed).

Implementations

impl TranslatingLlmClient

pub fn new(model_configs: &[ModelConfig]) -> Result<Self>

Builds a client over the given ModelConfigs, with a fresh shared HTTP client and the built-in translation codecs.

pub fn backend_for(&self, model: &str, format: WireFormat) -> Option<&Backend>

The backend serving model over format — the default backend when its format matches, otherwise a matching entry in other_backends; None when the model is unknown or has no backend for format.

pub fn supports_count_tokens(&self, model: &str) -> bool

Whether model has an Anthropic backend that supports token counting.

pub async fn count_tokens(&self, model: &str, request: Request) -> Result<Value>

Counts input tokens with model’s Anthropic backend.

Returns an error when the model has no Anthropic backend or the upstream request fails or returns invalid JSON.

pub async fn call_rewrite_model( &self, _ctx: Context, request: Request, model_name: Option<&str>, ) -> Result<Response>

Calls the backend for model_name (or the request’s own model), over the wire format the request pins in its metadata (else the model’s default backend), and returns the neutral response.

Resolution: model_name wins over request.llm_request.model; the resolved name is both the outer map key and the model id written into the request before translation. Missing models are invalid requests; unknown models or wire formats are configuration errors.

pub async fn call_rewrite_model_raw( &self, ctx: Context, raw_http_request: Value, http_headers: Option<HeaderMap>, model: Option<&str>, wire_format: WireFormat, ) -> Result<RawResponse>

The whole decode → call → encode path a wire endpoint needs, in one call.

Decodes raw_http_request from wire_format to the neutral IR, serves it via call_rewrite_model — the upstream wire format is resolved there from the model’s backend, independently of wire_format — then encodes the neutral response back into wire_format. The result is a buffered RawResponse::Buffered JSON body or a streamed RawResponse::Stream of wire events (the caller frames the stream as SSE). The response’s model is restamped with the model that actually served the call, so the body names the model that answered rather than the route the caller addressed.

http_headers are carried through as the request’s Metadata::http_headers and forwarded to the upstream (minus the reserved set); pass None to forward nothing.

Trait Implementations

impl RoutedLlmClient for TranslatingLlmClient

fn call<'life0, 'async_trait>( &'life0 self, ctx: Context, request: Request, decision: Arc<dyn Decision>, ) -> Pin<Box<dyn Future<Output = Result<Response>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Serve the call, returning the model’s response. Call the model named by decision.selected_model() — the target the algorithm routed to — mapping it to whatever provider model id this client hits. request.llm_request.model is the agent’s original name, carried through for reference, not a call target. ctx carries the request’s cross-cutting state.