Skip to main content

LlmProvider

agent_sdk

Trait LlmProvider 

Source 
pub trait LlmProvider: Send + Sync {

Show 14 methods    // Required methods
    fn chat<'life0, 'async_trait>(
        &'life0 self,
        request: ChatRequest,
    ) -> Pin<Box<dyn Future<Output = Result<ChatOutcome, Error>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             Self: 'async_trait;
    fn model(&self) -> &str;
    fn provider(&self) -> &'static str;

    // Provided methods
    fn list_models<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = Result<Vec<ModelInfo>, Error>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             Self: 'async_trait { ... }
    fn probe_connectivity<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = bool> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             Self: 'async_trait { ... }
    fn chat_stream(
        &self,
        request: ChatRequest,
    ) -> Pin<Box<dyn Stream<Item = Result<StreamDelta, Error>> + Send + '_>> { ... }
    fn route(&self) -> &str { ... }
    fn configured_thinking(&self) -> Option<&ThinkingConfig> { ... }
    fn capabilities(&self) -> Option<&'static ModelCapabilities> { ... }
    fn model_features(&self) -> Option<&'static ModelFeatures> { ... }
    fn validate_thinking_config(
        &self,
        thinking: Option<&ThinkingConfig>,
    ) -> Result<(), Error> { ... }
    fn resolve_thinking_config(
        &self,
        request_thinking: Option<&ThinkingConfig>,
    ) -> Result<Option<ThinkingConfig>, Error> { ... }
    fn default_max_tokens(&self) -> u32 { ... }
    fn structured_output_support(&self) -> StructuredOutputSupport { ... }

}

Required Methods§

Source

fn chat<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ) -> Pin<Box<dyn Future<Output = Result<ChatOutcome, Error>> + Send + 'async_trait>>
where 'life0: 'async_trait, Self: 'async_trait,

Non-streaming chat completion.

Source

fn model(&self) -> &str

Source

fn provider(&self) -> &'static str

Provided Methods§

Source

fn list_models<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Vec<ModelInfo>, Error>> + Send + 'async_trait>>
where 'life0: 'async_trait, Self: 'async_trait,

List the models the provider currently exposes, queried live from the provider’s own model-listing endpoint.

The default implementation returns an error: model discovery is an additive capability, so a provider that has not implemented it stays source-compatible while reporting that the operation is unsupported.

§Errors

Returns an error when the provider does not support live model listing, or when the underlying HTTP request or response parsing fails.

Source

fn probe_connectivity<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = bool> + Send + 'async_trait>>
where 'life0: 'async_trait, Self: 'async_trait,

Cheaply check whether the provider’s endpoint is currently reachable, without dispatching a billable request.

Durable runtimes poll this to wait out connectivity loss: after a transport-classified stream failure (StreamErrorKind::is_connectivity) they probe until reachability returns, then re-dispatch the real call. Only an unreachable answer keeps that wait free of charge, so implementations should answer from a cheap endpoint — probe_http_reachability against the API host — never a model call.

The default returns true (“reachable, or unknown”): a provider that cannot observe reachability keeps its callers on the bounded retry path instead of parking them in an offline wait it can never end.

Source

fn chat_stream( &self, request: ChatRequest, ) -> Pin<Box<dyn Stream<Item = Result<StreamDelta, Error>> + Send + '_>>

Streaming chat completion.

Returns a stream of StreamDelta events. The default implementation calls chat() and converts the result to a single-chunk stream.

Providers should override this method to provide true streaming support.

Source

fn route(&self) -> &str

The endpoint that actually serves this provider’s calls.

Self::provider names the API shape a request is built for, and several distinct endpoints share one shape: OpenRouter, Baseten and Fireworks are all reached through OpenAIProvider pointed at a different base_url, so all three answer "openai". Durable audit rows need to tell model-X via a gateway from model-X native, so this reports the serving route instead — the two coincide only for providers that front a single endpoint, which is why the default delegates.

Source

fn configured_thinking(&self) -> Option<&ThinkingConfig>

Provider-owned thinking configuration, if any.

Source

fn capabilities(&self) -> Option<&'static ModelCapabilities>

Canonical capability metadata for this provider/model, if known.

Source

fn model_features(&self) -> Option<&'static ModelFeatures>

API-shape feature metadata for this provider/model, when known.

This complements Self::capabilities with endpoint, reasoning, caching, and tool-control information that does not fit the legacy context/pricing record.

Source

fn validate_thinking_config( &self, thinking: Option<&ThinkingConfig>, ) -> Result<(), Error>

Validate a thinking configuration against the provider/model capabilities.

§Errors

Returns an error when the requested thinking mode is not supported by the active provider/model capability set.

Source

fn resolve_thinking_config( &self, request_thinking: Option<&ThinkingConfig>, ) -> Result<Option<ThinkingConfig>, Error>

Resolve the effective thinking configuration for a request.

Request-level thinking overrides provider-owned defaults when present.

§Errors

Returns an error when the resolved thinking configuration is not supported by the active provider/model capability set.

Source

fn default_max_tokens(&self) -> u32

Default maximum output tokens for this provider/model when the caller does not explicitly override AgentConfig.max_tokens.

Source

fn structured_output_support(&self) -> StructuredOutputSupport

How this provider satisfies a structured-output (ResponseFormat) request.

Providers with a native JSON-schema / JSON-mode wire field (OpenAI-family, Gemini, Vertex) report StructuredOutputSupport::Native and consume request.response_format directly. Providers without one (Anthropic) report StructuredOutputSupport::ToolForcing so the runner forces a single “respond” tool whose schema is the output schema. The default is the conservative StructuredOutputSupport::ToolForcing, which works for any tool-capable provider.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§