Skip to main content

Provider

llm_stack::provider

Trait Provider 

Source 
pub trait Provider: Send + Sync {
    // Required methods
    fn generate(
        &self,
        params: &ChatParams,
    ) -> impl Future<Output = Result<ChatResponse, LlmError>> + Send;
    fn stream(
        &self,
        params: &ChatParams,
    ) -> impl Future<Output = Result<ChatStream, LlmError>> + Send;
    fn metadata(&self) -> ProviderMetadata;
}
Expand description

The core trait every LLM provider implements.

Provider uses native async-fn-in-traits (Rust 2024 edition). Implementations are plain async fns — no #[async_trait] needed.

Cross-cutting concerns like retries, rate-limiting, and logging are handled by the interceptor system, keeping individual backends focused on HTTP mapping.

§Object safety

Provider is not object-safe because AFIT returns impl Future. When you need dynamic dispatch (e.g. Box<dyn _> or Arc<dyn _>), use DynProvider instead — every Provider automatically implements DynProvider via a blanket impl.

Required Methods§

Source

fn generate( &self, params: &ChatParams, ) -> impl Future<Output = Result<ChatResponse, LlmError>> + Send

Sends a chat completion request and returns the full response.

Source

fn stream( &self, params: &ChatParams, ) -> impl Future<Output = Result<ChatStream, LlmError>> + Send

Sends a chat completion request and returns a stream of events.

The returned ChatStream yields StreamEvents as they arrive from the provider.

Source

fn metadata(&self) -> ProviderMetadata

Returns static metadata describing this provider instance.

Dyn Compatibility§

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors§