Trait Provider 

pub trait Provider: Send + Sync {
    // Required methods
    fn chat_with_context<'life0, 'life1, 'async_trait>(
        &'life0 self,
        request: ChatRequest,
        context: &'life1 ProviderRequestContext,
    ) -> Pin<Box<dyn Future<Output = Result<ChatResponse, ProviderError>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             'life1: 'async_trait,
             Self: 'async_trait;
    fn stream_chat_with_context<'life0, 'life1, 'async_trait>(
        &'life0 self,
        request: ChatRequest,
        context: &'life1 ProviderRequestContext,
    ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, ProviderError>> + Send>>, ProviderError>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             'life1: 'async_trait,
             Self: 'async_trait;

    // Provided methods
    fn model(&self) -> Option<&str> { ... }
    fn capabilities(&self) -> ProviderCapabilities { ... }
    fn chat<'life0, 'async_trait>(
        &'life0 self,
        request: ChatRequest,
    ) -> Pin<Box<dyn Future<Output = Result<ChatResponse, ProviderError>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             Self: 'async_trait { ... }
    fn stream_chat<'life0, 'async_trait>(
        &'life0 self,
        request: ChatRequest,
    ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, ProviderError>> + Send>>, ProviderError>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             Self: 'async_trait { ... }
}

The interface for chatting with an LLM.

Implementations are responsible for communicating with a specific LLM service and mapping vendor responses back to this framework’s Message; chat returns the full reply at once, while stream_chat returns the same reply incrementally as a stream of events. Both share the same semantics and differ only in delivery.

Send + Sync guarantees that Box<dyn Provider> can be held across awaits in Agent implementations (for the same reason as Tool).

Examples

The calling convention is identical for every implementation; the example below uses FakeProvider:

use molo::provider::{ChatRequest, FakeProvider, FakeReply, Provider};

let fake = FakeProvider::new([FakeReply::Text("hi".into())]);
let response = fake.chat(ChatRequest::default()).await?;
assert_eq!(response.message, molo::message::Message::assistant("hi"));

Required Methods

fn chat_with_context<'life0, 'life1, 'async_trait>( &'life0 self, request: ChatRequest, context: &'life1 ProviderRequestContext, ) -> Pin<Box<dyn Future<Output = Result<ChatResponse, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Sends one turn with request-scoped provider context.

Errors

Network failures / timeouts / rate limits / vendor business errors are all returned as ProviderError; see that type’s docs for error classification and retry guidance.

fn stream_chat_with_context<'life0, 'life1, 'async_trait>( &'life0 self, request: ChatRequest, context: &'life1 ProviderRequestContext, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, ProviderError>> + Send>>, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Streams one turn with request-scoped provider context.

Runtimes should call this method so providers can observe cancellation, deadlines, request ids, and sanitized metadata.

Provided Methods

fn model(&self) -> Option<&str>

Model identifier exposed by this provider, when known.

Agents copy this value into run summaries for observability. Providers that are not bound to one model can keep the default None.

fn capabilities(&self) -> ProviderCapabilities

Capability metadata for this provider instance.

fn chat<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ) -> Pin<Box<dyn Future<Output = Result<ChatResponse, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Sends one turn of conversation and returns the model’s reply (text, or a request to call tools).

This direct-use convenience wrapper creates a generated run context. Runtimes that already have a RunContext should call chat_with_context.

fn stream_chat<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, ProviderError>> + Send>>, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Self: 'async_trait,

Streams one turn of conversation for direct provider use.

Semantically identical to chat, except that the reply is returned as a stream of events: several StreamEvent::Delta items concatenated in order form the full reply, and the stream ends with StreamEvent::Done.

Errors

Failures during request setup (connection / timeout / vendor rejection) are returned as Err; event errors after the stream is established are produced as Err items in the stream, and no success events are produced after an error item (see StreamEvent for termination semantics).

Trait Implementations

impl Provider for Box<dyn Provider>

Box<dyn Provider> is itself a Provider: re-exposes the trait object as a value, for assembly patterns that need to “hold an instance and create a new loop per call” (e.g., a sub-agent factory that captures a provider and constructs a fresh loop for each invocation).

fn model(&self) -> Option<&str>

Model identifier exposed by this provider, when known.

fn capabilities(&self) -> ProviderCapabilities

Capability metadata for this provider instance.

fn chat<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ) -> Pin<Box<dyn Future<Output = Result<ChatResponse, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Box<dyn Provider>: 'async_trait,

Sends one turn of conversation and returns the model’s reply (text, or a request to call tools).

fn chat_with_context<'life0, 'life1, 'async_trait>( &'life0 self, request: ChatRequest, context: &'life1 ProviderRequestContext, ) -> Pin<Box<dyn Future<Output = Result<ChatResponse, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Box<dyn Provider>: 'async_trait,

Sends one turn with request-scoped provider context.

fn stream_chat<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, ProviderError>> + Send>>, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Box<dyn Provider>: 'async_trait,

Streams one turn of conversation for direct provider use.

fn stream_chat_with_context<'life0, 'life1, 'async_trait>( &'life0 self, request: ChatRequest, context: &'life1 ProviderRequestContext, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, ProviderError>> + Send>>, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Box<dyn Provider>: 'async_trait,

Streams one turn with request-scoped provider context.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types

impl Provider for Arc<FakeProvider>

Arc<FakeProvider> is also a Provider: tests share one instance behind an Arc (used when wrapper tests assert internal request counts, such as RetryProvider’s attempt count).

fn capabilities(&self) -> ProviderCapabilities

fn chat<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ) -> Pin<Box<dyn Future<Output = Result<ChatResponse, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Arc<FakeProvider>: 'async_trait,

fn chat_with_context<'life0, 'life1, 'async_trait>( &'life0 self, request: ChatRequest, context: &'life1 ProviderRequestContext, ) -> Pin<Box<dyn Future<Output = Result<ChatResponse, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Arc<FakeProvider>: 'async_trait,

fn stream_chat<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, ProviderError>> + Send>>, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Arc<FakeProvider>: 'async_trait,

fn stream_chat_with_context<'life0, 'life1, 'async_trait>( &'life0 self, request: ChatRequest, context: &'life1 ProviderRequestContext, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, ProviderError>> + Send>>, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Arc<FakeProvider>: 'async_trait,

impl Provider for Box<dyn Provider>

Box<dyn Provider> is itself a Provider: re-exposes the trait object as a value, for assembly patterns that need to “hold an instance and create a new loop per call” (e.g., a sub-agent factory that captures a provider and constructs a fresh loop for each invocation).

fn model(&self) -> Option<&str>

fn capabilities(&self) -> ProviderCapabilities

fn chat<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ) -> Pin<Box<dyn Future<Output = Result<ChatResponse, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Box<dyn Provider>: 'async_trait,

fn chat_with_context<'life0, 'life1, 'async_trait>( &'life0 self, request: ChatRequest, context: &'life1 ProviderRequestContext, ) -> Pin<Box<dyn Future<Output = Result<ChatResponse, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Box<dyn Provider>: 'async_trait,

fn stream_chat<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, ProviderError>> + Send>>, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, Box<dyn Provider>: 'async_trait,

fn stream_chat_with_context<'life0, 'life1, 'async_trait>( &'life0 self, request: ChatRequest, context: &'life1 ProviderRequestContext, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, ProviderError>> + Send>>, ProviderError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Box<dyn Provider>: 'async_trait,

Implementors

impl Provider for FakeProvider

impl<I> Provider for RetryProvider<I>

where I: Provider + Send + Sync,