pub trait StreamingModelProvider: Send + Sync {
// Required method
fn stream_response(&self, context: &Context) -> LlmResponseStream;
// Provided methods
fn display_name(&self) -> String { ... }
fn context_window(&self) -> Option<u32> { ... }
fn model(&self) -> Option<LlmModel> { ... }
}Expand description
The core abstraction for all LLM providers. Every provider in this crate implements this trait, and all consumer code should depend on it rather than on concrete types.
The trait follows a streaming design: stream_response returns an LlmResponseStream that yields LlmResponse events as the model generates output. This allows callers to display tokens incrementally rather than waiting for the full response.
§Methods
-
stream_response(&self, context: &Context) -> LlmResponseStream– Send the conversationContextto the model and receive a stream of response events. The stream follows the lifecycle:Start->Text/Reasoning/ToolRequest*->Usage->Done. -
model(&self) -> Option<LlmModel>– TheLlmModelthis provider is configured to use. ReturnsNonefor test fakes or when the model identity is unknown at compile time. -
display_name(&self) -> String– A human-readable provider and model (e.g."Anthropic (claude-sonnet-4-5)"). Used in UI and logging. Defaults to themodel’s provider display name and ID. -
context_window(&self) -> Option<u32>– The model’s context window size in tokens. Defaults to themodel’s catalog entry;Nonewhere the window is unknown (e.g. Ollama, llama.cpp).
§Blanket implementations
The trait is implemented for Box<dyn StreamingModelProvider> and Arc<T> where T: StreamingModelProvider, so providers can be used behind trait objects or shared across threads without additional wrapping.
§Usage
use llm::{StreamingModelProvider, Context, ChatMessage, ContentBlock, LlmResponse};
use tokio_stream::StreamExt;
async fn ask(provider: &dyn StreamingModelProvider) {
let context = Context::new(
vec![ChatMessage::User {
message_id: llm::MessageId::new(),
content: vec![ContentBlock::text("Hello!")],
timestamp: llm::types::IsoString::now(),
}],
vec![],
);
let mut stream = provider.stream_response(&context);
while let Some(Ok(event)) = stream.next().await {
if let LlmResponse::Text { chunk } = event {
print!("{chunk}");
}
}
}§See also
Context– The conversation state passed tostream_response.LlmResponse– The events yielded by the response stream.AlloyedModelProvider– Round-robin wrapper over multiple providers.
Required Methods§
fn stream_response(&self, context: &Context) -> LlmResponseStream
Provided Methods§
Sourcefn display_name(&self) -> String
fn display_name(&self) -> String
Provider and model, e.g. Anthropic (claude-sonnet-4-5).
Sourcefn context_window(&self) -> Option<u32>
fn context_window(&self) -> Option<u32>
Context window size in tokens for the current model.
Returns None for unknown models (e.g. Ollama, LlamaCpp).
Trait Implementations§
Source§impl StreamingModelProvider for Box<dyn StreamingModelProvider>
impl StreamingModelProvider for Box<dyn StreamingModelProvider>
fn stream_response(&self, context: &Context) -> LlmResponseStream
Source§fn display_name(&self) -> String
fn display_name(&self) -> String
Anthropic (claude-sonnet-4-5).Source§fn context_window(&self) -> Option<u32>
fn context_window(&self) -> Option<u32>
None for unknown models (e.g. Ollama, LlamaCpp).Dyn Compatibility§
This trait is dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety".