pub trait StreamingModelProvider: Send + Sync {
// Required methods
fn stream_response(&self, context: &Context) -> LlmResponseStream;
fn display_name(&self) -> String;
fn context_window(&self) -> Option<u32>;
// Provided method
fn model(&self) -> Option<LlmModel> { ... }
}Expand description
The core abstraction for all LLM providers. Every provider in this crate implements this trait, and all consumer code should depend on it rather than on concrete types.
The trait follows a streaming design: stream_response returns an LlmResponseStream that yields LlmResponse events as the model generates output. This allows callers to display tokens incrementally rather than waiting for the full response.
§Methods
-
stream_response(&self, context: &Context) -> LlmResponseStream– Send the conversationContextto the model and receive a stream of response events. The stream follows the lifecycle:Start->Text/Reasoning/ToolRequest*->Usage->Done. -
display_name(&self) -> String– A human-readable name for this provider (e.g."Anthropic","OpenRouter"). Used in UI and logging. -
context_window(&self) -> Option<u32>– The model’s context window size in tokens. ReturnsNonefor models where the window is unknown (e.g. Ollama, llama.cpp). -
model(&self) -> Option<LlmModel>– TheLlmModelthis provider is configured to use. ReturnsNonefor test fakes or when the model identity is unknown at compile time.
§Blanket implementations
The trait is implemented for Box<dyn StreamingModelProvider> and Arc<T> where T: StreamingModelProvider, so providers can be used behind trait objects or shared across threads without additional wrapping.
§Usage
use llm::{StreamingModelProvider, Context, ChatMessage, ContentBlock, LlmResponse};
use tokio_stream::StreamExt;
async fn ask(provider: &dyn StreamingModelProvider) {
let context = Context::new(
vec![ChatMessage::User {
content: vec![ContentBlock::text("Hello!")],
timestamp: llm::types::IsoString::now(),
}],
vec![],
);
let mut stream = provider.stream_response(&context);
while let Some(Ok(event)) = stream.next().await {
if let LlmResponse::Text { chunk } = event {
print!("{chunk}");
}
}
}§See also
ProviderFactory– Construction trait, separated for dyn-compatibility.Context– The conversation state passed tostream_response.LlmResponse– The events yielded by the response stream.AlloyedModelProvider– Round-robin wrapper over multiple providers.
Required Methods§
fn stream_response(&self, context: &Context) -> LlmResponseStream
fn display_name(&self) -> String
Sourcefn context_window(&self) -> Option<u32>
fn context_window(&self) -> Option<u32>
Context window size in tokens for the current model.
Returns None for unknown models (e.g. Ollama, LlamaCpp).
Provided Methods§
Trait Implementations§
Source§impl StreamingModelProvider for Box<dyn StreamingModelProvider>
impl StreamingModelProvider for Box<dyn StreamingModelProvider>
fn stream_response(&self, context: &Context) -> LlmResponseStream
fn display_name(&self) -> String
Source§fn context_window(&self) -> Option<u32>
fn context_window(&self) -> Option<u32>
None for unknown models (e.g. Ollama, LlamaCpp).