Skip to main content

ModelProvider

mermaid_cli::providers::model

Trait ModelProvider 

Source 
pub trait ModelProvider: Send + Sync {
    // Required methods
    fn capabilities(&self) -> &Capabilities;
    fn chat<'life0, 'async_trait>(
        &'life0 self,
        request: ChatRequest,
        ctx: StreamContext,
    ) -> Pin<Box<dyn Future<Output = Result<FinalResponse>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;

    // Provided methods
    fn resolve_context_window<'life0, 'life1, 'async_trait>(
        &'life0 self,
        request: &'life1 ChatRequest,
    ) -> Pin<Box<dyn Future<Output = ContextSizing> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait { ... }
    fn verify_placement<'life0, 'async_trait>(
        &'life0 self,
        current_num_ctx: Option<usize>,
    ) -> Pin<Box<dyn Future<Output = Option<ModelPlacement>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait { ... }
}
Expand description

Provider-facing interface. A ModelProvider impl owns whatever HTTP client / state it needs and exposes chat() — that’s the whole surface.

Required Methods§

Source

fn capabilities(&self) -> &Capabilities

Capabilities the provider advertises. The reducer reads this when building the outgoing ChatRequest (e.g. whether to attach reasoning controls).

Source

fn chat<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ctx: StreamContext, ) -> Pin<Box<dyn Future<Output = Result<FinalResponse>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Stream a chat turn. Typed events flow through ctx.sink; the returned FinalResponse carries token usage and the Anthropic thinking-signature (opaque blob required to continue extended thinking across turns).

Cancellation: the provider MUST select! on ctx.token. cancelled() inside any await that could block for more than a few hundred ms. This is the contract that replaces the old check_interrupt polling pattern.

Provided Methods§

Source

fn resolve_context_window<'life0, 'life1, 'async_trait>( &'life0 self, request: &'life1 ChatRequest, ) -> Pin<Box<dyn Future<Output = ContextSizing> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Resolve the effective context window for a turn (what the model will actually enforce). The default returns the static advertised window; Ollama overrides this to probe the model’s real window and auto-fit num_ctx to host memory, honoring the request’s per-model ollama_num_ctx override. None means “let the backend decide”.

Awaited only on the effect runtime (never the reducer), so a probe never blocks the UI.

Source

fn verify_placement<'life0, 'async_trait>( &'life0 self, current_num_ctx: Option<usize>, ) -> Pin<Box<dyn Future<Output = Option<ModelPlacement>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Best-effort: where the loaded model currently sits in memory. The default returns None (unknown / not applicable); Ollama overrides it to probe /api/ps. Awaited only on the effect runtime, after a turn (when the model is resident), so it never blocks the UI.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§