Trait ModelProvider

Source

pub trait ModelProvider {
    // Required methods
    fn name(&self) -> &str;
    fn complete(
        &mut self,
        request: &CompletionRequest,
    ) -> Result<CompletionResponse, ProviderError>;

    // Provided methods
    fn streams(&self) -> bool { ... }
    fn complete_streaming(
        &mut self,
        request: &CompletionRequest,
        on_delta: DeltaSink<'_>,
    ) -> Result<CompletionResponse, ProviderError> { ... }
}

Expand description

A source of model completions.

Required Methods§

Source

fn name(&self) -> &str

Short name used in events and diagnostics, e.g. anthropic or replay.

Source

fn complete( &mut self, request: &CompletionRequest, ) -> Result<CompletionResponse, ProviderError>

Provided Methods§

Source

fn streams(&self) -> bool

Whether this provider delivers an answer incrementally.

Read by the interpreter before it decides how many output tokens one call may ask for: a service that must compose a whole body before sending it holds the connection open for the length of the answer, and several refuse a large max_tokens outright unless the request streams. The ceiling is therefore a property of the transport, and a provider that says false here keeps the smaller one.

Source

fn complete_streaming( &mut self, request: &CompletionRequest, on_delta: DeltaSink<'_>, ) -> Result<CompletionResponse, ProviderError>

Complete a request, handing text to on_delta as it arrives.

The default is ModelProvider::complete with the deltas dropped, which is the honest answer for a provider that has nothing live to show — a cassette replay produces its answer at once, and inventing deltas for it would make a replayed run look like a call that never happened.

The returned response is what the run uses. Whatever reached on_delta is a display artifact: on any error it is discarded, including when the answer was cut off part-way through.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types§

Source §

impl<P: ModelProvider + ?Sized> ModelProvider for Box<P>

Lets a boxed provider be used wherever a provider is expected — including as the inner provider of a crate::RecordingProvider, which is how the CLI wraps a recorder around a provider it chose at runtime.

Source §