Trait LargeLanguageModel

pub trait LargeLanguageModel {
    // Required method
    fn infer_stream(
        &self,
        prompt: Prompt,
    ) -> Result<impl Future<Output = Result<StringStream>> + Send>;

    // Provided method
    fn infer(
        &self,
        prompt: Prompt,
    ) -> Result<impl Future<Output = Result<String>> + Send> { ... }
}

LLM

Required Methods

fn infer_stream( &self, prompt: Prompt, ) -> Result<impl Future<Output = Result<StringStream>> + Send>

Run inference on a model, returning a stream

Provided Methods

fn infer( &self, prompt: Prompt, ) -> Result<impl Future<Output = Result<String>> + Send>

Run inference on a model, return once the complete answer has been computed. The default implementation call infer_stream until completion of the stream.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors