Skip to main content

ChatModel

studio_worker::host

Trait ChatModel 

Source 
pub trait ChatModel {
    // Required methods
    fn chat(
        &self,
        params: LlmParams,
        cancelled: &dyn Fn() -> bool,
        on_piece: &mut dyn FnMut(&str),
    ) -> Result<Value>;
    fn tokenize(&self, text: &str, add_special: bool) -> Result<Vec<i32>>;
}
Expand description

A loaded model that answers chat completions.

Required Methods§

Source

fn chat( &self, params: LlmParams, cancelled: &dyn Fn() -> bool, on_piece: &mut dyn FnMut(&str), ) -> Result<Value>

Run one completion; cancelled turns true when an unload starts (or a streaming client leaves). on_piece receives the answer’s text as it is generated, stop strings already cut. Returns OpenAI chat.completion-shaped JSON for the whole answer.

Source

fn tokenize(&self, text: &str, add_special: bool) -> Result<Vec<i32>>

The model’s token ids for text; add_special adds BOS as the model would for a prompt.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§