Trait kalosm_language_model::SyncModel

source · 
pub trait SyncModel {
    type Session: Session;

    // Required methods
    fn new_session(&self) -> Result<Self::Session>;
    fn feed_text(
        &self,
        session: &mut Self::Session,
        prompt: &str,
        top_k: Option<usize>
    ) -> Result<Logits>;
    fn feed_tokens(
        &self,
        session: &mut Self::Session,
        tokens: &[u32],
        top_k: Option<usize>
    ) -> Result<Logits>;
    fn stop_token(&self) -> Result<u32>;
    fn tokenizer(&self) -> Arc<dyn Tokenizer + Send + Sync>;
}
Expand description

A raw interface for a model that can be used to generate text synchronously. This provides a very low level interface to a model’s session:

§Example

use rphi::prelude::*;
use kalosm_language_model::Model;

#[tokio::main]
async fn main() {
    let mut llm = Phi::start().await;

    let tokenizer = llm.tokenizer();
    // Start a sync task on the model
    llm.run_sync(move |llm: &mut <Phi as Model>::SyncModel| {
        Box::pin(async move {
            let question = "What is 10 + 10?";

            // Create a new session of the model
            let mut session = llm.new_session().unwrap();

            // Feed the question into the model
            let mut logits = llm.feed_text(&mut session, question, None).unwrap();

            println!("logits: {:?}", logits);
        })
    })
    .unwrap();
}

Required Associated Types§

source

type Session: Session

The session type for this model.

Required Methods§

source

fn new_session(&self) -> Result<Self::Session>

Create a new session for this model.

source

fn feed_text( &self, session: &mut Self::Session, prompt: &str, top_k: Option<usize> ) -> Result<Logits>

Run the model synchronously. The model implementation may choose to return only the top k logits.

source

fn feed_tokens( &self, session: &mut Self::Session, tokens: &[u32], top_k: Option<usize> ) -> Result<Logits>

Run the model synchronously with a pre-tokenized input. The model implementation may choose to return only the top k logits.

source

fn stop_token(&self) -> Result<u32>

Get the token ID that represents the end of a sequence.

source

fn tokenizer(&self) -> Arc<dyn Tokenizer + Send + Sync>

Return the tokenizer associated with this model.

Implementors§