Trait llm::Model

pub trait Model: Send + Sync {
    // Required methods
    fn start_session(&self, config: InferenceSessionConfig) -> InferenceSession;
    fn evaluate(
        &self,
        session: &mut InferenceSession,
        params: &InferenceParameters,
        input_tokens: &[i32],
        output_request: &mut OutputRequest
    );
    fn vocabulary(&self) -> &Vocabulary;
    fn n_context_tokens(&self) -> usize;
    fn bot_token_id(&self) -> Option<i32>;
    fn eot_token_id(&self) -> i32;
    fn inference_parameters(&self) -> &InferenceParameters;
}

A type-erased model to allow for interacting with a model without knowing its hyperparameters.

Required Methods

fn start_session(&self, config: InferenceSessionConfig) -> InferenceSession

Starts a new InferenceSession for this model.

fn evaluate( &self, session: &mut InferenceSession, params: &InferenceParameters, input_tokens: &[i32], output_request: &mut OutputRequest )

This function is called by the provided InferenceSession; it will use this model and the InferenceParameters to generate output by evaluating the input_tokens. The OutputRequest is used to specify additional data to fetch from the model.

fn vocabulary(&self) -> &Vocabulary

Get the vocabulary (loaded from the GGML file) for this model.

fn n_context_tokens(&self) -> usize

Get the context size (configured with ModelParameters::n_context_tokens) used by this model.

fn bot_token_id(&self) -> Option<i32>

Get the beginning of text/beginning of string token ID, if available. This value is defined by model implementers.

fn eot_token_id(&self) -> i32

Get the end of text/end of string token ID. This value is defined by model implementers.

fn inference_parameters(&self) -> &InferenceParameters

Get the default InferenceParameters for this model (used by InferenceSession::infer). This value is configured through ModelParameters::inference_parameters.

Implementors

impl<H, M> Model for Mwhere H: Hyperparameters, M: KnownModel<Hyperparameters = H>,