Struct llama_cpp_2::model::LlamaModel

pub struct LlamaModel { /* private fields */ }

A safe wrapper around llama_model.

Implementations

impl LlamaModel

pub fn n_ctx_train(&self) -> u16

get the number of tokens the model was trained on

Panics

If the number of tokens the model was trained on does not fit into an u16. This should be impossible on most platforms due to llama.cpp returning a c_int (i32 on most platforms) which is almost certainly positive.

pub fn tokens( &self ) -> impl Iterator<Item = (LlamaToken, Result<String, TokenToStringError>)> + '_

Get all tokens in the model.

pub fn token_bos(&self) -> LlamaToken

Get the beginning of stream token.

pub fn token_eos(&self) -> LlamaToken

Get the end of stream token.

pub fn token_nl(&self) -> LlamaToken

Get the newline token.

pub fn token_to_str( &self, token: LlamaToken ) -> Result<String, TokenToStringError>

Convert single token to a string.

Errors

See TokenToStringError for more information.

pub fn tokens_to_str( &self, tokens: &[LlamaToken] ) -> Result<String, TokenToStringError>

Convert a vector of tokens to a single string.

Errors

See TokenToStringError for more information.

pub fn str_to_token( &self, str: &str, add_bos: bool ) -> Result<Vec<LlamaToken>, StringToTokenError>

Convert a string to a Vector of tokens.

Errors

• if str contains a null byte.

Panics

• if there is more than usize::MAX LlamaTokens in str.

use llama_cpp_2::model::LlamaModel;

use std::path::Path;
let backend = llama_cpp_2::llama_backend::LlamaBackend::init()?;
let model = LlamaModel::load_from_file(&backend, Path::new("path/to/model"), &Default::default())?;
let tokens = model.str_to_token("Hello, World!", true)?;

pub fn token_type(&self, LlamaToken: LlamaToken) -> LlamaTokenType

Get the type of a token.

Panics

If the token type is not known to this library.

pub fn token_to_str_with_size( &self, token: LlamaToken, buffer_size: usize ) -> Result<String, TokenToStringError>

Convert a token to a string with a specified buffer size.

Generally you should use LlamaModel::token_to_str instead as 8 bytes is enough for most words and the extra bytes do not really matter.

Errors

• if the token type is unknown
• the resultant token is larger than buffer_size.
• the string returend by llama-cpp is not valid utf8.

Panics

• if [buffer_size] does not fit into a c_int.
• if the returned size from llama-cpp does not fit into a usize. (this should never happen)

pub fn n_vocab(&self) -> i32

The number of tokens the model was trained on.

This returns a c_int for maximum compatibility. Most of the time it can be cast to an i32 without issue.

pub fn vocab_type(&self) -> VocabType

The type of vocab the model was trained on.

Panics

If llama-cpp emits a vocab type that is not known to this library.

pub fn n_embd(&self) -> c_int

This returns a c_int for maximum compatibility. Most of the time it can be cast to an i32 without issue.

pub fn load_from_file( _: &LlamaBackend, path: impl AsRef<Path>, params: &LlamaModelParams ) -> Result<Self, LlamaModelLoadError>

loads a model from a file.

Errors

See LlamaModelLoadError for more information.

pub fn new_context<'a>( &'a self, _: &LlamaBackend, params: &LlamaContextParams ) -> Result<LlamaContext<'a>, LlamaContextLoadError>

Create a new context from this model.

Errors

There is many ways this can fail. See LlamaContextLoadError for more information.

Trait Implementations

impl Debug for LlamaModel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for LlamaModel

fn drop(&mut self)

Executes the destructor for this type.

impl Send for LlamaModel

impl Sync for LlamaModel