Struct LlamaModel 

pub struct LlamaModel { /* private fields */ }

A safe wrapper around llama_model.

Implementations

impl LlamaModel

pub fn vocab(&self) -> LlamaVocab<'_>

Get the model’s vocabulary.

pub fn n_ctx_train(&self) -> u32

get the number of tokens the model was trained on

Panics

If the number of tokens the model was trained on does not fit into an u32. This should be impossible on most platforms due to llama.cpp returning a c_int (i32 on most platforms) which is almost certainly positive.

pub fn tokens( &self, decode_special: bool, ) -> impl Iterator<Item = (LlamaToken, Vec<u8>)> + '_

Get all tokens in the model.

pub fn decode_start_token(&self) -> LlamaToken

Get the decoder start token.

pub fn n_vocab(&self) -> i32

The number of tokens the model was trained on.

This returns a c_int for maximum compatibility. Most of the time it can be cast to an i32 without issue.

pub fn n_embd(&self) -> c_int

This returns a c_int for maximum compatibility. Most of the time it can be cast to an i32 without issue.

pub fn n_embd_out(&self) -> c_int

The model’s output embedding width (n_embd_out). This is the width llama.cpp actually extracts embeddings at — n_embd and n_embd_out diverge when {arch}.embedding_length_out is present (deepstack models like qwen3vl). Returns a c_int for maximum compatibility.

pub fn n_cls_out(&self) -> u32

The model’s classification output width (n_cls_out, default 1) — the width of a RANK-pooled embeddings read (llama.h:1029).

pub fn size(&self) -> u64

Returns the total size of all the tensors in the model in bytes.

pub fn n_params(&self) -> u64

Returns the number of parameters in the model.

pub fn is_recurrent(&self) -> bool

Returns whether the model is a recurrent network (Mamba, RWKV, etc)

pub fn is_hybrid(&self) -> bool

Returns whether the model is a hybrid network (Jamba, Granite, Qwen3xx, etc)

Hybrid models have both attention layers and recurrent/SSM layers. They require special handling for state checkpointing.

pub fn n_layer(&self) -> u32

Returns the number of layers within the model.

pub fn n_head(&self) -> u32

Returns the number of attention heads within the model.

pub fn n_head_kv(&self) -> u32

Returns the number of KV attention heads.

pub fn meta_val_str(&self, key: &str) -> Result<String, MetaValError>

Get metadata value as a string by key name

pub fn meta_count(&self) -> i32

Get the number of metadata key/value pairs

pub fn meta_key_by_index(&self, index: i32) -> Result<String, MetaValError>

Get metadata key name by index

pub fn meta_val_str_by_index(&self, index: i32) -> Result<String, MetaValError>

Get metadata value as a string by index

pub fn rope_type(&self) -> Option<RopeType>

Returns the rope type of the model.

pub fn chat_template( &self, name: Option<&str>, ) -> Result<LlamaChatTemplate, ChatTemplateError>

Get chat template from model by name. If the name parameter is None, the default chat template will be returned.

You supply this into Self::apply_chat_template to get back a string with the appropriate template substitution applied to convert a list of messages into a prompt the LLM can use to complete the chat.

You could also use an external jinja parser, like minijinja, to parse jinja templates not supported by the llama.cpp template engine.

Errors

• If the model has no chat template by that name
• If the chat template is not a valid CString.

pub fn load_from_file( _: &LlamaBackend, path: impl AsRef<Path>, params: &LlamaModelParams, ) -> Result<Self, LlamaModelLoadError>

Loads a model from a file.

Errors

See LlamaModelLoadError for more information.

pub fn lora_adapter_init( &mut self, path: impl AsRef<Path>, ) -> Result<LlamaLoraAdapter, LlamaLoraAdapterInitError>

Initializes a lora adapter from a file.

Errors

See LlamaLoraAdapterInitError for more information.

pub fn new_context<'a>( &'a self, _: &LlamaBackend, params: LlamaContextParams, ) -> Result<LlamaContext<'a>, LlamaContextLoadError>

Create a new context from this model.

Errors

There is many ways this can fail. See LlamaContextLoadError for more information.

pub fn new_context_with_ctx_other<'a>( &'a self, _: &LlamaBackend, params: LlamaContextParams, ctx_other: &LlamaContext<'_>, ) -> Result<LlamaContext<'a>, LlamaContextLoadError>

Create a new context bound to another context via llama.cpp’s ctx_other field.

This is required for MTP speculative decoding when the target model’s architecture uses LLM_ARCH_GEMMA4_ASSISTANT, which asserts that the draft context references the target context so KV state can be shared.

Errors

See LlamaContextLoadError.

pub fn new_context_with_samplers<'a>( &'a self, _: &LlamaBackend, params: LlamaContextParams, samplers: impl IntoIterator<Item = (i32, LlamaSampler)>, ) -> Result<LlamaContext<'a>, LlamaContextLoadError>

Creates a new context with backend samplers attached for specific sequences.

Ownership of the samplers is transferred to the context, ensuring they remain alive for the context’s lifetime. Only samplers that support backend execution (greedy, dist, temp, top_k, top_p, min_p, logit_bias) will run on the backend.

Arguments

• params - Context parameters
• samplers - Iterator of (seq_id, sampler) pairs where sampler must be a chain

Example

let sampler = LlamaSampler::chain([
    LlamaSampler::min_p(0.01, 64),
    LlamaSampler::temp(0.1),
    LlamaSampler::dist(42),
], false);

let ctx = model.new_context_with_samplers(
    &backend,
    ctx_params,
    [(0, sampler)],
)?;

pub fn apply_chat_template( &self, tmpl: &LlamaChatTemplate, chat: &[LlamaChatMessage], add_ass: bool, ) -> Result<String, ApplyChatTemplateError>

Apply the models chat template to some messages. See https://github.com/ggerganov/llama.cpp/wiki/Templates-supported-by-llama_chat_apply_template

Unlike the llama.cpp apply_chat_template which just randomly uses the ChatML template when given a null pointer for the template, this requires an explicit template to be specified. If you want to use “chatml”, then just do LlamaChatTemplate::new("chatml") or any other model name or template string.

Use Self::chat_template to retrieve the template baked into the model (this is the preferred mechanism as using the wrong chat template can result in really unexpected responses from the LLM).

You probably want to set add_ass to true so that the generated template string ends with a the opening tag of the assistant. If you fail to leave a hanging chat tag, the model will likely generate one into the output and the output may also have unexpected output aside from that.

Errors

There are many ways this can fail. See ApplyChatTemplateError for more information.

Trait Implementations

impl Debug for LlamaModel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for LlamaModel

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for LlamaModel

impl Sync for LlamaModel