Struct LlamaContext 

pub struct LlamaContext<'a> {
    pub model: &'a LlamaModel,
    /* private fields */
}

A safe wrapper around the llama_context C++ context.

This struct provides a safe interface to interact with the llama_context used by the LlamaModel. It encapsulates the raw C++ context pointer and provides additional fields for managing the model and context-specific settings like embeddings and logits.

The LlamaContext struct ensures that the C++ context is always valid by using the NonNull type for the context pointer, preventing it from being null. The struct also holds a reference to the model (LlamaModel) that the context is tied to, along with some internal state like whether embeddings are enabled and the initialized logits for the context.

Fields

• context: A non-null pointer to the raw C++ llama_context. This is the main context used for interacting with the model.
• model: A reference to the LlamaModel associated with this context. This model provides the data and parameters that the context interacts with.
• initialized_logits: A vector used to store the initialized logits. These are used in the model’s processing and are kept separate from the context data.
• embeddings_enabled: A boolean flag indicating whether embeddings are enabled in the context. This is useful for controlling whether embedding data is generated during the interaction with the model.

Fields

model: &'a LlamaModel

a reference to the contexts model.

Implementations

impl LlamaContext<'_>

pub fn copy_cache(&mut self, src: i32, dest: i32, size: i32)

Copy the cache from one sequence to another.

Parameters

• src - The sequence id to copy the cache from.
• dest - The sequence id to copy the cache to.
• size - The size of the cache to copy.

pub fn copy_kv_cache_seq( &mut self, src: i32, dest: i32, p0: Option<u32>, p1: Option<u32>, ) -> Result<(), KvCacheConversionError>

Copy the cache from one sequence to another.

Parameters

• src - The sequence id to copy the cache from.
• dest - The sequence id to copy the cache to.
• p0 - The start position of the cache to clear. If None, the entire cache is copied up to p1.
• p1 - The end position of the cache to clear. If None, the entire cache is copied starting from p0.

Errors

Returns KvCacheConversionError if either position exceeds the maximum i32 value.

pub fn clear_kv_cache_seq( &mut self, src: Option<u32>, p0: Option<u32>, p1: Option<u32>, ) -> Result<bool, KvCacheConversionError>

Clear the kv cache for the given sequence within the specified range [p0, p1).

Returns false only when partial sequence removals fail. Full sequence removals always succeed.

Parameters

• src - The sequence id to clear the cache for. If None, matches all sequences
• p0 - The start position of the cache to clear. If None, the entire cache is cleared up to p1.
• p1 - The end position of the cache to clear. If None, the entire cache is cleared from p0.

Errors

Returns KvCacheConversionError if the sequence id or either position exceeds the maximum i32 value.

pub fn clear_kv_cache(&mut self)

Clear the KV cache

pub fn llama_kv_cache_seq_keep(&mut self, seq_id: i32)

Removes all tokens that do not belong to the specified sequence

Parameters

• seq_id - The sequence id to keep

pub fn kv_cache_seq_add( &mut self, seq_id: i32, p0: Option<u32>, p1: Option<u32>, delta: i32, ) -> Result<(), KvCacheConversionError>

Adds relative position “delta” to all tokens that belong to the specified sequence and have positions in [p0, p1).

Parameters

• seq_id - The sequence id to update
• p0 - The start position of the cache to update. If None, the entire cache is updated up to p1.
• p1 - The end position of the cache to update. If None, the entire cache is updated starting from p0.
• delta - The relative position to add to the tokens

Errors

Returns KvCacheConversionError if either position exceeds the maximum i32 value.

pub fn kv_cache_seq_div( &mut self, seq_id: i32, p0: Option<u32>, p1: Option<u32>, d: NonZeroU8, ) -> Result<(), KvCacheConversionError>

Integer division of the positions by factor of d > 1.

Parameters

• seq_id - The sequence id to update
• p0 - The start position of the cache to update. If None, the entire cache is updated up to p1.
• p1 - The end position of the cache to update. If None, the entire cache is updated starting from p0.
• d - The factor to divide the positions by

Errors

Returns KvCacheConversionError if either position exceeds the maximum i32 value.

pub fn kv_cache_seq_pos_max(&self, seq_id: i32) -> i32

Returns the largest position present in the KV cache for the specified sequence

Parameters

• seq_id - The sequence id to get the max position for

impl LlamaContext<'_>

pub fn save_session_file( &self, path_session: impl AsRef<Path>, tokens: &[LlamaToken], ) -> Result<(), SaveSessionError>

Save the current session to a file.

Parameters

• path_session - The file to save to.
• tokens - The tokens to associate the session with. This should be a prefix of a sequence of tokens that the context has processed, so that the relevant KV caches are already filled.

Errors

Fails if the path is not a valid utf8, is not a valid c string, or llama.cpp fails to save the session file.

pub fn load_session_file( &mut self, path_session: impl AsRef<Path>, max_tokens: usize, ) -> Result<Vec<LlamaToken>, LoadSessionError>

Load a session file into the current context.

You still need to pass the returned tokens to the context for inference to work. What this function buys you is that the KV caches are already filled with the relevant data.

Parameters

• path_session - The file to load from. It must be a session file from a compatible context, otherwise the function will error.
• max_tokens - The maximum token length of the loaded session. If the session was saved with a longer length, the function will error.

Errors

Fails if the path is not a valid utf8, is not a valid c string, or llama.cpp fails to load the session file. (e.g. the file does not exist, is not a session file, etc.)

pub fn get_state_size(&self) -> usize

Returns the maximum size in bytes of the state (rng, logits, embedding and kv_cache) - will often be smaller after compacting tokens

pub unsafe fn copy_state_data(&self, dest: *mut u8) -> usize

Copies the state to the specified destination address.

Returns the number of bytes copied

Safety

Destination needs to have allocated enough memory.

pub unsafe fn set_state_data(&mut self, src: &[u8]) -> usize

Set the state reading from the specified address Returns the number of bytes read

Safety

help wanted: not entirely sure what the safety requirements are here.

impl<'model> LlamaContext<'model>

pub fn n_batch(&self) -> u32

Gets the max number of logical tokens that can be submitted to decode. Must be greater than or equal to n_ubatch.

pub fn n_ubatch(&self) -> u32

Gets the max number of physical tokens (hardware level) to decode in batch. Must be less than or equal to n_batch.

pub fn n_ctx(&self) -> u32

Gets the size of the context.

pub fn decode(&mut self, batch: &mut LlamaBatch) -> Result<(), DecodeError>

Decodes the batch.

Errors

• DecodeError if the decoding failed.

Panics

• the returned std::ffi::c_int from llama-cpp does not fit into a i32 (this should never happen on most systems)

pub fn encode(&mut self, batch: &mut LlamaBatch) -> Result<(), EncodeError>

Encodes the batch.

Errors

• EncodeError if the decoding failed.

Panics

• the returned std::ffi::c_int from llama-cpp does not fit into a i32 (this should never happen on most systems)

pub fn pooling_type(&self) -> LlamaPoolingType

Return Pooling type for Llama’s Context

pub fn embeddings_seq_ith(&self, i: i32) -> Result<&[f32], EmbeddingsError>

Get the embeddings for the ith sequence in the current context.

Returns

A slice containing the embeddings for the last decoded batch. The size corresponds to the n_embd parameter of the context’s model.

Errors

• When the current context was constructed without enabling embeddings.
• If the current model had a pooling type of llama_cpp_sys_4::LLAMA_POOLING_TYPE_NONE
• If the given sequence index exceeds the max sequence id.

Panics

• n_embd does not fit into a usize

pub fn embeddings_ith(&self, i: i32) -> Result<&[f32], EmbeddingsError>

Get the embeddings for the ith token in the current context.

Returns

A slice containing the embeddings for the last decoded batch of the given token. The size corresponds to the n_embd parameter of the context’s model.

Errors

• When the current context was constructed without enabling embeddings.
• When the given token didn’t have logits enabled when it was passed.
• If the given token index exceeds the max token id.

Panics

• n_embd does not fit into a usize

pub fn candidates(&self) -> impl Iterator<Item = LlamaTokenData> + '_

Get the logits for the last token in the context.

Returns

An iterator over unsorted LlamaTokenData containing the logits for the last token in the context.

Panics

• underlying logits data is null

pub fn token_data_array(&self) -> LlamaTokenDataArray

Get the token data array for the last token in the context.

This is a convience method that implements:

LlamaTokenDataArray::from_iter(ctx.candidates(), false)

Panics

• underlying logits data is null

pub fn get_logits(&self) -> &[f32]

Token logits obtained from the last call to decode(). The logits for which batch.logits[i] != 0 are stored contiguously in the order they have appeared in the batch. Rows: number of tokens for which batch.logits[i] != 0 Cols: n_vocab

Returns

A slice containing the logits for the last decoded token. The size corresponds to the n_vocab parameter of the context’s model.

Panics

• n_vocab does not fit into a usize
• token data returned is null

pub fn candidates_ith( &self, i: i32, ) -> impl Iterator<Item = LlamaTokenData> + '_

Get the logits for the ith token in the context.

Panics

• logit i is not initialized.

pub fn get_logits_ith(&self, i: i32) -> &[f32]

Get the logits for the ith token in the context.

Panics

• i is greater than n_ctx
• n_vocab does not fit into a usize
• logit i is not initialized.

pub fn n_ctx_seq(&self) -> u32

Get the number of context tokens per sequence.

pub fn n_seq_max(&self) -> u32

Get the maximum number of sequences.

pub fn n_threads(&self) -> i32

Get the number of threads used for generation.

pub fn n_threads_batch(&self) -> i32

Get the number of threads used for batch processing.

pub fn set_n_threads(&mut self, n_threads: i32, n_threads_batch: i32)

Set the number of threads used for generation and batch processing.

pub fn set_causal_attn(&mut self, causal_attn: bool)

Set whether to use causal attention.

If set to false, the model will use non-causal attention, which is needed for embedding models.

pub fn set_embeddings(&mut self, embeddings: bool)

Set whether to compute embeddings.

This allows toggling embedding mode at runtime (as opposed to only at context creation time).

pub fn set_warmup(&mut self, warmup: bool)

Mark the next computation as a warmup run.

Warmup runs are useful for GPU backends to compile kernels before actual inference begins.

pub fn synchronize(&mut self)

Wait for all pending async computations to finish.

pub fn get_embeddings(&self) -> Result<&[f32], EmbeddingsError>

Get all embeddings for the current context.

Returns a slice of all embeddings from the last decoded batch. For pooled embeddings use embeddings_seq_ith instead.

Errors

• When the current context was constructed without enabling embeddings.
• If the embeddings pointer is null.

Panics

• n_embd does not fit into a usize

pub fn reset_timings(&mut self)

Reset the timings for the context.

pub fn timings(&mut self) -> PerfContextData

Returns the timings for the context.

pub fn perf_context_reset(&mut self)

Reset the performance counters for the context.

pub fn memory_can_shift(&self) -> bool

Check if the KV cache memory supports shifting.

pub fn memory_seq_pos_min(&self, seq_id: i32) -> i32

Get the minimum position in a sequence’s KV cache.

pub fn memory_breakdown_print(&self)

Print a breakdown of the memory usage.

pub fn state_get_size(&mut self) -> usize

Get the size of the full context state in bytes.

This is the size needed for state_get_data and state_set_data.

pub fn state_get_data(&mut self, dst: &mut [u8]) -> usize

Copy the full context state into a byte buffer.

The buffer must be at least state_get_size bytes.

Returns the number of bytes written.

pub fn state_set_data(&mut self, src: &[u8]) -> usize

Restore the full context state from a byte buffer.

Returns the number of bytes read.

pub fn state_save_file( &mut self, path: impl AsRef<Path>, tokens: &[LlamaToken], ) -> bool

Save the context state to a file along with the given tokens.

Returns true on success.

Panics

Panics if the path contains null bytes.

pub fn state_load_file( &mut self, path: impl AsRef<Path>, tokens_out: &mut Vec<LlamaToken>, n_token_capacity: usize, ) -> bool

Load a context state from a file.

Returns true on success and fills tokens_out with the saved tokens.

Panics

Panics if the path contains null bytes.

pub fn state_seq_get_size(&mut self, seq_id: i32) -> usize

Get the size of a single sequence’s state in bytes.

pub fn state_seq_get_data(&mut self, dst: &mut [u8], seq_id: i32) -> usize

Copy a single sequence’s state into a byte buffer.

Returns the number of bytes written.

pub fn state_seq_set_data(&mut self, src: &[u8], dest_seq_id: i32) -> usize

Restore a single sequence’s state from a byte buffer.

Returns the number of bytes read.

pub fn state_seq_save_file( &mut self, path: impl AsRef<Path>, seq_id: i32, tokens: &[LlamaToken], ) -> usize

Save a single sequence’s state to a file.

Returns the number of bytes written (0 on failure).

Panics

Panics if the path contains null bytes.

pub fn state_seq_load_file( &mut self, path: impl AsRef<Path>, dest_seq_id: i32, tokens_out: &mut Vec<LlamaToken>, n_token_capacity: usize, ) -> usize

Load a single sequence’s state from a file.

Returns the number of bytes read (0 on failure).

Panics

Panics if the path contains null bytes.

pub fn set_adapter_cvec( &mut self, data: &[f32], n_embd: i32, il_start: i32, il_end: i32, ) -> Result<(), i32>

Set a control vector on the context.

Parameters

• data: The control vector data (embedding values). Pass an empty slice to clear.
• n_embd: The embedding dimension.
• il_start: The starting layer index (inclusive).
• il_end: The ending layer index (exclusive).

Errors

Returns Err with the error code if the operation fails.

pub fn get_sampled_token_ith(&self, i: i32) -> LlamaToken

Get sampled token debug info for the ith position.

Returns the sampled token at position i from the last decode call.

pub fn get_sampled_candidates_ith(&self, i: i32) -> &[LlamaToken]

Get sampled candidate tokens for the ith position.

Returns a slice of candidate tokens from the last decode call.

pub fn get_sampled_logits_count_ith(&self, i: i32) -> u32

Get the number of sampled logits for the ith position.

pub fn get_sampled_logits_ith(&self, i: i32) -> &[f32]

Get sampled logits for the ith position.

Returns a slice of logit values from the last decode call.

pub fn get_sampled_probs_count_ith(&self, i: i32) -> u32

Get the number of sampled probabilities for the ith position.

pub fn get_sampled_probs_ith(&self, i: i32) -> &[f32]

Get sampled probabilities for the ith position.

Returns a slice of probability values from the last decode call.

pub fn state_seq_get_size_ext(&mut self, seq_id: i32, flags: u32) -> usize

Get the size of a single sequence’s state with flags.

pub fn state_seq_get_data_ext( &mut self, dst: &mut [u8], seq_id: i32, flags: u32, ) -> usize

Copy a single sequence’s state into a byte buffer with flags.

Returns the number of bytes written.

pub fn state_seq_set_data_ext( &mut self, src: &[u8], dest_seq_id: i32, flags: u32, ) -> usize

Restore a single sequence’s state from a byte buffer with flags.

Returns the number of bytes read.

pub unsafe fn set_abort_callback( &mut self, callback: ggml_abort_callback, data: *mut c_void, )

Set an abort callback for the context.

The callback is called periodically during computation. If it returns true, the computation is aborted.

Safety

The callback data must remain valid for the lifetime of the context or until the callback is replaced.

pub unsafe fn attach_threadpool( &mut self, threadpool: ggml_threadpool_t, threadpool_batch: ggml_threadpool_t, )

Attach a thread pool to the context.

Safety

The thread pools must remain valid for the lifetime of the context or until they are detached.

pub fn detach_threadpool(&mut self)

Detach the thread pool from the context.

pub fn set_sampler(&mut self, seq_id: i32, sampler: &mut LlamaSampler) -> bool

Set a sampler for a specific sequence.

Returns true on success.

pub fn get_model_ptr(&self) -> *const llama_model

Get the raw model pointer from this context.

This is mainly useful for FFI interop. In normal usage, access the model via the model field instead.

pub fn lora_adapter_set( &self, adapter: &mut LlamaLoraAdapter, scale: f32, ) -> Result<(), LlamaLoraAdapterSetError>

Sets a lora adapter.

Errors

See LlamaLoraAdapterSetError for more information.

pub fn lora_adapter_remove( &self, _adapter: &mut LlamaLoraAdapter, ) -> Result<(), LlamaLoraAdapterRemoveError>

Remove all lora adapters from the context.

Note: as of llama.cpp b8249 the per-adapter remove API was replaced by llama_set_adapters_lora which operates on the full adapter list at once. Calling this function clears all adapters currently set on the context.

Errors

See LlamaLoraAdapterRemoveError for more information.

impl LlamaContext<'_>

pub fn as_ptr(&self) -> *mut llama_context

Expose the raw llama_context pointer for use with mtmd helpers.

Safety

The pointer is valid for the lifetime of this LlamaContext and must not be freed by the caller.

Trait Implementations

impl Debug for LlamaContext<'_>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for LlamaContext<'_>

fn drop(&mut self)

Executes the destructor for this type.