Struct LlamaContext

Source

pub struct LlamaContext<'a> {
    pub model: &'a LlamaModel,
    /* private fields */
}

Expand description

A safe wrapper around the llama_context C++ context.

This struct provides a safe interface to interact with the llama_context used by the LlamaModel. It encapsulates the raw C++ context pointer and provides additional fields for managing the model and context-specific settings like embeddings and logits.

The LlamaContext struct ensures that the C++ context is always valid by using the NonNull type for the context pointer, preventing it from being null. The struct also holds a reference to the model (LlamaModel) that the context is tied to, along with some internal state like whether embeddings are enabled and the initialized logits for the context.

§Fields

context: A non-null pointer to the raw C++ llama_context. This is the main context used for interacting with the model.
model: A reference to the LlamaModel associated with this context. This model provides the data and parameters that the context interacts with.
initialized_logits: A vector used to store the initialized logits. These are used in the model’s processing and are kept separate from the context data.
embeddings_enabled: A boolean flag indicating whether embeddings are enabled in the context. This is useful for controlling whether embedding data is generated during the interaction with the model.

Fields§

§model: &'a LlamaModel

a reference to the contexts model.

Implementations§

Source §

impl LlamaContext<'_>

Source

pub fn copy_cache(&mut self, src: i32, dest: i32, size: i32)

Copy the cache from one sequence to another.

§Parameters

src - The sequence id to copy the cache from.
dest - The sequence id to copy the cache to.
size - The size of the cache to copy.

Source

pub fn copy_kv_cache_seq( &mut self, src: i32, dest: i32, p0: Option<u32>, p1: Option<u32>, ) -> Result<(), KvCacheConversionError>

Copy the cache from one sequence to another.

§Parameters

src - The sequence id to copy the cache from.
dest - The sequence id to copy the cache to.
p0 - The start position of the cache to clear. If None, the entire cache is copied up to p1.
p1 - The end position of the cache to clear. If None, the entire cache is copied starting from p0.

§Errors

Returns KvCacheConversionError if either position exceeds the maximum i32 value.

Source

pub fn clear_kv_cache_seq( &mut self, src: Option<u32>, p0: Option<u32>, p1: Option<u32>, ) -> Result<bool, KvCacheConversionError>

Clear the kv cache for the given sequence within the specified range [p0, p1).

Returns false only when partial sequence removals fail. Full sequence removals always succeed.

§Parameters

src - The sequence id to clear the cache for. If None, matches all sequences
p0 - The start position of the cache to clear. If None, the entire cache is cleared up to p1.
p1 - The end position of the cache to clear. If None, the entire cache is cleared from p0.

§Errors

Returns KvCacheConversionError if the sequence id or either position exceeds the maximum i32 value.

Source

pub fn clear_kv_cache(&mut self)

Clear the KV cache

Source

pub fn llama_kv_cache_seq_keep(&mut self, seq_id: i32)

Removes all tokens that do not belong to the specified sequence

§Parameters

seq_id - The sequence id to keep

Source

pub fn kv_cache_seq_add( &mut self, seq_id: i32, p0: Option<u32>, p1: Option<u32>, delta: i32, ) -> Result<(), KvCacheConversionError>

Adds relative position “delta” to all tokens that belong to the specified sequence and have positions in [p0, p1).

§Parameters

seq_id - The sequence id to update
p0 - The start position of the cache to update. If None, the entire cache is updated up to p1.
p1 - The end position of the cache to update. If None, the entire cache is updated starting from p0.
delta - The relative position to add to the tokens

§Errors

Returns KvCacheConversionError if either position exceeds the maximum i32 value.

Source

pub fn kv_cache_seq_div( &mut self, seq_id: i32, p0: Option<u32>, p1: Option<u32>, d: NonZeroU8, ) -> Result<(), KvCacheConversionError>

Integer division of the positions by factor of d > 1.

§Parameters

seq_id - The sequence id to update
p0 - The start position of the cache to update. If None, the entire cache is updated up to p1.
p1 - The end position of the cache to update. If None, the entire cache is updated starting from p0.
d - The factor to divide the positions by

§Errors

Returns KvCacheConversionError if either position exceeds the maximum i32 value.

Source

pub fn kv_cache_seq_pos_max(&self, seq_id: i32) -> i32

Returns the largest position present in the KV cache for the specified sequence

§Parameters

seq_id - The sequence id to get the max position for

Source §

impl LlamaContext<'_>

Source

pub fn save_session_file( &self, path_session: impl AsRef<Path>, tokens: &[LlamaToken], ) -> Result<(), SaveSessionError>

Save the current session to a file.

§Parameters

path_session - The file to save to.
tokens - The tokens to associate the session with. This should be a prefix of a sequence of tokens that the context has processed, so that the relevant KV caches are already filled.

§Errors

Fails if the path is not a valid utf8, is not a valid c string, or llama.cpp fails to save the session file.

Source

pub fn load_session_file( &mut self, path_session: impl AsRef<Path>, max_tokens: usize, ) -> Result<Vec<LlamaToken>, LoadSessionError>

Load a session file into the current context.

You still need to pass the returned tokens to the context for inference to work. What this function buys you is that the KV caches are already filled with the relevant data.

§Parameters

path_session - The file to load from. It must be a session file from a compatible context, otherwise the function will error.
max_tokens - The maximum token length of the loaded session. If the session was saved with a longer length, the function will error.

§Errors

Fails if the path is not a valid utf8, is not a valid c string, or llama.cpp fails to load the session file. (e.g. the file does not exist, is not a session file, etc.)

Source

pub fn get_state_size(&self) -> usize

Returns the maximum size in bytes of the state (rng, logits, embedding and kv_cache) - will often be smaller after compacting tokens

Source

pub unsafe fn copy_state_data(&self, dest: *mut u8) -> usize

Copies the state to the specified destination address.

Returns the number of bytes copied

§Safety

Destination needs to have allocated enough memory.

Source

pub unsafe fn set_state_data(&mut self, src: &[u8]) -> usize

Set the state reading from the specified address Returns the number of bytes read

§Safety

help wanted: not entirely sure what the safety requirements are here.

Source §

impl<'model> LlamaContext<'model>

Source

pub fn n_batch(&self) -> u32

Gets the max number of logical tokens that can be submitted to decode. Must be greater than or equal to n_ubatch.

Source

pub fn n_ubatch(&self) -> u32

Gets the max number of physical tokens (hardware level) to decode in batch. Must be less than or equal to n_batch.

Source

pub fn n_ctx(&self) -> u32

Gets the size of the context.

Source

pub fn decode(&mut self, batch: &mut LlamaBatch) -> Result<(), DecodeError>

Decodes the batch.

§Errors

DecodeError if the decoding failed.

§Panics

the returned std::ffi::c_int from llama-cpp does not fit into a i32 (this should never happen on most systems)

Source

pub fn encode(&mut self, batch: &mut LlamaBatch) -> Result<(), EncodeError>

Encodes the batch.

§Errors

EncodeError if the decoding failed.

§Panics

the returned std::ffi::c_int from llama-cpp does not fit into a i32 (this should never happen on most systems)

Source

pub fn pooling_type(&self) -> LlamaPoolingType

Return Pooling type for Llama’s Context

Source

pub fn embeddings_seq_ith(&self, i: i32) -> Result<&[f32], EmbeddingsError>

Get the embeddings for the ith sequence in the current context.

§Returns

A slice containing the embeddings for the last decoded batch. The size corresponds to the n_embd parameter of the context’s model.

§Errors

When the current context was constructed without enabling embeddings.
If the current model had a pooling type of llama_cpp_sys_4::LLAMA_POOLING_TYPE_NONE
If the given sequence index exceeds the max sequence id.

§Panics

n_embd does not fit into a usize

Source

pub fn embeddings_ith(&self, i: i32) -> Result<&[f32], EmbeddingsError>

Get the embeddings for the ith token in the current context.

§Returns

A slice containing the embeddings for the last decoded batch of the given token. The size corresponds to the n_embd parameter of the context’s model.

§Errors

When the current context was constructed without enabling embeddings.
When the given token didn’t have logits enabled when it was passed.
If the given token index exceeds the max token id.

§Panics

n_embd does not fit into a usize

Source

pub fn candidates(&self) -> impl Iterator<Item = LlamaTokenData> + '_

Get the logits for the last token in the context.

§Returns

An iterator over unsorted LlamaTokenData containing the logits for the last token in the context.

§Panics

underlying logits data is null

Source

pub fn token_data_array(&self) -> LlamaTokenDataArray

Get the token data array for the last token in the context.

This is a convience method that implements:

LlamaTokenDataArray::from_iter(ctx.candidates(), false)

§Panics

underlying logits data is null

Source

pub fn get_logits(&self) -> &[f32]

Token logits obtained from the last call to decode(). The logits for which batch.logits[i] != 0 are stored contiguously in the order they have appeared in the batch. Rows: number of tokens for which batch.logits[i] != 0 Cols: n_vocab

§Returns

A slice containing the logits for the last decoded token. The size corresponds to the n_vocab parameter of the context’s model.

§Panics

n_vocab does not fit into a usize
token data returned is null

Source

pub fn candidates_ith( &self, i: i32, ) -> impl Iterator<Item = LlamaTokenData> + '_

Get the logits for the ith token in the context.

§Panics

logit i is not initialized.

Source

pub fn get_logits_ith(&self, i: i32) -> &[f32]

Get the logits for the ith token in the context.

§Panics

i is greater than n_ctx
n_vocab does not fit into a usize
logit i is not initialized.

Source

pub fn reset_timings(&mut self)

Reset the timings for the context.

Source

pub fn timings(&mut self) -> PerfContextData

Returns the timings for the context.

Source

pub fn lora_adapter_set( &self, adapter: &mut LlamaLoraAdapter, scale: f32, ) -> Result<(), LlamaLoraAdapterSetError>

Sets a lora adapter.

§Errors

See LlamaLoraAdapterSetError for more information.

Source

pub fn lora_adapter_remove( &self, _adapter: &mut LlamaLoraAdapter, ) -> Result<(), LlamaLoraAdapterRemoveError>

Remove all lora adapters from the context.

Note: as of llama.cpp b8249 the per-adapter remove API was replaced by llama_set_adapters_lora which operates on the full adapter list at once. Calling this function clears all adapters currently set on the context.