Struct LlamaContext

Source

pub struct LlamaContext<'a> { /* private fields */ }

Expand description

An inference context bound to a LlamaModel.

The lifetime 'a ties the context to its model (the model must outlive it).

Implementations§

Source §

impl LlamaContext<'_>

Source

pub fn embeddings_ith(&self, i: i32) -> Result<&[f32], EmbeddingsError>

Get the embeddings for the ith token in the current context.

§Returns

A slice with the embeddings for the last decoded batch of the given token. The size corresponds to the n_embd of the context’s model.

§Errors

EmbeddingsError::NotEnabled if the context was built without with_embeddings(true).
EmbeddingsError::LogitsNotEnabled if that token was not decoded with logits enabled (ik returns a null pointer), or i is out of range.

Source

pub fn embeddings_seq_ith(&self, seq: i32) -> Result<&[f32], EmbeddingsError>

Get the pooled embeddings for the seqth sequence in the current context.

§Returns

A slice with the pooled embeddings for the last decoded batch. The size corresponds to the n_embd of the context’s model.

§Errors

EmbeddingsError::NotEnabled if the context was built without with_embeddings(true).
EmbeddingsError::NonePoolType if the model uses LLAMA_POOLING_TYPE_NONE (ik returns a null pointer), or seq exceeds the max sequence id.

Source §

impl LlamaContext<'_>

Source

pub fn copy_cache(&mut self, src: i32, dest: i32, size: i32)

Copy the cache from one sequence to another.

§Parameters

src - The sequence id to copy the cache from.
dest - The sequence id to copy the cache to.
size - The size of the cache to copy.

Source

pub fn copy_kv_cache_seq( &mut self, src: i32, dest: i32, p0: Option<u32>, p1: Option<u32>, ) -> Result<(), KvCacheConversionError>

Copy the cache from one sequence to another.

§Returns

A Result indicating whether the operation was successful.

§Parameters

src - The sequence id to copy the cache from.
dest - The sequence id to copy the cache to.
p0 - The start position of the cache to clear. If None, the entire cache is copied up to p1.
p1 - The end position of the cache to clear. If None, the entire cache is copied starting from p0.

§Errors

If either position exceeds i32::MAX.

Source

pub fn clear_kv_cache_seq( &mut self, src: Option<u32>, p0: Option<u32>, p1: Option<u32>, ) -> Result<bool, KvCacheConversionError>

Clear the kv cache for the given sequence within the specified range [p0, p1) Returns false only when partial sequence removals fail. Full sequence removals always succeed.

§Returns

A Result indicating whether the operation was successful. If the sequence id or either position exceeds the maximum i32 value, no removal is attempted and an Err is returned.

§Parameters

src - The sequence id to clear the cache for. If None, matches all sequences
p0 - The start position of the cache to clear. If None, the entire cache is cleared up to p1.
p1 - The end position of the cache to clear. If None, the entire cache is cleared from p0.

§Errors

If the sequence id or either position exceeds i32::MAX.

Source

pub fn clear_kv_cache(&mut self)

Clear the KV cache

Source

pub fn llama_kv_cache_seq_keep(&mut self, seq_id: i32)

Removes all tokens that do not belong to the specified sequence

§Parameters

seq_id - The sequence id to keep

Source

pub fn kv_cache_seq_cp( &mut self, seq_id_src: i32, seq_id_dst: i32, p0: Option<u32>, p1: Option<u32>, ) -> Result<(), KvCacheConversionError>

Copy all tokens that belong to the specified sequence to another sequence If the KV cache is RoPEd, the KV data is updated accordingly:

lazily on next LlamaContext::decode
explicitly with Self::kv_cache_update

§Returns

A Result indicating whether the operation was successful.

§Parameters

seq_id_src - The sequence id to copy from. If negative, matches any sequence.
seq_id_dst - The sequence id to copy to
p0 - The start position of the cache to copy from. If None, the entire cache is updated up to p1.
p1 - The end position of the cache to copy from. If None, the entire cache is updated starting from p0.

§Errors

If either position exceeds i32::MAX.

Source

pub fn kv_cache_seq_add( &mut self, seq_id: i32, p0: Option<u32>, p1: Option<u32>, delta: i32, ) -> Result<(), KvCacheConversionError>

Adds relative position “delta” to all tokens that belong to the specified sequence and have positions in [p0, p1) If the KV cache is RoPEd, the KV data is updated accordingly:

lazily on next LlamaContext::decode
explicitly with Self::kv_cache_update

§Returns

A Result indicating whether the operation was successful.

§Parameters

seq_id - The sequence id to update
p0 - The start position of the cache to update. If None, the entire cache is updated up to p1.
p1 - The end position of the cache to update. If None, the entire cache is updated starting from p0.
delta - The relative position to add to the tokens

§Errors

If either position exceeds i32::MAX.

Source

pub fn kv_cache_seq_rm( &mut self, seq_id: i32, p0: Option<u32>, p1: Option<u32>, ) -> Result<(), KvCacheConversionError>

Removes all tokens that belong to the specified sequence and have positions in [p0, p1) If the KV cache is RoPEd, the KV data is updated accordingly:

lazily on next LlamaContext::decode
explicitly with Self::kv_cache_update

§Returns

A Result indicating whether the operation was successful.

§Parameters

seq_id - The sequence id to update
p0 - The start position of the cache to update. If None, the entire cache is updated up to p1.
p1 - The end position of the cache to update. If None, the entire cache is updated starting from p0.

§Errors

If either position exceeds i32::MAX.

Source

pub fn kv_cache_seq_div( &mut self, seq_id: i32, p0: Option<u32>, p1: Option<u32>, d: NonZeroU8, ) -> Result<(), KvCacheConversionError>

Integer division of the positions by factor of d > 1 If the KV cache is RoPEd, the KV data is updated accordingly:

lazily on next LlamaContext::decode
explicitly with Self::kv_cache_update

§Returns

A Result indicating whether the operation was successful.

§Parameters

seq_id - The sequence id to update
p0 - The start position of the cache to update. If None, the entire cache is updated up to p1.
p1 - The end position of the cache to update. If None, the entire cache is updated starting from p0.
d - The factor to divide the positions by

§Errors

If either position exceeds i32::MAX.

Source

pub fn kv_cache_seq_pos_min(&self, seq_id: i32) -> i32

Returns the smallest position present in the KV cache for the specified sequence

§Parameters

seq_id - The sequence id to get the min position for

Source

pub fn kv_cache_seq_pos_max(&self, seq_id: i32) -> i32

Returns the largest position present in the KV cache for the specified sequence

§Parameters

seq_id - The sequence id to get the max position for

Source

pub fn kv_cache_defrag(&mut self)

Defragment the KV cache This will be applied:

lazily on next LlamaContext::decode
explicitly with Self::kv_cache_update

Source

pub fn kv_cache_update(&mut self) -> i32

Apply the KV cache updates (such as K-shifts, defragmentation, etc.)

§Returns

The status returned by the underlying llama_kv_cache_update. Positive return values are not a fatal error, but rather a warning:

0 - success
1 - context overflow in a model where k-shift is not supported

Source §

impl LlamaContext<'_>

Source

pub fn save_session_file( &self, path_session: impl AsRef<Path>, tokens: &[LlamaToken], ) -> Result<(), SaveSessionError>

👎Deprecated since 0.1.136:

Use state_save_file instead

Save the current session to a file.

§Parameters

path_session - The file to save to.
tokens - The tokens to associate the session with. This should be a prefix of a sequence of tokens that the context has processed, so that the relevant KV caches are already filled.

§Errors

Fails if the path is not a valid utf8, is not a valid c string, or llama.cpp fails to save the session file.

Source

pub fn load_session_file( &mut self, path_session: impl AsRef<Path>, max_tokens: usize, ) -> Result<Vec<LlamaToken>, LoadSessionError>

👎Deprecated since 0.1.136:

Use state_load_file instead

Load a session file into the current context.

You still need to pass the returned tokens to the context for inference to work. What this function buys you is that the KV caches are already filled with the relevant data.

§Parameters

path_session - The file to load from. It must be a session file from a compatible context, otherwise the function will error.
max_tokens - The maximum token length of the loaded session. If the session was saved with a longer length, the function will error.

§Errors

Fails if the path is not a valid utf8, is not a valid c string, or llama.cpp fails to load the session file. (e.g. the file does not exist, is not a session file, etc.)

Source

pub fn state_save_file( &self, path_session: impl AsRef<Path>, tokens: &[LlamaToken], ) -> Result<(), SaveSessionError>

Save the full state to a file.

This is the non-deprecated replacement for save_session_file.

§Parameters

path_session - The file to save to.
tokens - The tokens to associate the state with. This should be a prefix of a sequence of tokens that the context has processed, so that the relevant KV caches are already filled.

§Errors

Fails if the path is not a valid utf8, is not a valid c string, or llama.cpp fails to save the state file.

Source

pub fn state_load_file( &mut self, path_session: impl AsRef<Path>, max_tokens: usize, ) -> Result<Vec<LlamaToken>, LoadSessionError>

Load a state file into the current context.

This is the non-deprecated replacement for load_session_file.

You still need to pass the returned tokens to the context for inference to work. What this function buys you is that the KV caches are already filled with the relevant data.

§Parameters

path_session - The file to load from. It must be a state file from a compatible context, otherwise the function will error.
max_tokens - The maximum token length of the loaded state. If the state was saved with a longer length, the function will error.

§Errors

Fails if the path is not a valid utf8, is not a valid c string, or llama.cpp fails to load the state file.

Source

pub fn state_seq_save_file( &self, filepath: impl AsRef<Path>, seq_id: i32, tokens: &[LlamaToken], ) -> Result<usize, SaveSeqStateError>

Save state for a single sequence to a file.

This enables saving state for individual sequences, which is useful for multi-sequence inference scenarios.

§Parameters

filepath - The file to save to.
seq_id - The sequence ID whose state to save.
tokens - The tokens to associate with the saved state.

§Errors

Fails if the path is not a valid utf8, is not a valid c string, or llama.cpp fails to save the sequence state file.

§Returns

The number of bytes written on success.

Source

pub fn state_seq_load_file( &mut self, filepath: impl AsRef<Path>, dest_seq_id: i32, max_tokens: usize, ) -> Result<(Vec<LlamaToken>, usize), LoadSeqStateError>

Load state for a single sequence from a file.

This enables loading state for individual sequences, which is useful for multi-sequence inference scenarios.

§Parameters

filepath - The file to load from.
dest_seq_id - The destination sequence ID to load the state into.
max_tokens - The maximum number of tokens to read.

§Errors

Fails if the path is not a valid utf8, is not a valid c string, or llama.cpp fails to load the sequence state file.

§Returns

A tuple of (tokens, bytes_read) on success.

Source

pub fn get_state_size(&self) -> usize

Returns the maximum size in bytes of the state (rng, logits, embedding and kv_cache) - will often be smaller after compacting tokens

Source

pub unsafe fn copy_state_data(&self, dest: *mut u8) -> usize

Copies the state to the specified destination address.

Returns the number of bytes copied

§Safety

Destination needs to have allocated enough memory.

Source

pub unsafe fn set_state_data(&mut self, src: &[u8]) -> usize

Set the state reading from the specified address Returns the number of bytes read

§Safety

help wanted: not entirely sure what the safety requirements are here.

Source

pub fn state_seq_get_size_ext( &self, seq_id: i32, flags: LlamaStateSeqFlags, ) -> usize

Get the size of the state for a single sequence with optional flags.

This is the extended version that supports flags for partial state operations.

§Parameters

seq_id - The sequence ID to get the state size for.
flags - Optional flags (e.g., LlamaStateSeqFlags::PARTIAL_ONLY).

§Returns

The size in bytes needed to store the sequence state.

Source

pub unsafe fn state_seq_get_data_ext( &self, dest: *mut u8, seq_id: i32, flags: LlamaStateSeqFlags, ) -> usize

Copy the state of a single sequence into the specified buffer with optional flags.

This is the extended version that supports flags for partial state operations.

§Parameters

dest - Destination buffer to copy state into.
seq_id - The sequence ID to get the state for.
flags - Optional flags (e.g., LlamaStateSeqFlags::PARTIAL_ONLY).

§Safety

Destination needs to have allocated enough memory.

§Returns

The number of bytes copied.

Source

pub unsafe fn state_seq_set_data_ext( &mut self, src: &[u8], dest_seq_id: i32, flags: LlamaStateSeqFlags, ) -> bool

Set the state for a single sequence from the specified buffer with optional flags.

This is the extended version that supports flags for partial state operations. Useful for restoring only the recurrent/partial state without affecting the KV cache.

§Parameters

src - Source buffer containing the state data.
dest_seq_id - The destination sequence ID to load the state into.
flags - Optional flags (e.g., LlamaStateSeqFlags::PARTIAL_ONLY).

§Safety

The source buffer must contain valid state data.

§Returns

true on success (the whole src buffer was consumed). ik returns SIZE_MAX when state IO is unsupported and 0 on other failures; both are reported as false (only n == src.len() counts as success).

Source

pub fn state_seq_get( &self, seq_id: i32, flags: LlamaStateSeqFlags, ) -> Result<SeqState, StateSeqError>

Serialize sequence seq_id’s state into an opaque SeqState.

Enables save/restore of context state at arbitrary points in a sequence. This is particularly useful on architectures where clear_kv_cache_seq cannot roll back partial state (Mamba, RWKV, Gated Delta Networks, and other recurrent / hybrid-recurrent models): pair with LlamaStateSeqFlags::PARTIAL_ONLY to save just the running recurrent and SWA state, then restore it via Self::state_seq_set to effectively “rewind” the sequence.

The returned SeqState is opaque — its bytes cannot be inspected or forged from safe code, so Self::state_seq_set only ever sees data produced by this method.

Wraps llama_state_seq_get_data.

§Errors

Returns StateSeqError::SizeMismatch if llama.cpp writes a different number of bytes than the reported state size.

Source

pub fn state_seq_set( &mut self, state: &SeqState, seq_id: i32, ) -> Result<(), StateSeqError>

Restore sequence state previously captured by Self::state_seq_get into seq_id.

Cross-sequence restore (seq_id different from the sequence the state was captured from) is supported — llama.cpp treats the destination sequence id independently of the source.

Wraps llama_state_seq_set_data.

§Errors

Returns StateSeqError::SizeMismatch if llama.cpp reads a different number of bytes than the state buffer contains — this covers shape mismatches (different n_ctx, n_layer, quantization, etc.) that llama.cpp’s own deserializer detects and aborts on.

Source §

impl<'a> LlamaContext<'a>

Source

pub fn new( model: &'a LlamaModel, params: &LlamaContextParams, ) -> Result<Self, LlamaError>

Create a context for model.

Source

pub fn decode(&mut self, batch: &mut LlamaBatch) -> Result<(), LlamaError>

Run a decode over batch. Errors on a non-zero status from llama_decode.

Source

pub fn get_logits_ith(&self, i: i32) -> Result<&[f32], LlamaError>

Logits for batch index i of the last decode (length = n_vocab).

i is the batch position (0..n_tokens), not a logits-rank; that entry must have been added with logits = true (else ik returns null → NoLogits).

Source

pub fn candidates_ith( &self, i: i32, ) -> impl Iterator<Item = LlamaTokenData> + '_

Candidate token data for batch index i (id + logit, p = 0).

Returns an empty iterator if logits are unavailable at i (e.g. that position was not decoded with logits = true).

Source

pub fn token_data_array_ith(&self, i: i32) -> LlamaTokenDataArray

Build a LlamaTokenDataArray from the logits at batch index i (matches llama-cpp-2). Empty if logits are unavailable at i.

Source

pub fn n_ctx(&self) -> u32

The context size (tokens).

Source

pub fn set_mtp_op_type(&mut self, op: MtpOpType)

Set the MTP operation mode for subsequent decodes.

Only meaningful for a context created with .with_mtp(true) on a model loaded with .with_mtp(true) (a NextN model). Drives the low-level MTP state machine (llama_set_mtp_op_type).

Source