Struct LlamaBatchExt 

pub struct LlamaBatchExt { /* private fields */ }

A safe wrapper around llama_batch_ext.

Build it with Self::new, add entries, configure each entry through the BatchEntry handle its add_* returned, then hand it to LlamaContext::process.

Positions are not implied

Unlike LlamaBatch::add, adding an entry does not take its position — upstream leaves it undefined and expects an explicit Self::set_pos. The underlying entry starts at position 0, so a forgotten call is not a type error. Set the position for every entry.

Context binding

llama_batch_ext_init reads the context’s batch size, sequence count and model geometry, and keeps a pointer to its memory module for position validation. The batch is therefore only usable with the context that built it: LlamaContext::process checks that and refuses anything else, so a batch outliving its context cannot be processed by a later one.

Implementations

impl LlamaBatchExt

pub fn new(ctx: &LlamaContext<'_>) -> Result<Self, BatchExtError>

Build a batch for ctx.

Capacity is the context’s n_batch; the sequence ids it will accept are 0..n_seq_max. Both are read from the context here, so they cannot drift from it later.

Errors

Returns BatchExtError::NullReturn if llama.cpp could not allocate.

pub fn clear(&mut self)

Drop every entry, keeping the allocation for the next round.

BatchEntry handles from before the call are no longer valid; the set_* methods reject them rather than writing to a recycled entry.

pub fn len(&self) -> usize

Number of entries in the batch.

pub fn is_empty(&self) -> bool

Whether the batch has no entries.

pub fn has_incomplete_entry(&self) -> bool

Whether any entry carries no input — neither a token id nor an input embedding — which makes the batch unprocessable.

Two things leave one behind: Self::add_empty that was never filled in with Self::set_token_embedding, and an Self::add_embedding whose block llama.cpp rejected after appending the entry. There is no upstream call to remove an entry, so the way out is Self::clear. LlamaContext::process checks this rather than submitting a batch with a hole in it.

pub fn capacity(&self) -> u32

Maximum entries this batch can hold — the context’s n_batch.

pub fn n_pos_per_embd(&self) -> usize

Positions an embedding entry in this batch requires: MROPE_SECTIONS on an M-RoPE model, 1 otherwise. A token entry always takes one.

pub fn outputs(&self) -> &[i32]

Entries with output enabled, in the order they were enabled.

pub fn add_token( &mut self, LlamaToken: LlamaToken, seq_id: llama_seq_id, ) -> Result<BatchEntry, BatchExtError>

Add a token entry for sequence seq_id.

Set its position with Self::set_pos; upstream leaves it undefined.

Errors

BatchExtError::Full when the batch is at capacity, BatchExtError::InvalidSeqId for an out-of-range sequence, or BatchExtError::InvalidToken for a token id outside the vocabulary.

pub fn add_embedding( &mut self, embd: EmbeddingRows<'_>, seq_id: llama_seq_id, ) -> Result<BatchEntry, BatchExtError>

Add an embedding-only entry for sequence seq_id.

The row width must match the model’s decoder input width (n_embd_inp) or its encoder input width, and every embedding in one batch must have the same width.

Positions come from Self::set_positions: an embedding entry takes Self::n_pos_per_embd of them, not one.

Errors

BatchExtError::Full, BatchExtError::InvalidSeqId, BatchExtError::EmbeddingWidthMismatch if it disagrees with the batch’s established width, or BatchExtError::EmbeddingRejected if llama.cpp would not take the block.

EmbeddingRejected here leaves an entry with no input in the batch — upstream appends before it validates, and offers no way to remove one. See Self::has_incomplete_entry.

pub fn add_empty( &mut self, seq_id: llama_seq_id, ) -> Result<BatchEntry, BatchExtError>

Add an entry with neither a token id nor an embedding, to be filled in with Self::set_token_embedding.

Treated as an embedding entry for positioning: it takes Self::n_pos_per_embd positions.

Until it is filled in, the entry counts as incomplete and LlamaContext::process will refuse the batch — see Self::has_incomplete_entry.

Errors

BatchExtError::Full or BatchExtError::InvalidSeqId.

pub fn add_seq( &mut self, entry: BatchEntry, seq_id: llama_seq_id, ) -> Result<(), BatchExtError>

Also feed this entry to seq_id. The entry keeps the sequence it was added with.

Errors

BatchExtError::NoSuchEntry for a stale handle, or BatchExtError::InvalidSeqId for an out-of-range sequence.

pub fn set_pos( &mut self, entry: BatchEntry, pos: llama_pos, ) -> Result<(), BatchExtError>

Set the position of a token entry.

Errors

BatchExtError::NoSuchEntry for a stale handle, or BatchExtError::PositionArity if the entry is an embedding entry on an M-RoPE model and needs Self::n_pos_per_embd positions — use Self::set_positions for those.

pub fn set_positions( &mut self, entry: BatchEntry, pos: &[llama_pos], ) -> Result<(), BatchExtError>

Set all positions of an entry.

pos must hold exactly the number of positions the entry takes: one for a token entry, Self::n_pos_per_embd for an embedding entry. On an M-RoPE model that is [t, y, x, z], where t is also the KV-cache position.

Errors

BatchExtError::NoSuchEntry for a stale handle, or BatchExtError::PositionArity for the wrong number of positions — llama.cpp reads the arity it expects straight off the pointer, so a short slice would read out of bounds.

pub fn set_token_embedding( &mut self, entry: BatchEntry, embd: EmbeddingRows<'_>, ) -> Result<(), BatchExtError>

Attach an input embedding to an entry that already has a token id, so the entry carries both.

Errors

BatchExtError::NoSuchEntry for a stale handle, BatchExtError::EmbeddingAlreadySet if the entry has one, BatchExtError::EmbeddingWidthMismatch if it disagrees with the batch’s established width, or BatchExtError::EmbeddingRejected if llama.cpp would not take the block.

pub fn set_state_embedding( &mut self, entry: BatchEntry, state: EmbeddingRows<'_>, ) -> Result<(), BatchExtError>

Attach a state embedding to an entry — hidden state carried over from an earlier stage, such as the target model’s layers for MTP or a vision encoder’s deepstack output.

Errors

BatchExtError::NoSuchEntry for a stale handle, BatchExtError::StateAlreadySet if the entry has one, or BatchExtError::EmbeddingRejected when the width is wrong or this context takes no state embedding at all — only an MTP context does, so check LlamaContext::context_type.

pub fn set_output( &mut self, entry: BatchEntry, value: bool, ) -> Result<(), BatchExtError>

Enable or disable output for an entry, making its logits and embeddings readable after processing.

Upstream exposes this twice — llama_batch_ext_set_output_logits and llama_batch_ext_set_output_embd — and the two call the same setter, so there is one method here. Should they ever diverge, this one keeps the logits meaning.

Errors

BatchExtError::NoSuchEntry for a stale handle.

pub fn set_decision_order( &mut self, entry: BatchEntry, order: LlamaDecisionOrder, ) -> Result<(), BatchExtError>

Mark what a joint decision head should read this entry as.

Only decision models (clef and friends) look at this; see LlamaDecisionOrder.

Errors

BatchExtError::NoSuchEntry for a stale handle.

Trait Implementations

impl Debug for LlamaBatchExt

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for LlamaBatchExt

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for LlamaBatchExt