Module llama_batch_ext 

Safe wrapper around llama_batch_ext, llama.cpp’s entry-wise batch.

LlamaBatch wraps the original llama_batch: six parallel arrays, one entry per token, and a single flat embd block for the whole batch. That shape cannot describe a batch where different entries carry differently-shaped inputs, which is what multimodal and speculative decoding need. llama_batch_ext (llama.cpp #24669) replaces the arrays with a list of entries that each own their token id, positions, input embedding and “state” embedding.

Reach for this instead of LlamaBatch when you need one of the things only an entry can express:

• an input embedding on some entries but not others — a vision encoder’s output interleaved with text tokens, where llama_batch forces the whole batch to be one or the other;
• M-RoPE positions, which are [t, y, x, z] per embedding entry rather than one scalar — see LlamaBatchExt::set_positions;
• a state embedding (llama_batch_ext_set_embd_state), the hidden state an MTP draft context reads from the target model.

For plain token decoding there is nothing to gain: LlamaBatch is the same work with less ceremony, and llama_decode is where this crate’s decode lifecycle hooks live — see LlamaContext::process.

Example

use llama_cpp_4::context::LlamaContext;
use llama_cpp_4::llama_batch_ext::{LlamaBatchExt, LlamaProcessType};
use llama_cpp_4::token::LlamaToken;

let mut batch = LlamaBatchExt::new(ctx)?;

let mut pos = 0;
let mut last = None;
for &token in tokens {
    let entry = batch.add_token(token, 0)?;
    batch.set_pos(entry, pos)?;
    pos += 1;
    last = Some(entry);
}
if let Some(entry) = last {
    batch.set_output(entry, true)?;
}

ctx.process(&mut batch, LlamaProcessType::Decode)?;

Structs

BatchEntry

A handle to one entry of a LlamaBatchExt, returned by the add_* methods and accepted by the set_* methods.

EmbeddingRows

A borrowed, row-major block of embedding values, mirroring llama_embd.

LlamaBatchExt

A safe wrapper around llama_batch_ext.

Enums

BatchExtError

Errors from building a LlamaBatchExt.

LlamaDecisionOrder

What a joint decision head (clef and friends) reads an entry as.

LlamaProcessType

Which graph to run a batch through.

Constants

MROPE_SECTIONS

Positions an embedding entry carries on an M-RoPE or interleaved-M-RoPE model: [t, y, x, z], where t doubles as the KV-cache position.