Struct LlamaBatch 

pub struct LlamaBatch<'tokens> {
    pub initialized_logits: Vec<i32>,
    pub llama_batch: llama_batch,
    /* private fields */
}

A safe wrapper around llama_batch.

PartialEq is intentionally not implemented because the underlying llama_batch from the C API contains raw pointers whose address comparison would be meaningless.

Fields

initialized_logits: Vec<i32>

The logits that are initialized. Used by [LlamaContext] to ensure that only initialized logits are accessed.

llama_batch: llama_batch

The underlying llama_batch from the C API.

Implementations

impl<'tokens> LlamaBatch<'tokens>

pub fn clear(&mut self)

Clear the batch. This does not free the memory associated with the batch, but it does reset the number of tokens to 0.

pub fn add( &mut self, sampled_token: &SampledToken, pos: llama_pos, seq_ids: &[i32], logits: bool, ) -> Result<(), BatchAddError>

add a token to the batch for sequences seq_ids at position pos. If logits is true, the token will be initialized and can be read from after the next decode.

Errors

Returns an error if there is insufficient space in the buffer or if integer conversions fail.

pub fn add_sequence( &mut self, tokens: &[LlamaToken], seq_id: i32, logits_all: bool, ) -> Result<(), BatchAddError>

Add a sequence of tokens to the batch for the given sequence id. If logits_all is true, the tokens will be initialized and can be read from after the next decode.

Either way the last token in the sequence will have its logits set to true.

Errors

Returns an error if there is insufficient space in the buffer or if integer conversions fail.

pub fn new(n_tokens: usize, n_seq_max: i32) -> Result<Self, BatchAddError>

Create a new LlamaBatch that can contain up to n_tokens tokens.

Arguments

• n_tokens: the maximum number of tokens that can be added to the batch
• n_seq_max: the maximum number of sequences that can be added to the batch (generally 1 unless you know what you are doing)

Errors

Returns an error if n_tokens exceeds i32::MAX.

pub fn get_one(tokens: &'tokens [LlamaToken]) -> Result<Self, BatchAddError>

llama_batch_get_one Return batch for single sequence of tokens

NOTE: this is a helper function to facilitate transition to the new batch API

Errors

Returns an error if the provided token buffer is empty or if integer conversions fail.

pub const fn n_tokens(&self) -> i32

Returns the number of tokens in the batch.

Trait Implementations

impl<'tokens> Debug for LlamaBatch<'tokens>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for LlamaBatch<'_>

fn drop(&mut self)

Drops the LlamaBatch.

let batch = LlamaBatch::new(512, 1)?;
// frees the memory associated with the batch. (allocated by llama.cpp)
drop(batch);

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.