Struct llama_cpp_2::llama_batch::LlamaBatch

pub struct LlamaBatch { /* private fields */ }

A safe wrapper around llama_batch.

Implementations

impl LlamaBatch

pub fn clear(&mut self)

Clear the batch. This does not free the memory associated with the batch, but it does reset the number of tokens to 0.

pub fn add( &mut self, LlamaToken: LlamaToken, pos: llama_pos, seq_ids: &[i32], logits: bool )

add a token to the batch for sequences [seq_ids] at position [pos]. If [logits] is true, the token will be initilized and can be read from after the next decode.

Panics

• [self.llama_batch.n_tokens] does not fit into a usize
• [seq_ids.len()] does not fit into a llama_seq_id

pub fn new(n_tokens: usize, n_seq_max: i32) -> Self

Create a new LlamaBatch that cab contain up to n_tokens tokens.

Arguments

• n_tokens: the maximum number of tokens that can be added to the batch
• n_seq_max: the maximum number of sequences that can be added to the batch (generally 1 unless you know what you are doing)

Panics

Panics if n_tokens is greater than i32::MAX.

pub fn n_tokens(&self) -> i32

Returns the number of tokens in the batch.

Trait Implementations

impl Debug for LlamaBatch

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for LlamaBatch

fn drop(&mut self)

Drops the LlamaBatch.

let batch = LlamaBatch::new(512, 1);
// frees the memory associated with the batch. (allocated by llama.cpp)
drop(batch);