Struct llama_cpp_2::llama_batch::LlamaBatch

pub struct LlamaBatch { /* private fields */ }

A safe wrapper around llama_batch.

Implementations

impl LlamaBatch

pub fn clear(&mut self)

Clear the batch. This does not free the memory associated with the batch, but it does reset the number of tokens to 0.

pub fn set_last_logit(&mut self, value: bool)

👎Deprecated: not compatible with multiple sequences. prefer setting logits while adding tokens

Set the last token in the batch to [value]. If [value] is true, the token will be initilized after a decode and can be read from. If [value] is false, the token will not be initilized (this is the default).

Panics

Panics if there are no tokens in the batch.

pub fn add( &mut self, LlamaToken: LlamaToken, pos: llama_pos, seq_ids: &[i32], logits: bool )

add a token to the batch for sequences [seq_ids] at position [pos]. If [logits] is true, the token will be initilized and can be read from after the next decode.

Panics

• [self.llama_batch.n_tokens] does not fit into a usize
• [seq_ids.len()] does not fit into a llama_seq_id

pub fn new(n_tokens: usize, embd: i32, n_seq_max: i32) -> Self

Create a new LlamaBatch that cab contain up to n_tokens tokens.

Panics

Panics if n_tokens is greater than i32::MAX.

pub fn add_prompt(&mut self, prompt: &[LlamaToken])

👎Deprecated: not compatible with multiple sequences. use add_prompt_seq instead

add a prompt to the batch at sequence id 0

pub fn add_prompt_seq(&mut self, prompt: &[LlamaToken], seq_ids: &[i32])

add a prompt to the batch at the given sequence ids. This must be the initial prompt as it will be added to the batch starting at position 0.

pub fn n_tokens(&self) -> i32

Returns the number of tokens in the batch.

Trait Implementations

impl Debug for LlamaBatch

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for LlamaBatch

fn drop(&mut self)

Drops the LlamaBatch.

let batch = LlamaBatch::new_from_prompt(&[]);
// frees the memory associated with the batch. (allocated by llama.cpp)
drop(batch);