Struct LlamaTokenDataArray 

pub struct LlamaTokenDataArray {
    pub data: Vec<LlamaTokenData>,
    pub selected: Option<usize>,
    pub sorted: bool,
}

a safe wrapper around llama_token_data_array.

Fields

data: Vec<LlamaTokenData>

the underlying data

selected: Option<usize>

the index of the selected token in data

sorted: bool

is the data sorted?

Implementations

impl LlamaTokenDataArray

pub fn new(data: Vec<LlamaTokenData>, sorted: bool) -> Self

Create a new LlamaTokenDataArray from a vector and whether or not the data is sorted.

let array = LlamaTokenDataArray::new(vec![
        LlamaTokenData::new(LlamaToken(0), 0.0, 0.0),
        LlamaTokenData::new(LlamaToken(1), 0.1, 0.1)
   ], false);
assert_eq!(array.data.len(), 2);
assert_eq!(array.sorted, false);

pub fn from_iter<TIterator>( data: TIterator, sorted: bool, ) -> LlamaTokenDataArray

where TIterator: IntoIterator<Item = LlamaTokenData>,

Create a new LlamaTokenDataArray from an iterator and whether or not the data is sorted.

let array = LlamaTokenDataArray::from_iter([
    LlamaTokenData::new(LlamaToken(0), 0.0, 0.0),
    LlamaTokenData::new(LlamaToken(1), 0.1, 0.1)
], false);
assert_eq!(array.data.len(), 2);
assert_eq!(array.sorted, false);

pub fn selected_token(&self) -> Option<LlamaToken>

Returns the current selected token, if one exists.

impl LlamaTokenDataArray

pub unsafe fn modify_as_c_llama_token_data_array<TResult>( &mut self, modify: impl FnOnce(&mut llama_token_data_array) -> TResult, ) -> TResult

Modify the underlying data as a llama_token_data_array. and reconstruct the LlamaTokenDataArray.

Panics

Panics if some of the safety conditions are not met. (we cannot check all of them at runtime so breaking them is UB)

Safety

The returned array formed by the data pointer and the length must entirely consist of initialized token data and the length must be less than the capacity of this array’s data buffer. If the data is not sorted, sorted must be false.

pub fn apply_sampler(&mut self, sampler: &LlamaSampler)

Modifies the data array by applying a sampler to it

pub fn with_sampler(self, sampler: &mut LlamaSampler) -> Self

Modifies the data array by applying a sampler to it

pub fn sample_token(&mut self, seed: u32) -> LlamaToken

Randomly selects a token from the candidates based on their probabilities.

Panics

If the internal llama.cpp sampler fails to select a token.

pub fn sample_token_greedy(&mut self) -> LlamaToken

Selects the token with the highest probability.

Panics

If the internal llama.cpp sampler fails to select a token.

Trait Implementations

impl Clone for LlamaTokenDataArray

fn clone(&self) -> LlamaTokenDataArray

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LlamaTokenDataArray

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for LlamaTokenDataArray

fn eq(&self, other: &LlamaTokenDataArray) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for LlamaTokenDataArray