Struct llama_cpp_2::token::data_array::LlamaTokenDataArray

pub struct LlamaTokenDataArray {
    pub data: Vec<LlamaTokenData>,
    pub sorted: bool,
}

a safe wrapper around llama_token_data_array.

Fields

data: Vec<LlamaTokenData>

the underlying data

sorted: bool

is the data sorted?

Implementations

impl LlamaTokenDataArray

pub fn new(data: Vec<LlamaTokenData>, sorted: bool) -> Self

Create a new LlamaTokenDataArray from a vector and weather or not the data is sorted.

let array = LlamaTokenDataArray::new(vec![
        LlamaTokenData::new(LlamaToken(0), 0.0, 0.0),
        LlamaTokenData::new(LlamaToken(1), 0.1, 0.1)
   ], false);
assert_eq!(array.data.len(), 2);
assert_eq!(array.sorted, false);

pub fn from_iter<T>(data: T, sorted: bool) -> LlamaTokenDataArray

where T: IntoIterator<Item = LlamaTokenData>,

Create a new LlamaTokenDataArray from an iterator and weather or not the data is sorted.

let array = LlamaTokenDataArray::from_iter([
    LlamaTokenData::new(LlamaToken(0), 0.0, 0.0),
    LlamaTokenData::new(LlamaToken(1), 0.1, 0.1)
], false);
assert_eq!(array.data.len(), 2);
assert_eq!(array.sorted, false);

impl LlamaTokenDataArray

pub fn sample_repetition_penalty( &mut self, ctx: Option<&mut LlamaContext<'_>>, last_tokens: &[LlamaToken], penalty_last_n: usize, penalty_repeat: f32, penalty_freq: f32, penalty_present: f32 )

Repetition penalty described in CTRL academic paper, with negative logit fix. Frequency and presence penalties described in OpenAI API.

Parameters

• ctx - the context to use. May be None if you do not care to record the sample timings.
• last_tokens - the last tokens in the context.
• penalty_last_n - the number of tokens to consider for the repetition penalty. (1.0 for no penalty)
• penalty_repeat - the repetition penalty. (0.0 for no penalty)
• penalty_freq - the frequency penalty. (0.0 for no penalty)

Example

let history = vec![
  LlamaToken::new(2),
  LlamaToken::new(1),
  LlamaToken::new(0),
];

let candidates = vec![
   LlamaToken::new(0),
   LlamaToken::new(1),
   LlamaToken::new(2),
   LlamaToken::new(3),
];

let mut candidates = LlamaTokenDataArray::from_iter(candidates.iter().map(|&token| LlamaTokenData::new(token, 0.0, 0.0)), false);

candidates.sample_repetition_penalty(None, &history, 2, 1.1, 0.1, 0.1);

let token_logits = candidates.data.into_iter().map(|token_data| (token_data.id(), token_data.logit())).collect::<BTreeMap<_, _>>();
assert_eq!(token_logits[&LlamaToken(0)], 0.0, "expected no penalty as it is out of `penalty_last_n`");
assert!(token_logits[&LlamaToken(1)] < 0.0, "expected penalty as it is in `penalty_last_n`");
assert!(token_logits[&LlamaToken(2)] < 0.0, "expected penalty as it is in `penalty_last_n`");
assert_eq!(token_logits[&LlamaToken(3)], 0.0, "expected no penalty as it is not in `history`");

Trait Implementations

impl Clone for LlamaTokenDataArray

fn clone(&self) -> LlamaTokenDataArray

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LlamaTokenDataArray

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for LlamaTokenDataArray

fn eq(&self, other: &LlamaTokenDataArray) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for LlamaTokenDataArray