Struct LlamaSampler 

pub struct LlamaSampler { /* private fields */ }

A safe wrapper around llama_sampler.

Implementations

impl LlamaSampler

pub fn new(params: LlamaSamplerChainParams) -> Result<Self, LlamaSamplerError>

Create a new LlamaSampler from the given parameters.

Errors

Returns an error if the underlying C++ code returns a null pointer.

pub fn add_greedy(self) -> Self

Samples the token with the largest probability.

pub fn add_dist(self, seed: u32) -> Self

Samples according to the probability distribution of the tokens.

pub fn add_top_k(self, k: i32) -> Self

Top-K sampling described in academic paper “The Curious Case of Neural Text Degeneration” https://arxiv.org/abs/1904.09751

pub fn add_top_p(self, p: f32, min_keep: usize) -> Self

Nucleus sampling described in academic paper “The Curious Case of Neural Text Degeneration” https://arxiv.org/abs/1904.09751

pub fn add_min_p(self, p: f32, min_keep: usize) -> Self

Minimum P sampling as described in https://github.com/ggerganov/llama.cpp/pull/3841

pub fn add_typical(self, p: f32, min_keep: usize) -> Self

Locally Typical Sampling implementation described in the paper https://arxiv.org/abs/2202.00666.

pub fn add_temp(self, t: f32) -> Self

Updates the logits l_i` = l_i/t. When t <= 0.0f, the maximum logit is kept at it’s original value, the rest are set to -inf

pub fn add_temp_ext(self, t: f32, delta: f32, exponent: f32) -> Self

Dynamic temperature implementation (a.k.a. entropy) described in the paper https://arxiv.org/abs/2309.02772.

pub fn add_mirostat( self, n_vocab: i32, seed: u32, tau: f32, eta: f32, m: i32, ) -> Self

Mirostat 1.0 algorithm described in the paper https://arxiv.org/abs/2007.14966. Uses tokens instead of words.

Arguments

• candidates - A vector of llama_token_data containing the candidate tokens, their probabilities (p), and log-odds (logit) for the current position in the generated text.
• tau - The target cross-entropy (or surprise) value you want to achieve for the generated text. A higher value corresponds to more surprising or less predictable text, while a lower value corresponds to less surprising or more predictable text.
• eta - The learning rate used to update mu based on the error between the target and observed surprisal of the sampled word. A larger learning rate will cause mu to be updated more quickly, while a smaller learning rate will result in slower updates.
• m - The number of tokens considered in the estimation of s_hat. This is an arbitrary value that is used to calculate s_hat, which in turn helps to calculate the value of k. In the paper, they use m = 100, but you can experiment with different values to see how it affects the performance of the algorithm.
• mu - Maximum cross-entropy. This value is initialized to be twice the target cross-entropy (2 * tau) and is updated in the algorithm based on the error between the target and observed surprisal.

pub fn add_mirostat_v2(self, seed: u32, tau: f32, eta: f32) -> Self

Mirostat 2.0 algorithm described in the paper https://arxiv.org/abs/2007.14966. Uses tokens instead of words.

Arguments

• candidates - A vector of llama_token_data containing the candidate tokens, their probabilities (p), and log-odds (logit) for the current position in the generated text.
• tau - The target cross-entropy (or surprise) value you want to achieve for the generated text. A higher value corresponds to more surprising or less predictable text, while a lower value corresponds to less surprising or more predictable text.
• eta - The learning rate used to update mu based on the error between the target and observed surprisal of the sampled word. A larger learning rate will cause mu to be updated more quickly, while a smaller learning rate will result in slower updates.
• mu - Maximum cross-entropy. This value is initialized to be twice the target cross-entropy (2 * tau) and is updated in the algorithm based on the error between the target and observed surprisal.

pub fn add_grammar( self, model: &LlamaModel, grammar_str: &str, grammar_root: &str, ) -> Self

Samples constrained by a context-free grammar in the GGML BNF (GBNF) format.

Panics

Panics if a provided string contains a null byte.

pub fn add_penalties( self, n_vocab: i32, special_eos_id: i32, linefeed_id: i32, penalty_last_n: i32, penalty_repeat: f32, penalty_freq: f32, penalty_present: f32, penalize_nl: bool, ignore_eos: bool, ) -> Self

Adds penalties to the sampler. This can be used to penalize certain patterns in the generated text, such as repeating the same token multiple times or using the same token too frequently.

pub fn sample(&self, ctx: &LlamaContext<'_>, idx: i32) -> LlamaToken

Sample and accept a token from the idx-th output of the last evaluation

pub fn accept(&mut self, token: LlamaToken)

Accepts a token from the sampler, possibly updating the internal state of certain samplers (e.g. grammar, repetition, etc.)

Trait Implementations

impl Debug for LlamaSampler

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for LlamaSampler

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.