Struct LlamaSampler 

pub struct LlamaSampler { /* private fields */ }

A safe wrapper around llama_sampler.

Implementations

impl LlamaSampler

pub fn new() -> Self

Create new sampler with default params.

Panics

Panics if llama.cpp returns a null pointer.

pub fn sample(&self, ctx: &LlamaContext<'_>, idx: i32) -> LlamaToken

Sample and accept a token from the idx-th output of the last evaluation

pub fn apply(&mut self, data_array: &mut LlamaTokenDataArray)

Applies this sampler to a LlamaTokenDataArray.

pub fn accept(&mut self, token: LlamaToken)

Accepts a token from the sampler, possibly updating the internal state of certain samplers (e.g. grammar, repetition, etc.)

pub fn accept_many( &mut self, tokens: impl IntoIterator<Item = impl Borrow<LlamaToken>>, )

Accepts several tokens from the sampler or context, possibly updating the internal state of certain samplers (e.g. grammar, repetition, etc.)

pub fn with_tokens( self, tokens: impl IntoIterator<Item = impl Borrow<LlamaToken>>, ) -> Self

Accepts several tokens from the sampler or context, possibly updating the internal state of certain samplers (e.g. grammar, repetition, etc.)

pub fn chain(samplers: impl IntoIterator<Item = Self>, no_perf: bool) -> Self

Combines a list of samplers into a single sampler that applies each component sampler one after another.

If you are using a chain to select a token, the chain should always end with one of LlamaSampler::greedy, LlamaSampler::dist, LlamaSampler::mirostat, and LlamaSampler::mirostat_v2.

Panics

Panics if llama.cpp returns a null pointer.

pub fn chain_simple(samplers: impl IntoIterator<Item = Self>) -> Self

Same as Self::chain with no_perf = false.

Panics

Panics if llama.cpp returns a null pointer.

Example

use llama_cpp_4::token::{
   LlamaToken,
   data::LlamaTokenData,
   data_array::LlamaTokenDataArray
};
use llama_cpp_4::sampling::LlamaSampler;

let mut data_array = LlamaTokenDataArray::new(vec![
    LlamaTokenData::new(LlamaToken(0), 0., 0.),
    LlamaTokenData::new(LlamaToken(1), 1., 0.),
    LlamaTokenData::new(LlamaToken(2), 2., 0.),
], false);

data_array.apply_sampler(&mut LlamaSampler::chain_simple([
    LlamaSampler::temp(0.5),
    LlamaSampler::greedy(),
]));

assert_eq!(data_array.data[0].logit(), 0.);
assert_eq!(data_array.data[1].logit(), 2.);
assert_eq!(data_array.data[2].logit(), 4.);

assert_eq!(data_array.data.len(), 3);
assert_eq!(data_array.selected_token(), Some(LlamaToken(2)));

pub fn temp(t: f32) -> Self

Updates the logits l_i’ = l_i/t. When t <= 0.0, the maximum logit is kept at its original value, the rest are set to -inf.

Panics

Panics if llama.cpp returns a null pointer.

Example:

use llama_cpp_4::token::{
   LlamaToken,
   data::LlamaTokenData,
   data_array::LlamaTokenDataArray
};
use llama_cpp_4::sampling::LlamaSampler;

let mut data_array = LlamaTokenDataArray::new(vec![
    LlamaTokenData::new(LlamaToken(0), 0., 0.),
    LlamaTokenData::new(LlamaToken(1), 1., 0.),
    LlamaTokenData::new(LlamaToken(2), 2., 0.),
], false);

data_array.apply_sampler(&mut LlamaSampler::temp(0.5));

assert_eq!(data_array.data[0].logit(), 0.);
assert_eq!(data_array.data[1].logit(), 2.);
assert_eq!(data_array.data[2].logit(), 4.);

pub fn temp_ext(t: f32, delta: f32, exponent: f32) -> Self

Dynamic temperature implementation (a.k.a. entropy) described in the paper https://arxiv.org/abs/2309.02772.

Panics

Panics if llama.cpp returns a null pointer.

pub fn top_k(k: i32) -> Self

Top-K sampling described in academic paper “The Curious Case of Neural Text Degeneration” https://arxiv.org/abs/1904.09751.

Panics

Panics if llama.cpp returns a null pointer.

Example:

use llama_cpp_4::token::{
   LlamaToken,
   data::LlamaTokenData,
   data_array::LlamaTokenDataArray
};
use llama_cpp_4::sampling::LlamaSampler;

let mut data_array = LlamaTokenDataArray::new(vec![
    LlamaTokenData::new(LlamaToken(0), 0., 0.),
    LlamaTokenData::new(LlamaToken(1), 1., 0.),
    LlamaTokenData::new(LlamaToken(2), 2., 0.),
    LlamaTokenData::new(LlamaToken(3), 3., 0.),
], false);

data_array.apply_sampler(&mut LlamaSampler::top_k(2));

assert_eq!(data_array.data.len(), 2);
assert_eq!(data_array.data[0].id(), LlamaToken(3));
assert_eq!(data_array.data[1].id(), LlamaToken(2));

pub fn typical(p: f32, min_keep: usize) -> Self

Locally Typical Sampling implementation described in the paper https://arxiv.org/abs/2202.00666.

Panics

Panics if llama.cpp returns a null pointer.

pub fn top_p(p: f32, min_keep: usize) -> Self

Nucleus sampling described in academic paper “The Curious Case of Neural Text Degeneration” https://arxiv.org/abs/1904.09751.

Panics

Panics if llama.cpp returns a null pointer.

pub fn min_p(p: f32, min_keep: usize) -> Self

Minimum P sampling as described in https://github.com/ggerganov/llama.cpp/pull/3841.

Panics

Panics if llama.cpp returns a null pointer.

pub fn xtc(p: f32, t: f32, min_keep: usize, seed: u32) -> Self

XTC sampler as described in https://github.com/oobabooga/text-generation-webui/pull/6335.

Panics

Panics if llama.cpp returns a null pointer.

pub fn grammar( model: &LlamaModel, grammar_str: &str, grammar_root: &str, ) -> Self

Grammar sampler

Panics

• If either of grammar_str or grammar_root contain null bytes.
• If llama.cpp returns a null pointer.

pub fn dry( &self, model: &LlamaModel, multiplier: f32, base: f32, allowed_length: i32, penalty_last_n: i32, seq_breakers: impl IntoIterator<Item = impl AsRef<[u8]>>, ) -> Self

DRY sampler, designed by p-e-w, as described in: https://github.com/oobabooga/text-generation-webui/pull/5677, porting Koboldcpp implementation authored by pi6am: https://github.com/LostRuins/koboldcpp/pull/982

Panics

• If any string in seq_breakers contains null bytes.
• If llama.cpp returns a null pointer.

pub fn penalties( n_vocab: i32, penalty_last_n: i32, penalty_repeat: f32, penalty_freq: f32, penalty_present: f32, ) -> Self

Penalizes tokens for being present in the context.

Parameters:

• n_vocab: LlamaModel::n_vocab
• penalty_last_n: last n tokens to penalize (0 = disable penalty)
• penalty_repeat: repetition penalty (must be > 0.0, 1.0 = disabled)
• penalty_freq: frequency penalty (must be finite, 0.0 = disabled)
• penalty_present: presence penalty (must be finite, 0.0 = disabled)

If penalty_last_n is 0, or every penalty sits at its disabled value, llama.cpp returns a no-op sampler named "?penalties" — see Self::name.

Panics

Panics if llama.cpp returns a null pointer.

pub fn penalties_simple( n_vocab: i32, penalty_last_n: i32, penalty_repeat: f32, ) -> Self

Same as Self::penalties with sensible defaults: penalty_freq = 0.0 and penalty_present = 0.0.

Parameters:

• n_vocab: LlamaModel::n_vocab
• penalty_last_n: last n tokens to penalize (0 = disable)
• penalty_repeat: repetition penalty (must be > 0.0, 1.0 = disabled)

Panics

Panics if llama.cpp returns a null pointer.

pub fn mirostat(n_vocab: i32, seed: u32, tau: f32, eta: f32, m: i32) -> Self

Mirostat 1.0 algorithm described in the paper https://arxiv.org/abs/2007.14966. Uses tokens instead of words.

Panics

Panics if llama.cpp returns a null pointer.

Parameters:

• n_vocab: LlamaModel::n_vocab
• seed: Seed to initialize random generation with.
• tau: The target cross-entropy (or surprise) value you want to achieve for the generated text. A higher value corresponds to more surprising or less predictable text, while a lower value corresponds to less surprising or more predictable text.
• eta: The learning rate used to update mu based on the error between the target and observed surprisal of the sampled word. A larger learning rate will cause mu to be updated more quickly, while a smaller learning rate will result in slower updates.
• m: The number of tokens considered in the estimation of s_hat. This is an arbitrary value that is used to calculate s_hat, which in turn helps to calculate the value of k. In the paper, they use m = 100, but you can experiment with different values to see how it affects the performance of the algorithm.

pub fn mirostat_v2(seed: u32, tau: f32, eta: f32) -> Self

Mirostat 2.0 algorithm described in the paper https://arxiv.org/abs/2007.14966. Uses tokens instead of words.

Panics

Panics if llama.cpp returns a null pointer.

Parameters:

• seed: Seed to initialize random generation with.
• tau: The target cross-entropy (or surprise) value you want to achieve for the generated text. A higher value corresponds to more surprising or less predictable text, while a lower value corresponds to less surprising or more predictable text.
• eta: The learning rate used to update mu based on the error between the target and observed surprisal of the sampled word. A larger learning rate will cause mu to be updated more quickly, while a smaller learning rate will result in slower updates.

pub fn dist(seed: u32) -> Self

Selects a token at random based on each token’s probabilities.

Panics

Panics if llama.cpp returns a null pointer.

pub fn greedy() -> Self

Selects the most likely token.

Panics

Panics if llama.cpp returns a null pointer.

Example:

use llama_cpp_4::token::{
   LlamaToken,
   data::LlamaTokenData,
   data_array::LlamaTokenDataArray
};
use llama_cpp_4::sampling::LlamaSampler;

let mut data_array = LlamaTokenDataArray::new(vec![
    LlamaTokenData::new(LlamaToken(0), 0., 0.),
    LlamaTokenData::new(LlamaToken(1), 1., 0.),
], false);

data_array.apply_sampler(&mut LlamaSampler::greedy());

assert_eq!(data_array.data.len(), 2);
assert_eq!(data_array.selected_token(), Some(LlamaToken(1)));

pub fn top_n_sigma(n: f32) -> Self

Top-N sigma sampling.

Keeps tokens within N standard deviations of the maximum logit.

Panics

Panics if llama.cpp returns a null pointer.

pub fn adaptive_p(target: f32, decay: f32, seed: u32) -> Self

Adaptive P sampling.

Panics

Panics if llama.cpp returns a null pointer.

Parameters

• target: Target probability.
• decay: Decay rate.
• seed: Random seed.

pub fn logit_bias(n_vocab: i32, biases: &[(LlamaToken, f32)]) -> Self

Logit bias sampler.

Applies additive bias to specific token logits before sampling.

Panics

Panics if llama.cpp returns a null pointer.

Parameters

• n_vocab: Number of tokens in the vocabulary (LlamaModel::n_vocab).
• biases: Slice of (token_id, bias) pairs.

pub fn infill(model: &LlamaModel) -> Self

Infill sampler.

Reorders token probabilities for fill-in-the-middle tasks.

Panics

Panics if llama.cpp returns a null pointer.

pub fn get_seed(&self) -> u32

Get the seed of the sampler.

Returns LLAMA_DEFAULT_SEED if the sampler is not seeded.

pub fn name(&self) -> String

Get the name of the sampler.

Disabled samplers

When a constructor is handed parameters that make it a no-op — e.g. Self::temp with 1.0, or Self::penalties with penalty_last_n = 0 — llama.cpp does not build that sampler. It substitutes an identity sampler whose name carries a ? prefix ("?temp", "?penalties"). Construction still succeeds, so this name is the only signal that the sampler will not do anything. Affects temp, temp_ext, top_k, top_p, min_p, typical, xtc, top_n_sigma, dry, and penalties.

Panics

Panics if the name is not valid UTF-8.

pub fn reset(&mut self)

Reset the sampler state (e.g. grammar, repetition penalties).

pub fn chain_n(&self) -> i32

Get the number of samplers in a chain.

Returns 0 if this sampler is not a chain.

pub fn chain_remove(&mut self, i: i32) -> Self

Remove and return the sampler at position i from a chain.

The returned sampler is owned by the caller and will be freed on drop.

Panics

Panics if i is out of range or if llama.cpp returns a null pointer.

pub fn grammar_lazy_patterns( model: &LlamaModel, grammar_str: &str, grammar_root: &str, trigger_patterns: &[&str], trigger_tokens: &[LlamaToken], ) -> Self

Grammar sampler with lazy activation via regex patterns.

The grammar is only activated when one of the trigger patterns or trigger tokens matches.

Panics

• If grammar_str or grammar_root contain null bytes.
• If any trigger pattern contains null bytes.
• If llama.cpp returns a null pointer.

pub fn clone_sampler(&self) -> Self

Clone this sampler.

Creates an independent copy of this sampler with the same state.

Panics

Panics if llama.cpp returns a null pointer.

pub fn copy_state_from(&mut self, src: &Self)

Copy mutable state from src into this sampler, in place.

Unlike Self::clone_sampler, which allocates a new sampler, this overwrites the state of an existing one and so is the cheap way to rewind a sampler to a checkpoint in a loop. Added upstream in llama.cpp b10470 (llama_sampler_copy).

Safety and preconditions

llama.cpp requires src and self to be the same sampler type with the same configuration — e.g. two dist samplers, or two chains built the same way. Copying between mismatched samplers is undefined behaviour upstream and is not checked here, so treat the pairing as the caller’s contract. A sampler produced by src.clone_sampler() always satisfies it.

pub fn perf_print(&self)

Print sampler performance data.

pub fn perf_reset(&mut self)

Reset sampler performance counters.

pub fn perf_data(&self) -> llama_perf_sampler_data

Get sampler performance data.

pub unsafe fn chain_get_ptr(&self, i: i32) -> *mut llama_sampler

Get a non-owning reference to the ith sampler in a chain.

Safety

The returned pointer is owned by the chain. Do not free it or use it after the chain is dropped or modified.

pub unsafe fn from_raw( iface: *mut llama_sampler_i, ctx: llama_sampler_context_t, ) -> Self

Create a sampler from a raw interface and context.

Safety

The caller must ensure that iface and ctx are valid and that the interface functions properly manage the context lifetime.

Panics

Panics if llama.cpp returns a null pointer.

pub fn common() -> Self

Creates a new instance of LlamaSampler with common sampling parameters.

This function initializes a LlamaSampler using default values from common_sampler_params and configures it with common settings such as top_k, top_p, temperature, and seed values.

Panics

Panics if llama.cpp returns a null pointer.

Returns

A LlamaSampler instance configured with the common sampling parameters.

Trait Implementations

impl Debug for LlamaSampler

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for LlamaSampler

fn default() -> Self

Returns the “default value” for a type.

impl Drop for LlamaSampler

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.