Struct Sampler 

pub struct Sampler {
    pub temperature: f32,
    pub top_k: usize,
    pub top_p: f32,
    /* private fields */
}

Next-token sampler: temperature, top-k and nucleus (top-p) filtering.

temperature <= 0 means greedy decoding. top_k == 0 and top_p >= 1 disable the respective filters.

Fields

temperature: f32

Softmax temperature.

top_k: usize

Keep only the top_k most likely tokens (0 = all).

top_p: f32

Keep the smallest set of tokens whose probability reaches top_p.

Implementations

impl Sampler

pub fn new( temperature: f32, top_k: usize, top_p: f32, seed: Option<u64>, ) -> Self

A sampler; seed makes the draws reproducible.

pub fn sample(&mut self, logits: &[f32]) -> u32

Pick a token id from one row of logits.

Trait Implementations

impl Default for Sampler

fn default() -> Self

Returns the “default value” for a type.