Struct Sampler 

pub struct Sampler { /* private fields */ }

Implementations

impl Sampler

pub fn new(vocab_size: usize, temperature: f32, topp: f32) -> Self

pub fn sample<S: StorageMut<f32, Dyn>>( &mut self, logits: &mut Vector<f32, Dyn, S>, ) -> usize

pub fn sample_argmax<S: StorageMut<f32, Dyn>>( probabilities: &Vector<f32, Dyn, S>, ) -> usize

pub fn sample_mult<S: StorageMut<f32, Dyn>>( probabilities: &Vector<f32, Dyn, S>, coin: f32, ) -> usize

Sample index from probabilities.

The probabilities must sum to 1.0. coin is a random number in [0, 1).

pub fn sample_topp<S: StorageMut<f32, Dyn>>( &mut self, probabilities: &Vector<f32, Dyn, S>, coin: f32, ) -> usize

Top-p sampling (or “nucleus sampling”) samples from the smallest set of tokens that exceed probability topp.

This way we never sample token that have very low probabilities and are less likely to go “off the rails”. Coin is a random number in [0, 1).