Struct llama_cpp_2::context::sample::sampler::Sampler

pub struct Sampler<'a, C> {
    pub steps: Vec<&'a SampleStep<C>>,
    pub finalizer: &'a SampleFinalizer<C>,
}

A series of sampling steps that will produce a vector of token data.

C is dynamic context that will be passed to the sampling functions. Some sampling steps may require state to be maintained across multiple samples, and this context can be used to store that state. For example, LlamaTokenDataArray::sample_token_mirostat_v2 requires a mu to be shared across multiple samples.

Fields

steps: Vec<&'a SampleStep<C>>

The steps to take when sampling.

finalizer: &'a SampleFinalizer<C>

The final step to select one or more tokens from the remaining candidates.

Implementations

impl<'a, T> Sampler<'a, T>

pub fn new(finalizer: &'a SampleFinalizer<T>) -> Self

Create a new sampler with a given finalizer.

pub fn push_step(&mut self, step: &'a SampleStep<T>)

Adds a step to the sampler.

pub fn sample( &mut self, context: &mut T, candidates: LlamaTokenDataArray ) -> Vec<LlamaTokenData>

Sample a token from the given candidates.

Trait Implementations

impl<T> Debug for Sampler<'_, T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.