Struct kalosm_sample::StructuredSampler

pub struct StructuredSampler<V: Parser<Error = E, Output = O, PartialState = PA>, E, O, PA> { /* private fields */ }

A sampler that enforces the given validator. Any tokens that form an invalid sequence will have a probability of 0

Implementations

impl<V: Parser<Error = E, Output = O, PartialState = PA>, E, O, PA> StructuredSampler<V, E, O, PA>

pub fn new( structure: V, current_token_count: usize, tokenizer: impl Into<DynTokenizer> ) -> Self

Create a new structured sampler that starts validating tokens at the given token count.

Trait Implementations

impl<V: Parser<Error = E, Output = O, PartialState = PA>, E, O, PA> Debug for StructuredSampler<V, E, O, PA>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<V: Parser<Error = E, Output = O, PartialState = PA> + CreateParserState + Send + Sync, E, O, PA> Sampler for StructuredSampler<V, E, O, PA>

fn sample<'a>( &mut self, res: &mut dyn HasSamplerResources, logits: &'a mut Logits ) -> Result<&'a mut Logits>

Runs the Sampler. Depending on the type of Sampler, this may produce a token id.

fn sampled_token_id(&self) -> Option<u32>

Returns the last sampled token id if available.

fn sample_token( &mut self, res: &mut dyn HasSamplerResources, logits: &mut Logits ) -> Result<Option<u32>, Error>

Run the sampler and return the last sampled token id if available.