Enum TokenizationStrategy 

pub enum TokenizationStrategy {
    Whitespace,
    CharacterNgram {
        n: usize,
    },
    Subword {
        vocab_size: usize,
    },
    ScriptAware,
}

Strategy that controls how normalized text is split into tokens.

Variants

Whitespace

Split on Unicode whitespace boundaries; empty tokens are discarded.

CharacterNgram

Sliding window of n characters with a step size of 1.

Fields

n: usize

Width of each n-gram window.

Subword

Simplified BPE approximation: whitespace split, then tokens longer than 10 characters are chunked into pieces of 5 characters each.

Fields

vocab_size: usize

Target vocabulary size (used for future extensions; currently determines the chunk length = max(3, vocab_size / 1000)).

ScriptAware

Contiguous runs of the same Script form individual tokens; whitespace acts as an additional boundary.

Trait Implementations

impl Clone for TokenizationStrategy

fn clone(&self) -> TokenizationStrategy

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizationStrategy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.