Struct UnicodeBpeConfig 

#[non_exhaustive]
pub struct UnicodeBpeConfig {
    pub vocab_size: usize,
    pub min_frequency: usize,
    pub normalize: bool,
    pub byte_fallback: bool,
}

Configuration for the Unicode-aware BPE tokenizer.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

vocab_size: usize

Target vocabulary size (base chars + merge operations).

min_frequency: usize

Minimum pair frequency for a merge operation to be kept.

normalize: bool

Apply NFC-style normalization (simplified: recompose via canonical form).

byte_fallback: bool

Represent characters absent from the training vocabulary as <0xHH> byte tokens.

Trait Implementations

impl Clone for UnicodeBpeConfig

fn clone(&self) -> UnicodeBpeConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for UnicodeBpeConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for UnicodeBpeConfig

fn default() -> Self

Returns the “default value” for a type.