Struct TokenizerConfig 

pub struct TokenizerConfig {
    pub max_length: usize,
    pub padding: bool,
    pub truncation: bool,
    pub add_special_tokens: bool,
}

Configuration for the tokenizer

Configuration Options

• max_length: Maximum sequence length (default: 512)
• padding: Enable padding to max_length (default: true)
• truncation: Enable truncation at max_length (default: true)
• add_special_tokens: Add special tokens like [CLS], [SEP] (default: true)

Recommended Settings

Production (default):

let config = TokenizerConfig::default();
assert_eq!(config.max_length, 512);
assert!(config.padding);
assert!(config.truncation);

Memory-constrained:

let config = TokenizerConfig {
    max_length: 256,
    padding: false,
    truncation: true,
    add_special_tokens: true,
};

Fields

max_length: usize

Maximum sequence length (tokens)

padding: bool

Enable padding to max_length

truncation: bool

Enable truncation at max_length

add_special_tokens: bool

Add model-specific special tokens ([CLS], [SEP], etc.)

Trait Implementations

impl Clone for TokenizerConfig

fn clone(&self) -> TokenizerConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizerConfig

fn default() -> Self

Returns the “default value” for a type.