Module tokenizer 

Tokenization utilities Tokenizer implementation using HuggingFace tokenizers

This module provides tokenization using the HuggingFace tokenizers library, with fallback to a basic tokenizer for simple use cases.

Structs

BatchTokenizer

Batch tokenization for multiple texts

SpecialTokens

Special tokens configuration

Tokenizer

Tokenizer implementation that wraps HuggingFace tokenizers

Constants

BOS_TOKEN_ID

Special token IDs (defaults, may be overridden by loaded tokenizer)

EOS_TOKEN_ID

PAD_TOKEN_ID

UNK_TOKEN_ID