Struct WordPieceTokenizer 

pub struct WordPieceTokenizer { /* private fields */ }

BERT-compatible WordPiece tokenizer.

Loads a vocab.txt file (one token per line, ID = line number) and performs:

1. Lowercasing and Unicode NFD accent stripping
2. Whitespace + punctuation pre-tokenization
3. Greedy longest-match WordPiece splitting (subwords prefixed with ##)
4. [CLS] / [SEP] framing and [PAD] / truncation to max_length

When no vocab file is available, WordPieceTokenizer::encode returns None so callers can fall back to hash-based tokenization.

Implementations

impl WordPieceTokenizer

pub fn load(vocab_path: &Path) -> Option<Self>

Try to load a vocab.txt from the given path.

Returns None if the file does not exist or cannot be read.

pub fn with_max_length(self, max_length: usize) -> Self

Create a tokenizer with a custom max sequence length.

pub fn encode(&self, text: &str) -> Option<(Vec<i64>, Vec<i64>, Vec<i64>)>

Encode text into (input_ids, attention_mask, token_type_ids).

All three vectors have length self.max_length, padded or truncated as needed. Returns None if the vocab is empty.