Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

A text tokenizer supporting BPE and WordPiece sub-word algorithms.

Implementations

impl Tokenizer

pub fn new(config: TokenizerConfig) -> Self

Create a new tokenizer with the given configuration.

Special tokens ([CLS], [SEP], [PAD], [UNK], [MASK]) are registered automatically.

pub fn bpe() -> Self

Build a default BPE tokenizer.

pub fn wordpiece() -> Self

Build a default WordPiece tokenizer.

pub fn add_token(&mut self, token: &str) -> u32

Add a token to the vocabulary. Returns its ID.

If the token already exists, the existing ID is returned.

pub fn remove_token(&mut self, token: &str) -> bool

Remove a token from the vocabulary. Returns true if it existed.

Special tokens cannot be removed; in that case false is returned.

pub fn vocab_size(&self) -> usize

Current vocabulary size (including special tokens).

pub fn contains_token(&self, token: &str) -> bool

Whether a token is in the vocabulary.

pub fn add_merge_rule(&mut self, left: &str, right: &str)

Add a BPE merge rule. The merged token is also added to the vocab.

pub fn merge_rule_count(&self) -> usize

Number of registered merge rules.

pub fn token_to_id(&self, token: &str) -> Option<u32>

Look up the ID of a token.

pub fn id_to_token(&self, id: u32) -> Option<&str>

Look up the token string for an ID.

pub fn unk_id(&self) -> u32

ID of the [UNK] token.

pub fn cls_id(&self) -> u32

ID of the [CLS] token.

pub fn sep_id(&self) -> u32

ID of the [SEP] token.

pub fn pad_id(&self) -> u32

ID of the [PAD] token.

pub fn encode(&self, text: &str) -> EncodeResult

Encode a text string into token IDs.

The output is truncated to config.max_length.

pub fn decode(&self, ids: &[u32]) -> String

Decode a sequence of token IDs back into a string.

WordPiece continuation tokens (##…) are merged back without spaces.

pub fn encode_batch(&self, texts: &[&str]) -> Vec<EncodeResult>

Encode a batch of texts.

pub fn max_length(&self) -> usize

Maximum sequence length.

pub fn mode(&self) -> &TokenizerMode

Active tokenization mode.

pub fn is_lowercase(&self) -> bool

Whether input is lowercased before tokenization.