Struct Tokenizer 

#[non_exhaustive]
pub struct Tokenizer<T: TokenType = Token> { /* private fields */ }

Construct via Tokenizer::train or Tokenizer::load. Use by calling encode / decode or inspect the vocabulary with Tokenizer::vocab_size, Tokenizer::token_to_string, Tokenizer::str_to_token, and Tokenizer::tokens.

Token representation

Tokens are T values where T: TokenType. By default, T = Token. Tokens 0..256 correspond to single bytes (with printable ASCII mapped to their character representation and control/high bytes to <hex> notation). 257th token is the word-start symbol ▁. Tokens 257.. are special tokens, plus Unicode codepoints extracted from the training data, plus those learned by BPE merges (if any).

Implementations

impl<T: TokenType> Tokenizer<T>

pub fn vocab_size(&self) -> usize

Returns the total vocabulary size (byte tokens + codepoint + BPE merge tokens).

pub fn token_to_string(&self, tok: T) -> Option<&str>

Maps a token ID to its string representation.

Returns None if the token ID exceeds the vocabulary range. For byte tokens 0..256, this returns the byte’s display form (printable ASCII as-is, others as <hex>).

pub fn str_to_token(&self, s: &str) -> Option<T>

Maps a string to its token ID, if the string corresponds to a single token.

Returns None if the string is not present as an atomic vocabulary entry. Note that multi-token sequences (e.g. "hello world") will not match — this only succeeds when the entire input maps to exactly one token.

pub fn tokens(&self) -> impl Iterator<Item = (T, &str)>

Iterates over all (token_id, string) pairs in the vocabulary.

Byte tokens come first (0..256), followed by sorted Unicode codepoints and BPE merge tokens in merge order.

pub fn load(path: &Path) -> Result<Self, TokenizerError>

Loads a tokenizer from a JSON file previously saved with Tokenizer::save.

pub fn save(&self, path: &Path) -> Result<(), TokenizerError>

Saves the tokenizer vocabulary to a JSON file.

Only the learned vocabulary (257..) is stored; the first 256 byte-tokens plus word-start ▁ are implicit. The file also records the crate version for forward compatibility.

pub fn train( s: &str, special_tokens: &[&str], max_extra_tokens: Option<usize>, ) -> Self

Trains a new BPE tokenizer on the given text.

The initial vocabulary always contains 257 initial tokens (256 single bytes plus the word start symbol ▁), in addition to the provided special tokens.

In case max_extra_tokens is Some(n), other unicode codepoints found in s are appended to the vocabulary first, and then BPE merge operations are applied until the vocabulary reaches 257 + special_tokens.len() + n entries.

If n is less than the number of unicode codepoints, then only the top-n are appended from those in s, sorted longest-first, then lexicographically.

pub fn encode(&self, s: &str) -> Vec<T>

Encodes a string into a sequence of token IDs.

The input is first normalized, then chunked by a GPT-style regex pattern, and each chunk is matched against the vocabulary using leftmost-longest Aho-Corasick search. Characters not in the vocabulary fall back to their UTF-8 byte tokens.

pub fn decode(&self, v: &[T]) -> String

Decodes a sequence of token IDs back into a string.

Consecutive byte tokens that form valid UTF-8 are reassembled into characters. The result is then denormalized (removing or converting ▁ markers back to spaces where appropriate).

pub fn token_size() -> usize

Returns the size in bytes of a single token value.