Crate toktok 

toktok — a fast, exact BPE tokenizer.

Token ids are byte-identical to tiktoken, and encoding runs ~3.5x faster than bpe-openai and ~10x faster than tiktoken-rs (see the repo’s bench/rust). The bundled encodings are embedded in the binary, so there is nothing to download or ship alongside it.

let tok = toktok::Tokenizer::builtin("cl100k_base")?;

let ids = tok.encode("Hello, toktok! 日本語 🚀".as_bytes());
assert_eq!(tok.decode(&ids), "Hello, toktok! 日本語 🚀".as_bytes());

assert_eq!(tok.count(b"how many tokens is this?"), 6);

One tokenizer is safe to share across threads — load it once:

let docs: Vec<&[u8]> = vec![b"first", b"second", b"third"];
let counts = tok.count_batch(&docs, 0, false);   // 0 threads = every core
let ids = tok.encode_batch(&docs, 0, false);

How it’s fast

Same algorithm as bpe-openai (exact backtracking BPE); the speed comes from data-structure engineering ported from quicktok’s C++: a 2-byte-radix trie whose walk consumes two input bytes per single 8-byte load, dense bijectively-mixed merge-validity memos, hand-compiled SIMD pretokenizers instead of a regex engine, and a single-pass machine that fuses pretokenization with token emission for ASCII text.

Re-exports

pub use pretok::UClass;

pub use pretok_o200k::UClassO;

pub use tokenizer::Scanner;

pub use tokenizer::Tokenizer;

pub use tokenizer::Truncation;

pub use vocab::Vocab;

pub use vocab::VocabError;

pub use vocab::RANK_MAX;

Modules

mb

Multibyte-piece encoder — the TRIE2 walk specialized for pieces that start with a 3-byte UTF-8 lead (CJK and friends). Port of quicktok’s src/trie2_mb.cpp.

pretok

Hand-coded pretokenizer for the FIXED cl100k_base regex — the “specialize the known pattern, skip the general regex engine” win. Reproduces the 8 alternatives: ’(?i:[sdmt]|ll|ve|re) | [^\r\n\p{L}\p{N}]?+\p{L}++ | \p{N}{1,3}+ | ?[^\s\p{L}\p{N}]++[\r\n]+ | \s++$ | \s[\r\n] | \s+(?!\S) | \s Unicode \p{L}/\p{N}/\s come from data/uniclass.bin (exact vs the reference engine).

pretok_o200k

Hand-coded pretokenizer for the FIXED o200k_base regex (GPT-4o). Reproduces the 7 alternatives (ordered, first-match, greedy): 1: [^\r\n\p{L}\p{N}]? [UPPER]* [LOWER]+ (?i:’s|’t|’re|’ve|’m|’ll|’d)? 2: [^\r\n\p{L}\p{N}]? [UPPER]+ [LOWER]* (?i:’s|’t|’re|’ve|’m|’ll|’d)? 3: \p{N}{1,3} 4: ?[^\s\p{L}\p{N}]+ [\r\n/]* 5: \s*[\r\n]+ 6: \s+(?!\S) 7: \s+ where UPPER=[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}], LOWER=[\p{Ll}\p{Lm}\p{Lo}\p{M}].

tokenizer

The public tokenizer: encoding registry, the fused pretok+merge product machines, specials, decode and batch encode.

vocab

Vocab: trie tables, merge-validity memo, and the backtracking BPE encoder.

Constants

BUILTIN_ENCODINGS

Names of the encodings this build can construct with Tokenizer::builtin.