Expand description
Tokenizers.
Dispatch matrix:
| Variant | Backend | Fidelity |
|---|---|---|
HfLlama3 { path } | HuggingFace tokenizers (lazy-loaded from a local tokenizer.json) | Exact, matches Llama3 billing |
TiktokenCl100k | tiktoken-rs | Exact (GPT-4, GPT-4o) |
TiktokenO200k | tiktoken-rs | Exact (GPT-5 family) |
Claude | tiktoken-rs | Approximation via cl100k_base |
Llama3 | calibrated | ~3.5 chars/token (mixed text) |
Llama2 | calibrated | ~3.3 chars/token |
CharsDiv4 | built-in | Offline fallback |
HfLlama3 lets users point at any local tokenizer.json (Meta-Llama-3,
Mistral, Qwen, etc. — anything the HuggingFace tokenizers crate can
load). The file is loaded once per path and cached in a process-global
HashMap, so repeated count() calls pay tokenization cost only.
A missing or unparseable vocab path silently falls back to the calibrated heuristic; the scrubber/budget never aborts a pack build because the user pointed at a bad path.