snaptokens is an optimized fork of fastokens that is significantly quicker and simplified to work in sinter. This repo will be archived once ported to sinter.
Give snaptokens a local Hugging Face tokenizer.json and get token IDs back. Opt in to a .tkz copy when faster future loads matter.
The current twelve-tokenizer, thirteen-host comparison is 2.19× faster than Gigatoken, 14.96× faster than upstream fastokens, and 49.39× faster than Hugging Face by geometric mean of paired medians. Snaptokens wins 777/780 Gigatoken cells; the full report keeps all three losses and the no-Gemma sensitivity result visible.
Optimizations
- Hand-written scanners replace general regex machinery for recognized production tokenizer patterns, with a table-driven ASCII path and exact Unicode fallback.
- Split, ByteLevel transformation, and BPE are fused so raw bytes map directly to initial token IDs and cache hits skip ByteLevel work entirely.
- BPE uses precomputed flat lookups: byte-to-token IDs, all 65,536 initial byte pairs, open-addressed merge ranks, and CSR merge adjacency.
- Pre-tokenized text stays in one contiguous buffer with byte ranges instead of allocating a string for every piece.
- Thread-local and shared caches are amortized across whole chunks, while a fixed Rayon pool keeps worker caches warm and balances uneven BPE work.
- Non-ByteLevel tokenizers split only at vocabulary-proven unbridgeable byte pairs; byte-fallback models stay on the normal exact merge path.
- Opt-in
.tkzsidecars cache validated native construction data. Across twelve models and thirteen hosts, direct.tkzloads are 1.55× faster than Snaptokens JSON and artifacts are 24.9% smaller by geometric mean.
Benchmarks
The 2026-07-31 suite uses DeepSeek R1, Gemma 3, GLM 4.7, GPT-2, GPT-OSS, Llama 3, MiniMax M2.1, Mistral Nemo, Nemotron 3, Phi-4 mini, Qwen 2.5, and Qwen 3. It runs five batch and input shapes on Apple M2 plus twelve four-core Modal placements across AWS, GCP, and OCI.
Every timed engine first passes Hugging Face token-ID parity. Gigatoken and Snaptokens cover all twelve models; upstream fastokens covers nine. Flat-ragged engines are compared only with the same flat-ragged output contract, nested engines only with nested output.
tokenizer.json and .tkz produce the same steady-state Snaptokens encoder, so the format comparison measures fresh load, first encode, disk size, and memory rather than drawing two redundant throughput bars. TKZ loads 1.55× faster than Snaptokens JSON, but is still 1.34× slower than Hugging Face JSON in this suite.
The full benchmark report contains the per-host, per-model, per-shape, load, footprint, variability, correctness, and limitation analysis. The complete accepted evidence includes every raw round, hardware capture, hash, summary, and an empty failure ledger.
Install
Install the Python package:
Or add the Rust crate:
[]
= "0.2"
Use
Python:
=
= .
Pass tkz_cache=True to create and reuse the adjacent .tkz file. A .tkz
path loads directly without that flag.
To replace a Transformers v4 or v5 fast-tokenizer backend:
Call snaptokens.unpatch_transformers() to restore the original backend.
Rust:
use ;
use Tokenizer;
/// Loads one local tokenizer and encodes a prompt.
Download the model yourself and pass the local JSON path. snaptokens never tries to infer or download a Hugging Face model.
Optional .tkz cache
let tokenizer = from_file_with_tkz_cache?;
The first enabled load atomically writes tokenizer.tkz; later loads validate and reuse it. Passing that .tkz path loads it directly without creating another copy. Tokenizer::from_file remains JSON-only and never writes a cache.
Scope
Snaptokens is only built for inference on BPE tokenizers, and to do just that, well.
Real-tokenizer validation covers GPT-2, GPT-OSS, DeepSeek, MiniMax, Qwen and Qwen3.5, GLM, Nemotron, Mistral, Gemma 4, Phi-4, and Kimi K2.5 families, including added-token flags and byte fallback where present.
Credits
Licensed under Apache-2.0.