What is splintr?
Splintr loads a tokenizer from four sources and dispatches it to four backends, all behind a single AnyTokenizer type — the calling code never changes with the vocabulary:
| Source | Loads | Backends |
|---|---|---|
Bundled (from_pretrained) |
13 vocabularies compiled in — load by name, no file needed | byte-level BPE, SPM-BPE |
tokenizer.json (from_json) |
Any HuggingFace file — normalizers, pre-tokenizers, decoders | byte-level BPE, Unigram, WordPiece |
Raw .tiktoken (Tokenizer(path, …)) |
A bare base64(bytes) rank file, with the pattern you supply |
byte-level BPE |
GGUF vocab (from_gguf_vocab) |
The tokenizer.ggml.* keys, parsed by your GGUF loader |
byte-level BPE, SPM-BPE, Unigram, WordPiece |
Correctness is differential: every family is fuzzed id-for-id against its reference implementation using strings built from each vocabulary's own added and special tokens. See CONTRIBUTING.md for how that is established.
Why it exists
Tokenization sits on the hot path of every LLM application — prompts, training corpora, RAG chunks, token counting for billing. Python-based tokenizers cannot use all your cores, so batch preprocessing turns into wall-clock latency. The usual escape is one library per format — tiktoken, sentencepiece, tokenizers — three dependencies, three APIs, no common handle, and no answer at all for a GGUF vocabulary. Splintr's answer is one handle over every format, at Rust speed, with reference implementations as the correctness oracle.
Performance
Batch encoding parallelizes across texts, which is where the gap is widest — and it widens with batch size, as the fixed cost of spinning up the pool is amortized over more work:
Single texts stay on the sequential path, and still lead across every content type:
Call it ~20x tiktoken on batches, ~5x on single texts. The ballpark holds across machines; the exact figure does not, since absolute throughput moves with hardware, CPU architecture and the versions compared against.
The tables below are a separate, more recent run than the charts — AMD Ryzen 9 5900X (24 cores, Linux), CPython 3.12, tiktoken 0.13.0, HuggingFace tokenizers 0.23.1, gigatoken 0.10.0, medians of three interleaved rounds. Where they disagree with the charts, which were plotted on different hardware against older versions, the tables are the measured ones.
Against tiktoken, each library loading cl100k_base through its own loader:
| Batch | Splintr | tiktoken | vs tiktoken |
|---|---|---|---|
| 1,000 texts | 104.8 MB/s | 5.1 MB/s | 20.4x |
| 500 texts | 93.7 MB/s | 4.5 MB/s | 21.0x |
| 100 texts | 56.0 MB/s | 2.3 MB/s | 24.8x |
| Single text | 1.27 ms | 6.36 ms | 5.0x |
Against the other Rust tokenizers, every engine loading the same tokenizer.json — the only way to compare with no loader asymmetry to argue about:
| Batch | Splintr | HF tokenizers | gigatoken |
|---|---|---|---|
| 1,000 texts | 89.0 MB/s | 29.5 MB/s | 107.9 MB/s |
| 500 texts | 91.4 MB/s | 29.6 MB/s | 124.2 MB/s |
| 100 texts | 55.3 MB/s | 22.4 MB/s | 124.3 MB/s |
| Single text | 1.55 ms | 20.89 ms | 1.77 ms |
Read that second table as it stands: gigatoken is faster on batch for the OpenAI vocabularies, splintr is faster on single texts and on load, and the two are level on batch for Qwen and GLM. Splintr's case is not that it wins every row — it is one handle over bundled, HuggingFace and GGUF vocabularies, across four backends, verified id-for-id against the reference implementations.
Getting current numbers
The tables above are a dated snapshot against the versions named. For where splintr stands against other tokenizers today, two ways, both running the same harness:
- View the latest perf run — every run publishes its full report to the run summary: the hardware and library versions it used, the id-parity check it had to pass before timing anything, and every vocabulary and corpus, including the ones not shown above.
- Run it yourself —
gh workflow run perf.ymlon a fork, or the scripts directly (.github/scripts/perf_bench.pyandperf_report.py) on your own machine. It builds the checkout rather than installing a release, so a branch can be measured before it ships, and it refuses to report timings for engines that disagree on ids.
benchmarks/benchmark_batch.py is the standalone script behind the charts. See docs/benchmarks.md for per-content-type latency, methodology and the PCRE2 backend.
Quick Start
Python
# Load a pretrained vocabulary
= # OpenAI GPT-4/3.5
# tokenizer = Tokenizer.from_pretrained("llama3") # Meta Llama 3 family
# tokenizer = Tokenizer.from_pretrained("deepseek_v3") # DeepSeek V3/R1
# tokenizer = Tokenizer.from_pretrained("qwen3") # Qwen 2/3, Baichuan-M2
# tokenizer = Tokenizer.from_pretrained("glm4") # GLM-4/4.5
# tokenizer = Tokenizer.from_pretrained("gpt-oss") # OpenAI gpt-oss
# Encode and decode
=
=
# Batch encode (parallel across texts)
=
See the API Guide for complete documentation and examples.
Rust
use from_pretrained;
let tokenizer = from_pretrained?;
let tokens = tokenizer.encode;
let batch_tokens = tokenizer.encode_batch;
let text = tokenizer.decode?;
See the API Guide and docs.rs for complete documentation.
Key Features
- Four backends, one handle — Byte-level/raw BPE, SentencePiece BPE, Unigram, and WordPiece all load as
AnyTokenizer, so calling code stays the same whichever vocabulary you use - Parallel batch encoding — Rayon across texts; sequential for single texts based on empirical benchmarking
- Four loading sources — 13 bundled vocabularies by name, any HuggingFace
tokenizer.json, a raw.tiktokenfile, or a GGUF vocabulary - Streaming decoder — Real-time LLM output with proper UTF-8 boundary handling; one decoder per tokenizer (guide)
- 54 agent tokens — ChatML, thinking, ReAct, tool-calling and RAG citation markers, on every bundled vocabulary (docs)
- Special-token policy —
encode_ordinary/encode_allowed_specialso untrusted text cannot forge a control token - Cross-platform — Python bindings via PyO3 (Linux, macOS, Windows), CPython 3.10+; native Rust library
Vocabularies
Four ways to load one, differing only in where the vocabulary data comes from. Routes 1, 2 and 4 return the same AnyTokenizer handle, so calling code never changes with the vocabulary; route 3 returns the concrete Tokenizer (byte-level BPE), since a bare rank file states no backend to dispatch on.
| # | Source | Call | Use it when |
|---|---|---|---|
| 1 | Bundled | Tokenizer.from_pretrained("qwen3") |
The model is one of the 13 below — no file, no download, no network |
| 2 | HuggingFace tokenizer.json |
from_json("tokenizer.json") |
Any other model; the file's own normalizer, pre-tokenizer and decoder are honoured |
| 3 | Raw .tiktoken |
Tokenizer("vocab.tiktoken", PATTERN) |
You have a bare rank file and will supply the pattern and special tokens yourself |
| 4 | GGUF vocabulary | splintr::from_gguf_vocab(…) (Rust only) |
You already parsed a GGUF and hold its tokenizer.ggml.* keys |
1. Bundled vocabularies
| Name | Used by | base_vocab_size |
|---|---|---|
cl100k_base |
GPT-4, GPT-3.5-turbo | 100,277 |
o200k_base |
GPT-4o | 200,019 |
gpt-oss |
OpenAI gpt-oss | 200,019 |
llama3 |
Llama 3, 3.1, 3.2, 3.3 | 128,256 |
qwen3 |
Qwen 2, Qwen 3, Baichuan-M2 | 151,669 |
glm4 |
GLM-4, GLM-4.5 | 151,365 |
kimi_k2 |
Kimi K2, K2.5, K2.6, K2.7 | 163,840 |
kimi_k3 |
Kimi K3 | 163,840 |
deepseek_v3 |
DeepSeek V3, DeepSeek R1 | 128,815 |
mistral_v1 |
Mistral 7B v0.1/v0.2 | 32,000 |
mistral_v2 |
Mistral 7B v0.3, Codestral | 32,768 |
mistral_v3 |
Mistral NeMo, Large 2, Pixtral | 131,072 |
whisper |
OpenAI Whisper multilingual | 51,865–51,866 |
Each also answers to the aliases you would expect (qwen, qwen2.5, glm-4.5, llama3.1, deepseek-v3, …); bare kimi resolves to K2, which covers seven published repos to K3's one.
The list is short for one reason and it is not capability: the whole set is ~23 MB of embedded data, so the bar for adding one is that it covers a family people reach for. Each sits behind a vocab-* cargo feature (all on by default, all in the Python wheel), so a Rust build can keep only what it needs.
What a bundled vocabulary adds over the same vocabulary loaded from a file:
| 54 agent tokens | ChatML, thinking, ReAct, tool-calling and RAG citation markers, appended above every original id. Whisper is the exception — it carries its own 1,608 standard tokens instead. |
base_vocab_size |
Where the model's own ids end and splintr's begin |
| Model ids win on collision | Where a vocabulary already ships one of those names — Qwen's <|im_start|>, GLM's <|system|> — it resolves to the model's own id, so a chat template still encodes what the checkpoint was trained on |
2. Any HuggingFace tokenizer.json
=
Reads the file's real configuration — normalizers, the multi-stage pre-tokenizer, BPE merge order, added_tokens, the decoder chain — and dispatches to byte-level BPE, Unigram or WordPiece as model.type says. Verified id-for-id against HuggingFace tokenizers across GPT-2, RoBERTa, BART, Qwen, Whisper, T5, Albert, XLNet, BERT, DistilBERT, Falcon, StarCoder2, DeepSeek-Coder and GPT-NeoX. It raises rather than approximating a config it does not fully support, so wrong-ids-with-no-signal is not a possible outcome.
3. A raw .tiktoken file
=
=
A .tiktoken file is base64(token bytes) rank per line and carries nothing else — no pattern, no special tokens, no decoder chain — so you supply the pattern and any special tokens. That is the whole difference from routes 1 and 2, which read those from the vocabulary itself, and the reason this one returns a Tokenizer rather than an AnyTokenizer. Ids are identical: loading vocabs/cl100k_base.tiktoken this way encodes exactly as from_pretrained("cl100k_base") does. In Rust: Tokenizer::from_file(path, pattern, special_tokens), or from_bytes for a vocabulary you already hold.
4. A GGUF vocabulary (Rust only)
Splintr never opens a GGUF container — parsing one is the model runtime's job — so the caller fills a GgufVocab from the file's tokenizer.ggml.* keys and hands it to splintr::from_gguf_vocab, which returns the same AnyTokenizer.
Which one do I use?
One question decides it: does the vocabulary already exist, or are you choosing one?
| Situation | Use | Why |
|---|---|---|
| Inference, serving, token counting, fine-tuning | Whatever the model ships — 1 if bundled, else 2 | The ids must be the ones the model was trained on, or every embedding lookup is wrong |
| Training a new model | 1, a bundled vocabulary | A proven merge table plus 54 agent tokens already allocated at deterministic ids |
| Training a new model, own vocabulary or own markers | 2 or 3 | You own the merge table and the id layout; see Best Practices |
Two things to know before you size a model against a bundled vocabulary:
- Agent tokens sit above
base_vocab_size. A published checkpoint has no embedding rows for them, so never feed it an id at or above that number. - Training on one? Size to
vocab_size, notbase_vocab_size— that is what makes<\|think\|>,<\|plan\|>and<\|function\|>trainable tokens from the first step.
=
# 151723 — what splintr knows
# 151669 — what the checkpoint knows
The value there is not the vocabulary — you could pull Qwen's from HuggingFace. It is that a new model needs markers no published vocabulary contains, and the usual answer is hand-editing a tokenizer.json, choosing ids and hoping nothing collides. Splintr allocates them at the same offsets in every vocabulary, so a model trained on cl100k and one trained on Qwen agree on what <|think|> means. Splintr is a tokenizer runtime and does not train vocabularies.
See docs/vocabularies.md for per-vocabulary special-token lists, pre-tokenizer patterns, the feature flags and the GGUF example.
Streaming Decoder
For real-time LLM output where tokens arrive one at a time:
=
BPE tokens don't align with UTF-8 boundaries. A multi-byte character might split across tokens. The streaming decoder buffers incomplete sequences and only outputs complete characters. One decoder per tokenizer, built by that tokenizer, so "".join(chunks) + flush() equals decode(ids) for any vocabulary. See API Guide for details and best practices.
Special Tokens in Untrusted Text
A tokenizer that matches special tokens will promote text that spells a control token to that token's real id. <|im_start|> typed by a user becomes the same id the server emits — downstream, nothing can tell them apart. Encoding takes an explicit mode:
| Mode | Behaviour |
|---|---|
encode_with_special(text) / All |
Match every configured special token found in text |
encode_ordinary(text) / Ordinary |
Match none — special spellings stay ordinary content |
encode_allowed_special(text, allowed) / Allow |
Match only the named tokens; raise on any other |
All three are on every Python tokenizer type — Tokenizer, AnyTokenizer, SpmTokenizer, SentencePieceTokenizer, WordPieceTokenizer — alongside encode (model-ready with boundary template), encode_raw (content tokens only), and encode_batch.
=
=
# Default: literal control token becomes real control-token id
# Ordinary: never match special tokens
# Allow-list: reject anything outside it
See docs/special_tokens.md for detailed guidance and a guide to the token list per vocabulary.
How It Works
Pre-tokenization runs on regexr, a pure-Rust regex engine with JIT and SIMD, and special tokens are matched with Aho-Corasick in a single pass. Merging uses a linked list rather than a vector, so pathological inputs stay linear, with an LRU cache over repeated chunks and FxHashMap for rank lookups. Batches are encoded in parallel with Rayon; single texts stay sequential, which measures faster below roughly 1 MB.
The other three backends are real implementations of their algorithms, not approximations: SentencePiece Unigram uses Viterbi maximum-score segmentation, SentencePiece BPE merges by score, and WordPiece does greedy longest-match with the ## continuation prefix.
Contributing
Bug reports, feature suggestions and pull requests are welcome — see CONTRIBUTING.md for development setup, the checks CI runs, and how correctness is established against the reference tokenizers.
Acknowledgments
Splintr builds on concepts from tiktoken, SentencePiece and tokenizers — which also serve as the reference implementations its output is checked against.
Citation
If you use Splintr in your research, please cite:
License
MIT — see LICENSE.