Expand description
A single-threaded, lightweight, and efficient token counter.
toklen counts tokens in a text — no token-list allocation, no token IDs, no decoding — so it’s fast and small.
It’s single-threaded by choice — no async, no Rayon. Despite this, it is thread-safe: all internal state is thread-local, so you can safely share the tokenizer across threads.
It loads the standard tokenizer.json format used by HuggingFace tokenizers. Same config, zero conversion — drop it in and start counting.
§Usage
Add to your Cargo.toml:
[dependencies]
toklen = "0.2"Rust code:
use toklen::Tokenizer;
// Load a standard tokenizer.json (e.g. from a HuggingFace model).
let json = std::fs::read_to_string("tokenizer.json").unwrap();
let tokenizer = Tokenizer::from_json(&json).unwrap();
// Count tokens.
let count = tokenizer.encode_len("Hello, world!").unwrap();
println!("Token count: {count}");
// On failure, encode_len returns Err(estimate) — input.len() / 4,
// a rough fallback that is always safe for progress reporting.For CLI tools or WASM builds, embed the JSON at compile time:
const TOKENIZER_JSON: &[u8] = include_bytes!("tokenizer.json");
fn main() {
let tokenizer = toklen::Tokenizer::from_json(TOKENIZER_JSON).unwrap();
let count = tokenizer.encode_len("Some text to count").unwrap();
println!("{count} tokens");
}§API
| Method | Description |
|---|---|
Tokenizer::from_json(json) | Build a tokenizer from tokenizer.json bytes. |
tokenizer.encode_len(text) | Count tokens in text. Returns Ok(count) or Err(estimate). |
encode_len never panics. Errors from the underlying regex or normalization
steps are converted into Err(estimate) (text.len() / 4), so callers can
always degrade gracefully.
§Performance
toklen’s single-threaded throughput is on par with multi-threaded fastokens and roughly 5× faster than tokenizers on equivalent hardware.
See /bench for the benchmark suite and raw numbers.
Note:
toklen is thread-safe and designed to be stored as a global resource (e.g. static LazyLock<_>),
allowing the tokenizer configuration to be reused across calls without re-initialization overhead.
§How it works
input text
→ added-token split (Aho–Corasick)
→ normalization (Sequence: NFC/NFD/…, Lowercase, StripAccents, Replace, …)
→ pre-tokenization (ByteLevel, Split regex, Whitespace, …)
→ model counting (BPE priority-queue merge loop, or WordLevel lookup)
→ token count§Supported tokenizer.json subset
toklen reads the tokenizer.json format produced by HuggingFace
tokenizers.
The components and their semantics are documented in the
tokenizers component reference.
Configs that use an unsupported (or disabled) type return an error from
Tokenizer::from_json.
§added_tokens
Fully supported — added tokens are matched with Aho–Corasick (plus a memchr prefilter), and each match counts as exactly one token. Always compiled in (no feature gate).
§normalizer
| Type | Cargo feature |
|---|---|
| Sequence | always available |
| NFC | nfc |
| NFD | nfd |
| NFKC | nfkc |
| NFKD | nfkd |
| Lowercase | lowercase |
| StripAccents | strip_accents (pulls in icu_properties) |
| Strip | strip |
| Replace | replace |
| Prepend | prepend |
| ByteLevel | byte_level |
Note: BertNormalizer is not supported yet.
§pre_tokenizer
| Type | Cargo feature |
|---|---|
| Sequence | always available |
| ByteLevel | byte_level |
| Split | split (PCRE2 JIT + fancy_regex fallback) |
| Whitespace | whitespace |
| WhitespaceSplit | whitespace_split |
| Punctuation | punctuation |
| Metaspace | metaspace |
| Digits | digits |
| CharDelimiterSplit | char_delimiter_split |
§model
| Type | Cargo feature |
|---|---|
| BPE | bpe |
| WordLevel | word_level |
Note: Unigram and WordPiece are not supported yet.
§decoder / post_processor / other
Ignored — toklen only counts the tokens produced by the model, so decoding
and post-processing never run. Counts therefore correspond to
encode(text, add_special_tokens=False): special tokens inserted by a
post-processor (e.g. [CLS]/[SEP]) are not included.
§Cargo features
Everything above is enabled by default (default = ["full"]). Each component
sits behind a cargo feature of the same name, so you can turn off whatever your
tokenizer configs never use — fewer compiled variants shrink the library and
tighten the dispatch match on the hot path:
[dependencies]
# For example, only what a GPT-style BPE tokenizer needs:
toklen = { version = "0.2", default-features = false, features = ["bpe", "byte_level", "split"] }The meta features normalizers, pre_tokenizers, and models enable a whole
group at once; full (the default) enables all three.
§License
Licensed under either of Apache License 2.0 or MIT License, at your option.
Structs§
- Tokenizer
- A token-counting tokenizer backed by
tokenizer.json.