toklen 0.2.0

A single-threaded, lightweight, and efficient token counter.

toklen counts tokens in a text — no token-list allocation, no token IDs, no decoding — so it's fast and small.

It's single-threaded by choice — no async, no Rayon. Despite this, it is thread-safe: all internal state is thread-local, so you can safely share the tokenizer across threads.

It loads the standard tokenizer.json format used by HuggingFace tokenizers. Same config, zero conversion — drop it in and start counting.

Usage

Add to your Cargo.toml:

[dependencies]
toklen = "0.2"

Rust code:

use toklen::Tokenizer;

// Load a standard tokenizer.json (e.g. from a HuggingFace model).
let json = std::fs::read_to_string("tokenizer.json").unwrap();
let tokenizer = Tokenizer::from_json(&json).unwrap();

// Count tokens.
let count = tokenizer.encode_len("Hello, world!").unwrap();
println!("Token count: {count}");

// On failure, encode_len returns Err(estimate) — input.len() / 4,
// a rough fallback that is always safe for progress reporting.

For CLI tools or WASM builds, embed the JSON at compile time:

const TOKENIZER_JSON: &[u8] = include_bytes!("tokenizer.json");

fn main() {
    let tokenizer = toklen::Tokenizer::from_json(TOKENIZER_JSON).unwrap();
    let count = tokenizer.encode_len("Some text to count").unwrap();
    println!("{count} tokens");
}

API

encode_len never panics. Errors from the underlying regex or normalization steps are converted into Err(estimate) (text.len() / 4), so callers can always degrade gracefully.

Performance

toklen's single-threaded throughput is on par with multi-threaded fastokens and roughly 5× faster than tokenizers on equivalent hardware.

See /bench for the benchmark suite and raw numbers.

Note:

toklen is thread-safe and designed to be stored as a global resource (e.g. static LazyLock<_>), allowing the tokenizer configuration to be reused across calls without re-initialization overhead.

How it works

input text
  → added-token split (Aho–Corasick)
  → normalization (Sequence: NFC/NFD/…, Lowercase, StripAccents, Replace, …)
  → pre-tokenization (ByteLevel, Split regex, Whitespace, …)
  → model counting (BPE priority-queue merge loop, or WordLevel lookup)
  → token count

Supported tokenizer.json subset

toklen reads the tokenizer.json format produced by HuggingFace tokenizers.

The components and their semantics are documented in the tokenizers component reference. Configs that use an unsupported (or disabled) type return an error from Tokenizer::from_json.

added_tokens

Fully supported — added tokens are matched with Aho–Corasick (plus a memchr prefilter), and each match counts as exactly one token. Always compiled in (no feature gate).

normalizer

Note: BertNormalizer is not supported yet.

pre_tokenizer

model

Note: Unigram and WordPiece are not supported yet.

decoder / post_processor / other

Ignored — toklen only counts the tokens produced by the model, so decoding and post-processing never run. Counts therefore correspond to encode(text, add_special_tokens=False): special tokens inserted by a post-processor (e.g. [CLS]/[SEP]) are not included.

Cargo features

Everything above is enabled by default (default = ["full"]). Each component sits behind a cargo feature of the same name, so you can turn off whatever your tokenizer configs never use — fewer compiled variants shrink the library and tighten the dispatch match on the hot path:

[dependencies]
# For example, only what a GPT-style BPE tokenizer needs:
toklen = { version = "0.2", default-features = false, features = ["bpe", "byte_level", "split"] }

The meta features normalizers, pre_tokenizers, and models enable a whole group at once; full (the default) enables all three.

License

Licensed under either of Apache License 2.0 or MIT License, at your option.