toklen

A single-threaded, lightweight, and efficient token counter for BPE tokenizers.

toklen is a focused Rust library that counts tokens in a text—without allocating a token list, without decoding, and without generating token IDs. So it's faster.

It loads the standard tokenizer.json format used by HuggingFace tokenizers. Same config, zero conversion—drop it in and start counting.

It's single-threaded by choice — no Async, no Rayon, no thread pools. Despite this, it is thread-safe: all internal state is thread-local, so you can safely share the tokenizer across threads.

Usage

Add to your Cargo.toml:

[dependencies]
toklen = "0.1"

Rust code:

use toklen::Tokenizer;

// Load a standard tokenizer.json (e.g. from a HuggingFace model).
let json = std::fs::read_to_string("tokenizer.json").unwrap();
let tokenizer = Tokenizer::from_json(&json).unwrap();

// Count tokens.
let count = tokenizer.encode_len("Hello, world!").unwrap();
println!("Token count: {count}");

// On failure, encode_len returns Err(estimate) —— input.len() / 4
// as a rough fallback that is always safe for progress reporting.

For CLI tools or WASM builds, embed the JSON at compile time:

const TOKENIZER_JSON: &[u8] = include_bytes!("tokenizer.json");

fn main() {
    let tokenizer = toklen::Tokenizer::from_json(TOKENIZER_JSON).unwrap();
    let count = tokenizer.encode_len("Some text to count").unwrap();
    println!("{count} tokens");
}

API

The encode_len method never panics on valid UTF-8 input. Errors bubble up from underlying regex or normalization steps; the fallback estimate (text.len() / 4) lets callers degrade gracefully.

Performance

Its single-threaded throughput is on par with multi-threaded fastokens and roughly 5× faster than tokenizers on equivalent hardware.

See /bench for the benchmark suite and raw numbers.

Note:

• toklen is thread-safe and designed to be stored as a global static resource (e.g., with LazyLock), allowing the tokenizer configuration to be reused across calls without re-initialization overhead.
• The first call to encode_len() incurs cold-start overhead (lazy initialization of internal caches). Subsequent calls are stable and fast—this behavior is consistent with other tokenizer implementations.

How it works

input text
  → added-token split (Aho–Corasick)
  → normalization (NFC, sequence)
  → pre-tokenization (ByteLevel, Split, regex)
  → BPE merge counting (priority-queue merge loop)
  → token count

Supported tokenizer.json subset

License

Licensed under either of Apache License 2.0 or MIT License, at your option.