tokenx-rs

Rust port of tokenx by Johann Schopplich

Fast token count estimation for LLMs at 96% accuracy without a full tokenizer.

Why?

• No vocabulary files: Full tokenizers like tiktoken require 2-4MB of BPE vocabulary data
• Zero dependencies: No regex, no allocations — just a single-pass character scanner
• Fast: ~8x faster than the original Node.js implementation on Latin text
• Accurate enough: 96% accuracy is sufficient for token budget estimation and streaming display
• Universal: Works across all LLM providers (OpenAI, Anthropic, Google, etc.)

Installation

[dependencies]
tokenx-rs = "0.1"

Usage

use tokenx_rs::estimate_token_count;

let tokens = estimate_token_count("Hello, world!");
println!("Estimated tokens: {}", tokens);

Check token limits

use tokenx_rs::is_within_token_limit;

if is_within_token_limit(prompt, 4096) {
    // safe to send
}

Split text into chunks

use tokenx_rs::split_by_tokens;

let chunks = split_by_tokens(long_text, 1000);

How it works

The estimator makes a single pass over the input string, classifying each character and grouping runs of similar characters into segments. Each segment is scored by type:

Language-specific diacritics (German, French, Spanish) adjust the chars_per_token ratio for more accurate estimates on non-English text.

There is no regex, no string splitting allocation, and no runtime dependencies.

Performance

Benchmarked against the original tokenx Node.js library on the same machine (Apple Silicon, Node v20, rustc 1.91):

Run benchmarks yourself with cargo bench.

Accuracy

Accuracy benchmarks from the original tokenx project (compared against tiktoken cl100k_base):

License

MIT