token-estimate

Estimate LLM token counts from text without loading a tokenizer.

Real tokenizers need a multi-megabyte vocabulary and a non-trivial dependency tree. That is right when you need an exact count, and wrong when you only need to answer "will this roughly fit in the context window" — the common case in CLI tooling, log processing and pre-flight budget checks.

use token_estimate::{estimate, estimate_auto, fits, Profile};

let n = estimate("The quick brown fox jumps over the lazy dog.", Profile::Prose);
assert!(fits("short input", 100, Profile::Prose));

// Profile inferred from punctuation density
let _ = estimate_auto("if (a == b) { c[d] = e; }");

Accuracy, stated honestly

For ordinary English prose this lands within roughly 10-15% of a BPE tokenizer. It degrades predictably:

If you need an exact count, use a real tokenizer. This is for budgeting.

Use estimate_with_margin when overflowing a context window costs more than sending less.

License

MIT