Crate token_estimate 

Estimate LLM token counts from text without loading a tokenizer.

Real tokenizers (BPE, SentencePiece) need a vocabulary file of several megabytes and a non-trivial dependency tree. That is the right tool when you need an exact count. It is the wrong tool when you only need to answer “will this roughly fit in the context window”, which is the common case in CLI tooling, log processing, and pre-flight budget checks.

This crate gives a fast, dependency-free estimate and is explicit about how wrong it can be.

Accuracy, stated honestly

For ordinary English prose the character-based estimate is typically within about 10-15% of a BPE tokenizer. It degrades in predictable ways:

• Code and markup tokenize denser than prose (more punctuation, more short symbols), so the estimate tends to run low. Use Profile::Code.
• Non-Latin scripts often use one token per character or worse, so a chars/4 heuristic can underestimate severely. estimate detects non-ASCII density and adjusts, but treat CJK figures as a rough floor.
• Long runs of whitespace or repeated punctuation compress well and the estimate runs high.

If you need an exact count, use a real tokenizer. This is for budgeting.

Example

use token_estimate::{estimate, Profile};

let prose = "The quick brown fox jumps over the lazy dog.";
let n = estimate(prose, Profile::Prose);
assert!(n > 0);

// Code is denser, so the same byte count yields more tokens.
let code = "fn main() { let x: Vec<u8> = vec![1,2,3]; }";
assert!(estimate(code, Profile::Code) > estimate(code, Profile::Prose));

Enums

Profile

Text kind, which changes the characters-per-token assumption.

Functions

estimate

Estimated token count for text under profile.

estimate_auto

Estimate with the profile inferred from the text itself.

estimate_with_margin

A conservative upper bound, inflating the estimate by margin_pct.

fits

Whether text is estimated to fit in budget tokens.

infer_profile

Infer a Profile from punctuation density.

remaining

Remaining budget after text, saturating at zero.