char-token-est

Tokenless token-count estimator for LLM prompts. ~10% accurate on typical prompts, fast, zero deps. Use when a real BPE tokenizer is too heavy (routing, budget gates, log lines, progress bars).

Usage

use char_token_est::{estimate, Family};

let n = estimate("The quick brown fox jumps over the lazy dog.", Family::Gpt);
println!("~{n} tokens");

Or supply your own ratio:

use char_token_est::estimate_with_ratio;
let n = estimate_with_ratio("...", 4.0);

Calibration

Calibration is best-effort on English + code + JSON. Pure non-Latin input deviates further; use a real tokenizer for billing.

License

MIT or Apache-2.0.