Expand description
§prompt-token-counter
Approximate token counts for LLM messages, system prompts, and tools.
Most agent code needs to answer: “Will this request fit in the model’s
context window?” The honest answers are vendor-specific (tiktoken for
OpenAI, Anthropic’s counts API for Claude, sentencepiece for Llama).
The cheap-and-acceptable answer is chars/4, which is within ~15% for
English prose.
TokenCounter wraps either approach behind one API.
§Quick example
use prompt_token_counter::TokenCounter;
use serde_json::json;
let counter = TokenCounter::new();
assert_eq!(counter.count("Hello world"), 3); // 11/4 ceil
let messages = vec![
json!({"role": "system", "content": "You are helpful."}),
json!({"role": "user", "content": [{"type": "text", "text": "hi"}]}),
];
let _ = counter.count_messages(&messages);§BYO tokenizer
use prompt_token_counter::TokenCounter;
// Pretend tokenizer: one token per char.
let counter = TokenCounter::new().with_tokenizer(|s: &str| s.chars().count());
assert_eq!(counter.count("hello"), 5);§Budget check
use prompt_token_counter::{TokenCounter, FitsRequest, SystemSpec};
use serde_json::json;
let counter = TokenCounter::new();
let result = counter.fits(FitsRequest {
context_window: 200_000,
system: Some(SystemSpec::Text("...".into())),
messages: &[json!({"role": "user", "content": "..."})],
tools: &[],
reserved_output: 4096,
});
assert!(result.fits);Structs§
- Budget
Result - Outcome of
TokenCounter::fits. - Fits
Request - Named-args bundle for
TokenCounter::fits. - Token
Counter - Approximate token counts for LLM messages, system prompts, and tools.
Enums§
- System
Spec - System-prompt shape passed to
TokenCounter::fits.
Functions§
- chars_
per_ 4_ estimate - Default token estimate:
ceil(len / 4). Within ~15% for English prose.