Skip to main content

Crate prompt_token_counter

Crate prompt_token_counter 

Source
Expand description

§prompt-token-counter

Approximate token counts for LLM messages, system prompts, and tools.

Most agent code needs to answer: “Will this request fit in the model’s context window?” The honest answers are vendor-specific (tiktoken for OpenAI, Anthropic’s counts API for Claude, sentencepiece for Llama). The cheap-and-acceptable answer is chars/4, which is within ~15% for English prose.

TokenCounter wraps either approach behind one API.

§Quick example

use prompt_token_counter::TokenCounter;
use serde_json::json;

let counter = TokenCounter::new();
assert_eq!(counter.count("Hello world"), 3); // 11/4 ceil

let messages = vec![
    json!({"role": "system", "content": "You are helpful."}),
    json!({"role": "user", "content": [{"type": "text", "text": "hi"}]}),
];
let _ = counter.count_messages(&messages);

§BYO tokenizer

use prompt_token_counter::TokenCounter;
// Pretend tokenizer: one token per char.
let counter = TokenCounter::new().with_tokenizer(|s: &str| s.chars().count());
assert_eq!(counter.count("hello"), 5);

§Budget check

use prompt_token_counter::{TokenCounter, FitsRequest, SystemSpec};
use serde_json::json;

let counter = TokenCounter::new();
let result = counter.fits(FitsRequest {
    context_window: 200_000,
    system: Some(SystemSpec::Text("...".into())),
    messages: &[json!({"role": "user", "content": "..."})],
    tools: &[],
    reserved_output: 4096,
});
assert!(result.fits);

Structs§

BudgetResult
Outcome of TokenCounter::fits.
FitsRequest
Named-args bundle for TokenCounter::fits.
TokenCounter
Approximate token counts for LLM messages, system prompts, and tools.

Enums§

SystemSpec
System-prompt shape passed to TokenCounter::fits.

Functions§

chars_per_4_estimate
Default token estimate: ceil(len / 4). Within ~15% for English prose.