Module tokens 

Token counting via tiktoken BPE tokenizer.

All token estimation goes through tiktoken’s cl100k_base encoding (GPT-4, GPT-3.5-turbo). BPE tokenizers are similar enough across providers that this gives reasonable accuracy for Anthropic, Gemini, and others.

Provider-reported exact token counts (from API responses) should always be preferred when available. This module is for pre-call budget estimation and offline token sizing where no provider response exists yet.

Constants

CHARS_PER_TOKEN

Approximate bytes per token used by the byte-length heuristic.

Functions

estimate_tokens

Count the number of tokens in text using tiktoken BPE.

format_token_count

Format a token count with a compact decimal suffix for transcript labels.

heuristic_tokens_from_byte_len

Approximate token count from a byte length without running the tokenizer.

heuristic_tokens_from_byte_len_u64

u64 twin of heuristic_tokens_from_byte_len for lifetime byte counters that never narrow to usize.

truncate_to_tokens

Truncate text to at most max_tokens tokens.