Module estimation 

Token estimation for mixed-language text

Provides fast token count estimation supporting ASCII, CJK, and other Unicode. Used across the codebase for LLM request sizing, chunking, and content compression.

Structs

TokenEstimator

Token estimator for mixed-language text

Constants

MESSAGE_OVERHEAD_TOKENS

Tokens per message metadata overhead (role, separators, etc.)

SYMBOL_FACTOR

Tokens per ASCII symbol (punctuation) character (default: 0.5)

Functions

estimate_tokens

Estimate tokens using default estimator