Skip to main content

estimate_tokens

matrixcode_core::compress

Function estimate_tokens 

Source 
pub fn estimate_tokens(message: &Message) -> u32
Expand description

Estimate token count for a message (improved approximation). For mixed content (code, Chinese, English), use a weighted estimate:

  • ASCII chars: ~4 chars per token (0.25 tokens/char)
  • Non-ASCII (Chinese, etc): ~1.5 chars per token (0.67 tokens/char)
  • JSON/structured data: typically more tokens per char