Skip to main content

estimate_tokens

Function estimate_tokens 

Source
pub fn estimate_tokens(text: &str) -> u32
Expand description

Estimates the token count for a string.

This is a rough heuristic (approximately 4 characters per token for English). Use provider-reported token counts when available for accuracy.

§Arguments

  • text - The text to estimate tokens for

§Returns

Estimated token count (always at least 1 for non-empty text).