Skip to main content

estimate_token_count

Function estimate_token_count 

Source
pub fn estimate_token_count(text: &str) -> usize
Expand description

Estimate token count from string (rough approximation)

Simple estimation: 1 token ≈ 4 characters. Returns a minimum of 1 for non-empty strings.