Skip to main content

count_tokens

Function count_tokens 

Source
pub fn count_tokens(content: &str) -> usize
Expand description

Count tokens in a string using a simple heuristic. For code: ~4 characters per token For prose: ~1.3 words per token