Skip to main content

estimate_tokens

kernel::bench

Function estimate_tokens 

Source 
pub fn estimate_tokens(text: &str) -> i64
Expand description

Tokens a reply of text is worth, when nothing counted them: its characters over [CHARS_PER_TOKEN], and never zero for text that exists.