estimate_tokens

Function estimate_tokens 

Source
pub fn estimate_tokens(text: &str) -> usize
Expand description

Estimates the number of tokens in a text string using a real tokenizer