Skip to main content

estimate_token_count

Function estimate_token_count 

Source
pub fn estimate_token_count(text: &str) -> usize
Expand description

Count the number of tokens in text using tiktoken BPE.

Returns 0 for empty strings. Falls back to a character-based heuristic if the BPE tokenizer is unavailable.