Skip to main content

estimate_tokens

Function estimate_tokens 

Source
pub fn estimate_tokens(text: &str) -> usize
Expand description

Real token count via the cl100k_base BPE tokenizer (GPT-4-class; see cli-contract.md §8). A deterministic function of the text, so byte stability is unaffected. The bundled encoding is parsed once and cached.