Skip to main content

estimate_tokens

Function estimate_tokens 

Source
pub fn estimate_tokens(text: &str) -> usize
Expand description

Estimate the number of LLM tokens in text.

Uses the simple heuristic of ceil(bytes / 4).