Skip to main content

estimate_tokens

Function estimate_tokens 

Source
pub fn estimate_tokens(text: &str) -> u32
Expand description

Rough token estimate for already-extracted text. Owner: parser agent.

ceil(chars / 4) — i.e. (n + 3) / 4, expressed via div_ceil.