Skip to main content

estimate_tokens

phi_core::context::token

Function estimate_tokens 

Source 
pub fn estimate_tokens(text: &str) -> usize
Expand description

Rough token estimate: ~4 chars per token for English text. See TokenCounter for pluggable alternatives.