Skip to main content

truncate_to_token_limit

Function truncate_to_token_limit 

Source
pub fn truncate_to_token_limit(text: &str, max_tokens: usize) -> String
Expand description

Truncate string to approximate token limit

Returns a truncated string that fits within the approximate token limit. Tries to truncate at word boundaries when possible to avoid mid-word cuts.