Skip to main content

count_message_tokens

Function count_message_tokens 

Source
pub fn count_message_tokens(
    tokenizer: &dyn Tokenizer,
    message: &Message,
) -> usize
Expand description

Sum the token counts of a message’s text content (text and reasoning content items) using tokenizer.