code2prompt_core::tokenizer

Function count_tokens

Source 
pub fn count_tokens(rendered: &str, tokenizer_type: &TokenizerType) -> usize
Expand description

Counts the tokens in the provided text using the specified tokenizer type.

§Arguments

  • rendered - The text to count tokens in
  • tokenizer_type - The tokenizer encoding to use

§Returns

  • usize - The number of tokens in the text