Trait Tokenizer 

pub trait Tokenizer {
    // Required method
    fn count_tokens(&self, text: &str) -> u32;
}

Trait for counting tokens in text. Implement this to integrate with specific tokenizers (e.g., tiktoken, HuggingFace tokenizers).

Required Methods

fn count_tokens(&self, text: &str) -> u32

Count the number of tokens in the given text.

Implementors

impl Tokenizer for CharRatioTokenizer