Skip to main content

Tokenizer

token_count::tokenizers

Trait Tokenizer 

Source 
pub trait Tokenizer: Send + Sync {
    // Required methods
    fn count_tokens(&self, text: &str) -> Result<usize>;
    fn get_model_info(&self) -> ModelInfo;
}
Expand description

Trait for tokenizing text with a specific model

Required Methods§

Source

fn count_tokens(&self, text: &str) -> Result<usize>

Count the number of tokens in the given text

Source

fn get_model_info(&self) -> ModelInfo

Get information about the model

Implementors§