Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Token counting utility using character-based estimation Uses ~4 characters per token as a reasonable approximation

Implementations

impl Tokenizer

pub fn new(model_name: &str) -> Self

Create a new tokenizer for the given model

pub fn count_tokens(&self, text: &str) -> Result<usize>

Count tokens in a single text string (~4 chars per token)

pub fn count_chat_tokens(&self, messages: &[(String, String)]) -> Result<usize>

Count tokens in a chat message format

pub fn get_max_tokens(&self) -> usize

Get the maximum tokens for a model

pub fn remaining_tokens(&self, used_tokens: usize) -> usize

Calculate remaining tokens in context window