Struct TokenEstimator 

pub struct TokenEstimator;

Import token estimation from ck-embed Simple token estimation for code and text This is a rough approximation since we don’t have access to the actual model tokenizer

Implementations

impl TokenEstimator

pub fn estimate_tokens(text: &str) -> usize

Estimate token count for text Based on empirical analysis of code and text tokenization:

• Code: ~4.2 characters per token
• Text: ~4.8 characters per token
• Average: ~4.5 characters per token

pub fn exceeds_limit(text: &str, max_tokens: usize) -> bool

Check if text exceeds token limit for a given model

pub fn get_model_limit(model_name: &str) -> usize

Get token limit for different models