Struct TokenCounter 

pub struct TokenCounter { /* private fields */ }

Token counter for estimating token usage

Implementations

impl TokenCounter

pub fn new() -> Self

Create a new token counter

pub fn count_tokens_openai(&self, content: &str, model: &str) -> usize

Count tokens for OpenAI models using estimation

This uses a heuristic-based approach:

• Average English word is ~4.7 characters
• Average token is ~4 characters
• Special tokens and formatting add overhead

pub fn count(&self, content: &str, model: &str) -> Result<usize, ProviderError>

Count tokens for content (unified interface)

This method provides a unified interface for token counting across providers. It returns a Result type for better error handling.

pub fn clear_cache(&self)

Clear the token count cache

pub fn cache_size(&self) -> usize

Get cache size

Trait Implementations

impl Default for TokenCounter

fn default() -> Self

Returns the “default value” for a type.

impl TokenCounterTrait for TokenCounter

fn count_tokens( &self, content: &str, model: &str, ) -> Result<usize, ProviderError>

Count tokens for content in a specific model

fn clear_cache(&self)

Clear the token count cache

fn cache_size(&self) -> usize

Get cache size