Struct TokenCounter 

pub struct TokenCounter {}

Token counter for multiple LLM models

Provides unified interface for counting tokens across different models. Uses model-specific tokenizers internally.

Implementations

impl TokenCounter

pub fn new() -> Self

Create a new token counter

pub fn count(&self, text: &str, model: ModelType) -> TokenInfo

Count tokens for the given text and model

Arguments

• text - The text to tokenize
• model - The model type to use for tokenization

Returns

TokenInfo containing count, IDs, and decoded tokens

pub fn count_all(&self, text: &str) -> HashMap<ModelType, TokenInfo>

Count tokens for all supported models

pub fn count_primary_models(&self, text: &str) -> HashMap<ModelType, TokenInfo>

Count tokens for the 4 primary models (OpenAI, Claude, Gemini, Other) as required by the token efficiency display feature.

Returns counts for:

• OpenAI (GPT-4o)
• Claude (Sonnet 4)
• Gemini (Gemini 3)
• Other (generic model)

pub fn summary(&self, text: &str) -> String

Get a summary of token counts for all models

Trait Implementations

impl Default for TokenCounter

fn default() -> Self

Returns the “default value” for a type.