Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Accurate token counter with fallback to estimation

The tokenizer supports caching to avoid re-computing token counts for the same content. This is particularly useful when processing files multiple times or across different operations.

Implementations

impl Tokenizer

pub fn new() -> Self

Create a new tokenizer with exact mode and caching enabled

pub fn estimation_only() -> Self

Create a tokenizer that only uses estimation (faster but less accurate)

pub fn without_cache() -> Self

Create a tokenizer without caching (useful for benchmarks or one-off counts)

pub fn count(&self, text: &str, model: TokenModel) -> u32

Count tokens for a specific model.

When caching is enabled, results are stored in a global cache keyed by content hash and model. This provides significant speedup for repeated tokenization of the same content.

Returns

The token count for the specified model. For OpenAI models (GPT-4o, GPT-4, etc.), this is exact via tiktoken. For other models, it’s a calibrated estimation.

pub fn count_all(&self, text: &str) -> TokenCounts

Count tokens for all supported models at once

Returns counts for representative models from each encoding family:

• o200k: GPT-5.x, GPT-4o, O1/O3/O4 (all use same tokenizer)
• cl100k: GPT-4, GPT-3.5-turbo (legacy, same tokenizer)
• Other vendors use estimation

pub fn most_efficient_model(&self, text: &str) -> (TokenModel, u32)

Estimate which model will have the lowest token count

pub fn truncate_to_budget<'a>( &self, text: &'a str, model: TokenModel, budget: u32, ) -> &'a str

Truncate text to fit within a token budget

pub fn exceeds_budget(&self, text: &str, model: TokenModel, budget: u32) -> bool

Check if text exceeds a token budget

Trait Implementations

impl Default for Tokenizer

fn default() -> Self

Returns the “default value” for a type.