Struct ExactTokenCounter 

pub struct ExactTokenCounter { /* private fields */ }

Exact token counter with BPE tokenization

In production, this would use tiktoken-rs or tokenizers crate. This implementation provides a sophisticated approximation.

Implementations

impl ExactTokenCounter

pub fn new(config: ExactTokenConfig) -> Self

Create a new exact token counter

pub fn default_counter() -> Self

Create with default configuration

pub fn stats(&self) -> &Arc<TokenCacheStats>

Get cache statistics

Trait Implementations

impl TokenCounter for ExactTokenCounter

fn count(&self, text: &str) -> usize

Count tokens in text

fn count_for_model(&self, text: &str, model: TokenizerModel) -> usize

Count tokens with model hint

fn tokenize(&self, text: &str) -> Vec<u32>

Tokenize text (returns token IDs)

fn decode(&self, tokens: &[u32]) -> String

Decode tokens back to text

fn model(&self) -> TokenizerModel

Get the model being used

fn is_exact(&self) -> bool

Check if this counter uses exact tokenization