Struct TokenCounter 

pub struct TokenCounter { /* private fields */ }

Main tokenizer interface for accurate token counting

Implementations

impl TokenCounter

pub fn new(config: TokenizerConfig) -> Result<Self>

Create a new token counter with the specified configuration

pub fn default() -> Result<Self>

Create a new token counter with default configuration (GPT-4)

pub fn global() -> &'static TokenCounter

Get a reference to the shared global token counter instance This is highly optimized and avoids re-initialization costs

pub fn count_tokens(&self, content: &str) -> Result<usize>

Count tokens in the given text content

pub fn count_tokens_batch(&self, contents: &[&str]) -> Result<usize>

Count tokens in multiple content strings and return the total

pub fn estimate_file_tokens( &self, content: &str, file_path: &Path, ) -> Result<usize>

Estimate tokens for a file based on its content and metadata

pub fn fits_budget(&self, content: &str) -> Result<bool>

Check if content fits within the token budget

pub fn remaining_budget(&self, used_tokens: usize) -> Option<usize>

Calculate remaining budget after accounting for content

pub fn chunk_content( &self, content: &str, chunk_size: usize, ) -> Result<Vec<String>>

Split content into chunks that fit within a token limit

pub fn config(&self) -> &TokenizerConfig

Get the current tokenizer configuration

pub fn set_token_budget(&mut self, budget: Option<usize>)

Update the token budget

Trait Implementations

impl Debug for TokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.