Struct TokenCounter

pub struct TokenCounter { /* private fields */ }

Token counter with caching support

Implementations

impl TokenCounter

pub fn new() -> Result<Self>

Create a new token counter with cl100k_base encoding (GPT-4)

pub fn count_tokens(&self, text: &str) -> Result<usize>

Count tokens in a single text

pub fn count_tokens_parallel(&self, texts: &[String]) -> Result<Vec<usize>>

Count tokens in multiple texts in parallel

pub fn count_file_tokens( &self, content: &str, path: &str, ) -> Result<FileTokenCount>

Count tokens for a file’s content with metadata

pub fn estimate_total_tokens( &self, files: &[(String, String)], ) -> Result<TotalTokenEstimate>

Estimate tokens for multiple files

Trait Implementations

impl Default for TokenCounter

fn default() -> Self

Returns the “default value” for a type.