Struct TokenCounter 

pub struct TokenCounter { /* private fields */ }

Estimates token counts using the cl100k_base tokenizer (GPT-4 / Claude compatible). Accuracy is within ~5% across non-OpenAI providers; the budget is a soft guarantee.

Implementations

impl TokenCounter

pub fn new() -> Result<Self, PackError>

Create a new token counter backed by the cl100k_base encoding.

Errors

Returns PackError::Io if the tokenizer cannot be loaded (should only happen if the tiktoken-rs data files are missing).

pub fn count(&self, text: &str) -> usize

Count the number of tokens in a UTF-8 string.

pub fn count_bytes(&self, text: &[u8]) -> usize

Count the number of tokens in a byte slice, handling non-UTF-8 data gracefully via lossy conversion.