Struct TokenCounter 

pub struct TokenCounter { /* private fields */ }

Token counter using tiktoken

Uses cl100k_base encoding (GPT-4, GPT-3.5-turbo, text-embedding-ada-002)

Implementations

impl TokenCounter

pub fn new() -> Result<Self, TokenCounterError>

Create a new token counter with cl100k_base encoding

pub fn count(&self, text: &str) -> usize

Count tokens in a string

pub fn count_message(&self, role: &str, content: &str) -> usize

Count tokens for a chat message (includes role overhead)

Approximates the token overhead for chat formatting:

• Each message has ~4 tokens overhead (role, separators)
• System messages have additional overhead

pub fn count_messages(&self, messages: &[(String, String)]) -> usize

Count tokens for multiple messages

pub fn truncate(&self, text: &str, max_tokens: usize) -> (String, usize)

Truncate text to fit within a token limit

Returns the truncated text and the actual token count.

pub fn chunk(&self, text: &str, chunk_size: usize) -> Vec<String>

Split text into chunks of approximately equal token size

pub fn will_fit(&self, text: &str, budget: usize) -> bool

Estimate if text will fit within budget

Trait Implementations

impl Default for TokenCounter

fn default() -> Self

Returns the “default value” for a type.