Struct TokenCounter 

pub struct TokenCounter { /* private fields */ }

Implementations

impl TokenCounter

pub fn new() -> Self

Create a new counter. Falls back to chars/4 if tiktoken init fails.

pub fn count_tokens(&self, text: &str) -> usize

Count tokens in text. Uses cache, falls back to heuristic.

Inputs exceeding 64 KiB bypass BPE and use chars/4 without caching to avoid CPU amplification from oversized inputs.

pub fn count_message_tokens(&self, msg: &Message) -> usize

Estimate token count for a message the way the LLM API will see it.

When structured parts exist, counts from parts matching the API payload structure. Falls back to content (flattened text) when parts is empty.

pub fn count_tool_schema_tokens(&self, schema: &Value) -> usize

Count tokens for an OpenAI tool/function schema JSON value.

Trait Implementations

impl Default for TokenCounter

fn default() -> Self

Returns the “default value” for a type.