Struct TokenCounter 

pub struct TokenCounter { /* private fields */ }

Token counter backed by tiktoken cl100k_base BPE encoding.

Estimates how many tokens a piece of text or a full Message will consume when sent to an LLM API. Uses a process-scoped OnceLock so BPE data is loaded once and shared across all instances.

Falls back to a chars/4 heuristic when tiktoken init fails or when the input exceeds MAX_INPUT_LEN bytes (64 KiB).

Examples

use zeph_memory::TokenCounter;

let counter = TokenCounter::new();
let n = counter.count_tokens("Hello, world!");
assert!(n > 0);

Implementations

impl TokenCounter

pub fn new() -> Self

Create a new counter. Falls back to chars/4 if tiktoken init fails.

BPE data is loaded once and cached in a OnceLock for the process lifetime.

pub fn count_tokens(&self, text: &str) -> usize

Count tokens in text. Uses cache, falls back to heuristic.

Inputs exceeding 64 KiB bypass BPE and use chars/4 without caching to avoid CPU amplification from oversized inputs.

pub fn count_message_tokens(&self, msg: &Message) -> usize

Estimate token count for a message the way the LLM API will see it.

When structured parts exist, counts from parts matching the API payload structure. Falls back to content (flattened text) when parts is empty.

pub fn count_tool_schema_tokens(&self, schema: &Value) -> usize

Count tokens for an OpenAI tool/function schema JSON value.

Trait Implementations

impl Default for TokenCounter

fn default() -> Self

Returns the “default value” for a type.