Struct HeuristicTokenCounter 

pub struct HeuristicTokenCounter { /* private fields */ }

A provider-agnostic TokenCounter that approximates token counts from UTF-8 byte lengths.

This is intended as a zero-dependency default. It is not a substitute for a tokenizer and will under- or over-count by up to ~30 % on real content, but it is monotonic in message size and stable across runs, which is enough for TokenWindowMemory to enforce a budget that trends with provider billing.

Strategy

For every text-bearing block (Text, reasoning text, tool-result text) the counter sums UTF-8 byte lengths (str::len, an O(1) call) and divides by bytes_per_token, rounded up. Bytes are used instead of Unicode scalars because the cost is O(1), modern BPE tokenizers operate on byte sequences, and per-message budgeting only needs the rough order of magnitude. For ASCII text bytes and characters coincide; for non-ASCII text the counter slightly over-estimates, which is the safe direction for a hard budget.

Tool calls are charged the JSON-serialised length of their ToolFunction payload. Each message is charged a flat per_message_overhead to model the per-turn role/separator tokens that providers add internally. Non-text blocks (images, audio, video, documents) are charged per_attachment_tokens each because their real cost is provider-specific and rarely text-derived.

Presets

The defaults match OpenAI’s published rule of thumb (~4 bytes per token, ~4 tokens of per-message overhead). HeuristicTokenCounter::anthropic uses a slightly denser ratio that better fits Claude’s tokenizer.

Example

use rig_memory::{HeuristicTokenCounter, TokenWindowMemory};

let policy = TokenWindowMemory::new(2_000, HeuristicTokenCounter::default());

Implementations

impl HeuristicTokenCounter

pub fn new( bytes_per_token: f32, per_message_overhead: usize, per_attachment_tokens: usize, ) -> Self

Create a counter with explicit parameters.

bytes_per_token is clamped to a minimum of 1.0 so the counter never panics or produces zero-cost messages on degenerate input.

pub fn openai() -> Self

Preset matching OpenAI’s chat-completion token rule of thumb.

Equivalent to HeuristicTokenCounter::default.

pub fn anthropic() -> Self

Preset tuned for Anthropic Claude’s tokenizer.

pub fn gemini() -> Self

Preset tuned for Google Gemini.

Trait Implementations

impl Clone for HeuristicTokenCounter

fn clone(&self) -> HeuristicTokenCounter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HeuristicTokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for HeuristicTokenCounter

fn default() -> Self

Returns the “default value” for a type.

impl TokenCounter for HeuristicTokenCounter

fn count(&self, message: &Message) -> usize

Approximate the number of tokens contributed by message.

impl Copy for HeuristicTokenCounter