Struct HeuristicTokenCounter 

pub struct HeuristicTokenCounter { /* private fields */ }

Estimates token cost from UTF-8 lengths, rounded up per text-bearing part. Tool calls count name bytes plus serialized argument bytes. Attachments and message overhead have fixed costs. These estimates do not bound provider usage.

use rig_memory::{HeuristicTokenCounter, TokenWindowMemory};

let policy = TokenWindowMemory::new(2_000, HeuristicTokenCounter::default());

Implementations

impl HeuristicTokenCounter

pub fn new( bytes_per_token: f32, per_message_overhead: usize, per_attachment_tokens: usize, ) -> Self

Create a counter with explicit parameters.

Non-finite ratios and ratios below 1.0 become 1.0. Overhead and attachment costs may be zero.

pub fn anthropic() -> Self

Uses 3.5 bytes per token, four tokens per message, and 256 per attachment.

Trait Implementations

impl Clone for HeuristicTokenCounter

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for HeuristicTokenCounter

impl Debug for HeuristicTokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for HeuristicTokenCounter

fn default() -> Self

Uses four bytes per token, four tokens per message, and 256 per attachment.

impl TokenCounter for HeuristicTokenCounter

fn count(&self, message: &Message) -> usize

Approximate the number of tokens contributed by message.