Struct HeuristicTokenCounter 

pub struct HeuristicTokenCounter { /* private fields */ }

Heuristic token counter using character-based estimation.

Uses the approximation: tokens ≈ characters / 4, with a 10% safety margin plus additional overhead for message metadata (role, timestamps, etc.).

This is intentionally conservative to avoid underestimating token usage.

Implementations

impl HeuristicTokenCounter

pub fn new( chars_per_token: f64, safety_margin: f64, metadata_overhead: u32, ) -> HeuristicTokenCounter

Create a new heuristic counter with custom parameters.

pub fn with_defaults() -> HeuristicTokenCounter

Create with default parameters (chars/4 + 10% margin + 10 metadata overhead).

Trait Implementations

impl Clone for HeuristicTokenCounter

fn clone(&self) -> HeuristicTokenCounter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HeuristicTokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for HeuristicTokenCounter

fn default() -> HeuristicTokenCounter

Returns the “default value” for a type.

impl TokenCounter for HeuristicTokenCounter

fn count_message(&self, message: &Message) -> u32

Count tokens in a single message.

fn count_text(&self, text: &str) -> u32

Count tokens in a plain text string.

fn count_messages(&self, messages: &[Message]) -> u32

Count tokens in multiple messages.