Struct TokenCounter 

pub struct TokenCounter { /* private fields */ }

Estimates token counts from text using a configurable chars-per-token ratio.

This is a heuristic estimator — real tokenization varies per model. The default ratio of 4.0 chars/token approximates GPT-family and Claude models.

Example

use neuron_context::TokenCounter;

let counter = TokenCounter::new();
let estimate = counter.estimate_text("Hello, world!");
assert!(estimate > 0);

Implementations

impl TokenCounter

pub fn new() -> Self

Creates a new TokenCounter with the default ratio of 4.0 chars/token.

pub fn with_ratio(chars_per_token: f32) -> Self

Creates a new TokenCounter with a custom chars-per-token ratio.

pub fn estimate_text(&self, text: &str) -> usize

Estimates the number of tokens in a text string.

pub fn estimate_messages(&self, messages: &[Message]) -> usize

Estimates the total token count for a slice of messages.

Iterates all content blocks and sums their estimated token counts.

pub fn estimate_tools(&self, tools: &[ToolDefinition]) -> usize

Estimates the total token count for a slice of tool definitions.

Trait Implementations

impl Default for TokenCounter

fn default() -> Self

Returns the “default value” for a type.