Struct TokenEstimator

Source

pub struct TokenEstimator { /* private fields */ }

Expand description

Estimates token counts for messages and tracks cumulative usage across turns.

§Token Estimation Strategy

Uses character-based approximation:

ASCII characters: 4 chars ≈ 1 token
Non-ASCII characters (CJK, emoji, etc.): 2 chars ≈ 1 token

This provides a reasonable estimate within ~20% of actual token counts for most common text patterns.

§Example

use talos_agent::token::{TokenEstimator, ModelPricing};
use talos_core::message::{Message, Usage};

let mut estimator = TokenEstimator::new();

// Estimate tokens for a set of messages
let messages = vec![
    Message::User { content: "Hello, world!".into() },
    Message::Assistant { content: "Hi there!".into(), tool_calls: vec![], reasoning: None },
];
let estimated = estimator.estimate(&messages);

// Track actual usage from a turn
estimator.track_usage(Usage {
    input_tokens: 100,
    output_tokens: 50,
    cache_read_tokens: 80,
    cache_write_tokens: 20,
    reasoning_tokens: 0,
});

// Get cumulative usage
let total = estimator.total_usage();
assert_eq!(total.input_tokens, 100);

// Estimate cost
let pricing = ModelPricing {
    input_per_1k: 0.003,
    output_per_1k: 0.015,
    cache_read_per_1k: 0.001,
    cache_write_per_1k: 0.002,
};
let cost = estimator.estimated_cost(&pricing);

Implementations§

Source §

impl TokenEstimator

Source

pub fn new() -> Self

Creates a new token estimator with zero cumulative usage.

Source

pub fn estimate(&self, messages: &[Message]) -> u32

Estimates the token count for a slice of messages.

Iterates over all message content (user text, assistant text, tool results) and applies character-based heuristics to approximate token count.

§Arguments

messages — The messages to estimate tokens for.

§Returns

The estimated total token count across all messages.

§Example

use talos_agent::token::TokenEstimator;
use talos_core::message::Message;

let estimator = TokenEstimator::new();
let messages = vec![
    Message::User { content: "Hello!".into() },
];
let tokens = estimator.estimate(&messages);
assert!(tokens > 0);

Source

pub fn estimate_text(text: &str) -> u32

Estimates the token count for a single string of text.

Uses character-based heuristics:

ASCII characters: 4 chars ≈ 1 token
Non-ASCII characters: 2 chars ≈ 1 token

Empty strings return 0 tokens.

§Arguments

text — The text to estimate tokens for.

§Returns

The estimated token count.

§Example

use talos_agent::token::TokenEstimator;

// English text: ~4 chars per token
let english = TokenEstimator::estimate_text("Hello, world!");
assert!(english > 0);

// CJK text: ~2 chars per token
let cjk = TokenEstimator::estimate_text("你好世界");
assert!(cjk > 0);

// Empty text: 0 tokens
let empty = TokenEstimator::estimate_text("");
assert_eq!(empty, 0);

Source

pub fn track_usage(&mut self, turn_usage: Usage)

Accumulates usage from a single turn into the cumulative total.

§Arguments

turn_usage — The usage statistics from a single turn.

§Example

use talos_agent::token::TokenEstimator;
use talos_core::message::Usage;

let mut estimator = TokenEstimator::new();
estimator.track_usage(Usage {
    input_tokens: 100,
    output_tokens: 50,
    cache_read_tokens: 80,
    cache_write_tokens: 20,
    reasoning_tokens: 0,
});

let total = estimator.total_usage();
assert_eq!(total.input_tokens, 100);
assert_eq!(total.output_tokens, 50);

Source

pub fn total_usage(&self) -> Usage

Returns the cumulative usage across all tracked turns.

§Returns

A Usage struct with the sum of all tracked turn usage.

§Example

use talos_agent::token::TokenEstimator;
use talos_core::message::Usage;

let mut estimator = TokenEstimator::new();
estimator.track_usage(Usage {
    input_tokens: 100,
    output_tokens: 50,
    cache_read_tokens: 0,
    cache_write_tokens: 0,
    reasoning_tokens: 0,
});
estimator.track_usage(Usage {
    input_tokens: 200,
    output_tokens: 75,
    cache_read_tokens: 100,
    cache_write_tokens: 50,
    reasoning_tokens: 0,
});

let total = estimator.total_usage();
assert_eq!(total.input_tokens, 300);
assert_eq!(total.output_tokens, 125);
assert_eq!(total.cache_read_tokens, 100);
assert_eq!(total.cache_write_tokens, 50);

Source

pub fn estimated_cost(&self, pricing: &ModelPricing) -> f64

Calculates the estimated cost based on cumulative usage and model pricing.

Uses simple multiplication: (tokens / 1000) * price_per_1k for each usage category.

§Arguments

pricing — The pricing information for the model.

§Returns

The estimated total cost in the currency unit of the pricing.

§Example

use talos_agent::token::{TokenEstimator, ModelPricing};
use talos_core::message::Usage;

let mut estimator = TokenEstimator::new();
estimator.track_usage(Usage {
    input_tokens: 1000,
    output_tokens: 500,
    cache_read_tokens: 800,
    cache_write_tokens: 200,
    reasoning_tokens: 0,
});

let pricing = ModelPricing {
    input_per_1k: 0.003,
    output_per_1k: 0.015,
    cache_read_per_1k: 0.001,
    cache_write_per_1k: 0.002,
};

let cost = estimator.estimated_cost(&pricing);
// (1000/1000)*0.003 + (500/1000)*0.015 + (800/1000)*0.001 + (200/1000)*0.002
// = 0.003 + 0.0075 + 0.0008 + 0.0004 = 0.0117
assert!((cost - 0.0117).abs() < 0.0001);