Trait TokenCounter

Source

pub trait TokenCounter: Send + Sync {
    // Required method
    fn estimate_text(&self, text: &str) -> usize;

    // Provided methods
    fn estimate_content(&self, content: &[Content]) -> usize { ... }
    fn estimate_message(&self, msg: &AgentMessage) -> usize { ... }
    fn estimate_messages(&self, msgs: &[AgentMessage]) -> usize { ... }
}

Expand description

Pluggable token counting strategy.

The default implementation (HeuristicTokenCounter) uses a ~4 chars/token heuristic — fast and sufficient for context budgeting. Provide a custom implementation for model-specific tokenizers (e.g., tiktoken for OpenAI models, or Anthropic’s native token-counting API).

Only estimate_text needs to be overridden. The higher-level methods (estimate_content, estimate_message, estimate_messages) have default implementations that delegate to estimate_text.

§Example

use phi_core::context::token::{TokenCounter, HeuristicTokenCounter};

let counter = HeuristicTokenCounter;
assert_eq!(counter.estimate_text("hello"), 2); // 5 chars / 4 = 2 (rounded up)