Skip to main content

TokenCounter

Trait TokenCounter 

Source
pub trait TokenCounter {
    // Required methods
    fn id(&self) -> &str;
    fn revision(&self) -> &str;
    fn count(&self, text: &str) -> u32;

    // Provided methods
    fn count_parts(&self, parts: &[&str]) -> u32 { ... }
    fn count_message(&self, message: &ChatMessage) -> u32 { ... }
    fn count_tool_schema(&self, schema: &Value) -> u32 { ... }
}
Expand description

Counts tokens for a budget. Implementations name themselves so a ledger can say which counter produced a number.

Vendor tokenizers (OpenAI/Codex, Hugging Face JSON for Qwen, a Claude-calibrated table) plug in here. Runtime compile uses ConservativeCounter until one of those is wired for the active model.

Required Methods§

Source

fn id(&self) -> &str

Source

fn revision(&self) -> &str

Source

fn count(&self, text: &str) -> u32

Provided Methods§

Source

fn count_parts(&self, parts: &[&str]) -> u32

Source

fn count_message(&self, message: &ChatMessage) -> u32

Source

fn count_tool_schema(&self, schema: &Value) -> u32

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§