pub trait TokenCounter {
// Required methods
fn id(&self) -> &str;
fn revision(&self) -> &str;
fn count(&self, text: &str) -> u32;
// Provided methods
fn count_parts(&self, parts: &[&str]) -> u32 { ... }
fn count_message(&self, message: &ChatMessage) -> u32 { ... }
fn count_tool_schema(&self, schema: &Value) -> u32 { ... }
}Expand description
Counts tokens for a budget. Implementations name themselves so a ledger can say which counter produced a number.
Vendor tokenizers (OpenAI/Codex, Hugging Face JSON for Qwen, a
Claude-calibrated table) plug in here. Runtime compile uses
ConservativeCounter until one of those is wired for the active model.
Required Methods§
Provided Methods§
fn count_parts(&self, parts: &[&str]) -> u32
fn count_message(&self, message: &ChatMessage) -> u32
fn count_tool_schema(&self, schema: &Value) -> u32
Dyn Compatibility§
This trait is dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety".