Struct CharTokenizer 

pub struct CharTokenizer;

Fallback Tokenizer that approximates tokens as text.len() / 4.

This is the rule-of-thumb every provider documentation suggests for back-of-envelope sizing — accurate enough to spot the difference between “10 tokens” and “10k tokens”, but not a substitute for a real tokenizer when budgets are tight. It is the silent default in ailoop-history::HistoryBuilder so dev/test code does not have to wire one up; production callers should pass an explicit provider-specific tokenizer (e.g. ailoop_anthropic::OnlineCalibratedTokenizer) via HistoryBuilder::tokenizer.

Trait Implementations

impl Tokenizer for CharTokenizer

fn count_text(&self, text: &str) -> usize

Count tokens in a flat string. The only required method.

fn count_message(&self, message: &Message) -> usize

Count tokens in a single Message, walking every block kind the provider sees. Defaults to summing block-level Self::count_text calls; override if your tokenizer has a cheaper batch path.

fn count_messages(&self, messages: &[Message]) -> usize

Count tokens in a slice of messages — the budget unit History::compact_if_needed measures against.