Trait Measurer 

pub trait Measurer: Send + Sync {
    // Required methods
    fn tokenize(&self, text: &str, model: &Model) -> Result<u32, TokenizerError>;
    fn supported(&self, model: &Model) -> bool;
}

Count real tokens for the target model. No estimation, no synthetic fallback.

Required Methods

fn tokenize(&self, text: &str, model: &Model) -> Result<u32, TokenizerError>

Returns the actual token count for text under model’s tokenizer. Implementations MUST return Err for unregistered models rather than silently approximating — this is a DoR gate (no hardcoded ratios).

fn supported(&self, model: &Model) -> bool

Returns true iff the tokenizer for model is registered and callable.

Implementors

impl Measurer for LocalMeasurer