Struct LocalMeasurer 

pub struct LocalMeasurer { /* private fields */ }

Adapter that implements Measurer over the registry.

Implementations

impl LocalMeasurer

pub fn new(registry: TokenizerRegistry) -> Self

Build from an explicit registry.

pub fn with_defaults() -> Result<Self, TokenizerError>

Build with the default tokenizer set.

Errors

Same as TokenizerRegistry::with_defaults.

pub fn cross_tokenize(&self, text: &str) -> Vec<(String, u32)>

Tokenize the given text against EVERY registered tokenizer rather than the model-specific one. Returns (name, count) tuples in stable order so cross-tokenizer audit rows replay.

This is the telemetry that powers tokenizer-resilient closed-loop tuning: when an upstream provider ships a new tokenizer, the per-tokenizer savings stop matching the model-specific savings and the auto-tuner sees the drift before users do.

Trait Implementations

impl Clone for LocalMeasurer

fn clone(&self) -> LocalMeasurer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Measurer for LocalMeasurer

fn tokenize(&self, text: &str, model: &Model) -> Result<u32, TokenizerError>

Returns the actual token count for text under model’s tokenizer. Implementations MUST return Err for unregistered models rather than silently approximating — this is a DoR gate (no hardcoded ratios).

fn supported(&self, model: &Model) -> bool

Returns true iff the tokenizer for model is registered and callable.