Struct HeuristicEstimator 

pub struct HeuristicEstimator;

Byte / character heuristic tuned for modern LLM tokenizers.

• ASCII bytes are counted as ceil(bytes / 4) - the OpenAI rule of thumb, accurate within ~20% for English prose under cl100k_base and o200k_base tokenizers.
• Non-ASCII characters (Unicode scalars outside [0x00, 0x7F]) are counted as ceil(chars / 1.5) - roughly one token per CJK glyph and two per emoji or Arabic/Cyrillic run, again within ~25% of actual tokenizer output.

The two contributions are summed. Good enough for budget packing; swap in a real tokenizer for exact accounting.

Trait Implementations

impl Clone for HeuristicEstimator

fn clone(&self) -> HeuristicEstimator

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HeuristicEstimator

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for HeuristicEstimator

fn default() -> HeuristicEstimator

Returns the “default value” for a type.

impl TokenEstimator for HeuristicEstimator

fn estimate(&self, text: &str) -> u32

Estimate the number of tokens text consumes under the target tokenizer. Returning zero for the empty string is required; otherwise the return value MAY be conservative (overestimate) but MUST NOT vary between calls with the same input.

impl Copy for HeuristicEstimator