Struct ApproxTokenizer 

pub struct ApproxTokenizer;

Character-based heuristic (chars / 4).

Under-counts in practice; use as a last resort (e.g. for Anthropic where no stable public tokenizer exists) and add a 1.3x safety factor when comparing against a hard context window.

Trait Implementations

impl Clone for ApproxTokenizer

fn clone(&self) -> ApproxTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for ApproxTokenizer

impl Debug for ApproxTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ApproxTokenizer

fn default() -> ApproxTokenizer

Returns the “default value” for a type.

impl Tokenizer for ApproxTokenizer

fn count(&self, text: &str) -> u32

Count tokens in text.

fn name(&self) -> &str

Human-readable identifier for logging.