Struct CharacterTokenizer 

pub struct CharacterTokenizer { /* private fields */ }

Simple character-based token estimator Uses ~4 characters per token heuristic (common approximation)

Implementations

impl CharacterTokenizer

pub fn new() -> Self

Create a new tokenizer with the default 4 characters per token

pub fn with_chars_per_token(chars_per_token: usize) -> Self

Create a tokenizer with a custom characters-per-token ratio

pub fn estimate_tokens(&self, text: &str) -> usize

Estimate the number of tokens in the given text

Trait Implementations

impl Clone for CharacterTokenizer

fn clone(&self) -> CharacterTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CharacterTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CharacterTokenizer

fn default() -> CharacterTokenizer

Returns the “default value” for a type.