Struct CharTokenCounter 

pub struct CharTokenCounter;

Default counter: CJK characters count as 1 token each, other characters count as 1 token per 4 characters (rounded up).

The common chars/4 approximation underestimates Chinese by roughly 4x (Chinese is roughly 1 token per character), so this implementation distinguishes CJK to stay closer to reality; it is still an approximation — inject a custom implementation when exact counts are needed.

Example

use molo::memory::{CharTokenCounter, TokenCounter};

let counter = CharTokenCounter;
// 5 characters → 2 tokens (1 token per 4 characters, rounded up).
assert_eq!(counter.count("hello").await?, 2);
// 4 characters → exactly 1 token.
assert_eq!(counter.count("abcd").await?, 1);
// 7 characters → 2 tokens (rounded up).
assert_eq!(counter.count("morning").await?, 2);

Trait Implementations

impl Clone for CharTokenCounter

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for CharTokenCounter

impl Debug for CharTokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CharTokenCounter

fn default() -> Self

Returns the “default value” for a type.

impl Eq for CharTokenCounter

impl PartialEq for CharTokenCounter

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for CharTokenCounter

impl TokenCounter for CharTokenCounter

fn count<'life0, 'life1, 'async_trait>( &'life0 self, text: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<usize, MemoryError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Counts the token number of a text.