Struct CharApproxCounter 

pub struct CharApproxCounter;

Char-count approximation: 4 chars ≈ 1 token. Used when no real tokeniser is available. More accurate than byte-count for CJK text (3 bytes/char but ~0.5 tokens/char).

Trait Implementations

impl TokenCounter for CharApproxCounter

fn count(&self, text: &str) -> u32

Count tokens in a UTF-8 string.

fn truncate<'a>(&self, text: &'a str, max_tokens: u32) -> &'a str

Return the longest prefix of text that fits within max_tokens. The returned slice is always a valid UTF-8 prefix of text.