Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Token counter with cached BPE instance.

Implementations

impl Tokenizer

pub fn new(backend: TokenizerBackend) -> Self

pub fn count(&self, text: &str) -> u32

Count tokens in text.

pub fn count_batch(&self, texts: &[&str]) -> Vec<u32>

Count tokens for multiple texts.

pub fn truncate<'a>(&self, text: &'a str, max_tokens: u32) -> &'a str

Truncate text to fit within a token budget. Returns the longest prefix that fits.