Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

A tokenizer that can count tokens in text.

Implementations

impl Tokenizer

pub fn new(kind: TokenizerKind) -> Self

Create a new tokenizer of the given kind. Falls back to heuristic if the BPE model fails to load.

pub fn count_tokens(&self, content: &str, is_prose: bool) -> usize

Count tokens in the given text.

pub fn kind(&self) -> &TokenizerKind

Return the kind of this tokenizer.

pub fn is_real(&self) -> bool

Whether this tokenizer uses real BPE encoding (not heuristic).