Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Wraps a CoreBPE for one specific encoding.

Implementations

impl Tokenizer

pub fn for_model(model: &str) -> Result<Self>

Construct from an OpenAI model name ("gpt-4", "gpt-4o", "gpt-4.1", "gpt-5", etc.). Tries tiktoken_rs::get_bpe_from_model first; if that fails (the model is too new for the bundled mapping), falls back to encoding inference via name pattern.

pub fn for_encoding(name: &str) -> Result<Self>

Construct from an encoding name. Accepts "cl100k_base" and "o200k_base".

pub fn encoding_name(&self) -> &str

Encoding name ("cl100k_base" or "o200k_base").

pub fn count(&self, text: &str) -> usize

Count BPE tokens in text, ignoring special tokens.

pub fn count_many(&self, texts: &[&str], parallel: bool) -> Vec<usize>

Bulk count. With parallel = true distributes across rayon’s pool.

pub fn encode(&self, text: &str) -> Vec<u32>

Encode to BPE token IDs (ordinary mode, no special tokens).

pub fn decode(&self, tokens: &[u32]) -> Result<String>

Decode a slice of BPE token IDs back to a string.

pub fn fits(&self, text: &str, budget: usize) -> bool

True iff text encodes to <= budget BPE tokens.

pub fn truncate_to(&self, text: &str, budget: usize) -> Result<String>

Encode text, truncate to the first budget tokens, and decode back. If text already fits, returns it unchanged. Boundary handling is whatever tiktoken-rs’s decode does on a mid-token cut, which is well-defined for cl100k/o200k since each token decodes to a complete UTF-8 sequence in the merged-vocabulary case.