Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Thread-safe tokenizer with persistent single-document and Rayon-worker caches. Token IDs remain those of the source tokenizer.json.

Implementations

impl Tokenizer

pub fn from_file(path: impl AsRef<Path>) -> Result<Self>

Load a byte-level BPE tokenizer from a Hugging Face tokenizer.json.

pub fn from_bytes(bytes: &[u8]) -> Result<Self>

Load a byte-level BPE tokenizer from in-memory tokenizer.json bytes.

pub fn encode(&self, text: &str, add_special_tokens: bool) -> Result<Vec<u32>>

Encode one UTF-8 string, optionally applying the tokenizer’s supported single-sequence post-processor.

pub fn encode_batch( &self, texts: Vec<String>, add_special_tokens: bool, ) -> Result<Vec<Vec<u32>>>

Encode documents in input order. Small batches stay on the caller; larger batches use persistent per-Rayon-thread tokenizer forks.

pub fn decode(&self, ids: &[u32], skip_special_tokens: bool) -> Result<String>

Decode IDs through the byte-level vocabulary. Invalid UTF-8 fragments use the replacement character, matching the display behavior of the previous tokenizer backend.

pub fn id_to_token(&self, id: u32) -> Option<String>

Return the exact vocabulary spelling stored in tokenizer.json.

pub fn token_to_id(&self, token: &str) -> Option<u32>

Resolve an exact model or added-token spelling to its ID.

pub fn vocab_size(&self) -> usize

Vocabulary extent including added tokens.