Struct HfTokenizerWrapper 

pub struct HfTokenizerWrapper { /* private fields */ }

Wrapper for HuggingFace tokenizer.json files.

Parses the JSON to extract the vocabulary map, then delegates to the existing WordPieceTokenizer for actual tokenization. Supports both WordPiece and BPE vocab formats (both map tokens → IDs).

Implementations

impl HfTokenizerWrapper

pub fn from_file(path: &Path) -> Result<Self>

Load from a HuggingFace tokenizer.json file.

pub fn from_file_static(path: &Path) -> Result<Self>

Load the tokenizer of a static embedding model, which may have no [CLS]/[SEP]; use Self::encode_plain with it.

pub fn encode_plain(&self, text: &str, max_tokens: usize) -> Vec<i32>

pub fn vocab_size(&self) -> usize

pub fn encode(&self, text: &str, max_len: usize) -> TokenizedInput