Struct HfTokenizer 

pub struct HfTokenizer { /* private fields */ }

Cached HuggingFace tokenizer

Implementations

impl HfTokenizer

pub fn load(name_or_path: &str) -> Result<Self>

Load a tokenizer from HuggingFace hub or local path (native only)

Examples:

• "Alibaba-NLP/gte-Qwen2-1.5B-instruct" - from HuggingFace hub
• "/path/to/tokenizer.json" - from local file

pub fn from_bytes(bytes: &[u8]) -> Result<Self>

Load a tokenizer from bytes (e.g., read from Directory)

This allows loading tokenizers from any Directory implementation, including remote storage like S3 or HTTP. This is the primary method for WASM targets.

pub fn from_source(source: &TokenizerSource) -> Result<Self>

Load from a TokenizerSource (native only)

pub fn tokenize(&self, text: &str) -> Result<Vec<(u32, u32)>>

Tokenize text and return token IDs

Returns a vector of (token_id, count) pairs where count is the number of times each token appears in the text.

pub fn tokenize_unique(&self, text: &str) -> Result<Vec<u32>>

Tokenize text and return unique token IDs (for weighting: one)