Struct HuggingFaceTokenizer 

pub struct HuggingFaceTokenizer { /* private fields */ }

Implementations

impl HuggingFaceTokenizer

pub fn from_file(model_name: &str) -> Result<Self>

Load from tokenizer.json, merging in special tokens declared only in a sibling tokenizer_config.json’s added_tokens_decoder. Without this, some releases (Qwen2-VL-2B’s <|image_pad|>) BPE-shatter and silently break MM-aware routing. The merge is idempotent.

pub fn from_tokenizer(tokenizer: HfTokenizer) -> Self

pub fn from_tokenizer_with_model_dir( tokenizer: HfTokenizer, model_dir: &Path, ) -> Self

Wrap an already-loaded HfTokenizer and merge in the sibling tokenizer_config.json special tokens; see Self::from_file.

Trait Implementations

impl Decoder for HuggingFaceTokenizer

fn decode( &self, token_ids: &[TokenIdType], skip_special_tokens: bool, ) -> Result<DecodeResult>

impl Encoder for HuggingFaceTokenizer

fn encode(&self, input: &str) -> Result<Encoding>

fn encode_batch(&self, inputs: &[&str]) -> Result<Vec<Encoding>>

impl From<Tokenizer> for HuggingFaceTokenizer

fn from(tokenizer: HfTokenizer) -> Self

Converts to this type from the input type.

impl Tokenizer for HuggingFaceTokenizer

fn with_options(self, options: TokenizerOptions) -> Self

HF honors TokenizerOptions::add_special_tokens (BOS/EOS per the tokenizer’s post-processor template).

fn validate_prefix_cache(&self) -> Result<()>

Validate that this tokenizer can be safely wrapped in the prefix cache.