Function load_tokenizer 

pub fn load_tokenizer(model_repo: &str) -> Result<Tokenizer>

Load a tokenizer from a HuggingFace model repository.

Downloads tokenizer.json on first call; subsequent calls use the cache.

Errors

Returns an error if the tokenizer file cannot be downloaded or parsed.