pub fn load_hf_from_json(json: &str) -> Result<BpeTokenizer, AprenderError>Expand description
Load tokenizer from HuggingFace tokenizer.json format.
Uses the optimized loading path (GH-378): pre-sized HashMaps, owned-string
moves, and fast merge loading. Applies to all vocab sizes via
config_from_vocab_size dispatch (Qwen2, Whisper, GPT-2, LLaMA).
§Arguments
json- JSON string of tokenizer configuration
§Returns
Loaded tokenizer with vocabulary and merge rules
§Errors
Returns error if parsing fails.