Function load_from_files 

pub fn load_from_files(
    vocab_json: &str,
    merges_txt: &str,
) -> Result<BpeTokenizer>

Load tokenizer from vocab.json and merges.txt files.

Arguments

• vocab_json - JSON string of vocabulary mapping (token -> id)
• merges_txt - Text file with merge rules (one “pair1 pair2” per line)

Returns

Loaded tokenizer

Errors

Returns error if parsing fails.