Skip to main content

load_tokenizer

rlx_text::tokenizer

Function load_tokenizer 

Source 
pub fn load_tokenizer(path: &Path) -> Result<TokenizerHandle>
Expand description

Load a HF-format tokenizer.json from disk.