Function get_tokenizer
Source pub fn get_tokenizer(models_dir: &Path) -> Result<&'static Tokenizer, AppError>
Expand description
Returns the process-wide Tokenizer singleton, initializing it on first call.
ยงErrors
Returns Err when the tokenizer files cannot be loaded from models_dir.