pub fn load_tokenizer(
ctx: &Context,
default_eos_token: &str,
) -> Result<(Tokenizer, Option<EosTokenId>)>Expand description
Load the tokenizer and resolve EOS token ID(s).
default_eos_token is the model-specific fallback (e.g. “<|eot_id|>” for LLaMA,
“<|endoftext|>” for Qwen2).