pub fn eog_token_ids(file: &impl TensorSource) -> HashSet<u32>Expand description
Every token id that ends generation, not just eos_token_id.
A single EOS id is wrong for most modern chat checkpoints: Llama-3
ends turns with <|eot_id|> while its eos_token_id is
<|end_of_text|>, and gemma-4 ends with <turn|>. Stopping only on
the metadata EOS means the model keeps generating past the end of its
turn and starts a new one — the “it answers, then interviews itself”
failure.
Mirrors llama.cpp: the literal-name list above, plus the
eos/eot/eom metadata ids, which it folds in with a warning when
they were not already caught by name.