Expand description
Turning a row of logits into the next token id.
The model produces logits; choosing from them is a separate decision with its own knobs, and every one of them is a way of throwing away part of the distribution. Greedy decoding keeps only the argmax, temperature reshapes what is left, and the two truncations cut the tail that a model with a large vocabulary spreads real probability mass across.
Structsยง
- Sampler
- How the next token is drawn from a row of logits.