Skip to main content

Module sampling

Module sampling 

Source
Expand description

Turning a row of logits into the next token id.

The model produces logits; choosing from them is a separate decision with its own knobs, and every one of them is a way of throwing away part of the distribution. Greedy decoding keeps only the argmax, temperature reshapes what is left, and the two truncations cut the tail that a model with a large vocabulary spreads real probability mass across.

Structsยง

Sampler
How the next token is drawn from a row of logits.