pub fn sampling_distribution(
logits: &[f32],
params: &SamplingParams,
history: &[usize],
) -> Vec<f32>Expand description
The exact distribution Sampler::sample draws from for these
logits, params and history: penalties applied, temperature divided
in, top-k and top-p filtered, renormalised to sum to 1.
This is what makes lossless speculative verification possible. The
speculative-sampling rejection rule compares p_target(x) against
the draft’s q(x), and “the target’s probability” is meaningless
unless it is the probability the configured sampler would actually
have used – a rule that compared against the raw softmax while the
server sampled with top_p = 0.9 would be lossless with respect to
a model nobody is running.
Greedy (temperature <= 0.0) is a distribution too: the point mass
on the argmax. Returning it as one rather than as a special case is
why the same verification code is correct at every temperature.