Function sample_token 

pub fn sample_token(probs: &[f32], rng: &mut Rng) -> usize

👎Deprecated since 0.1.0:

allocates a vocab-sized Vec per call; use sample_token_into on hot paths

Sample a token index from a probability distribution.

Builds a prefix-sum (CDF) then uses binary search for O(log V) lookup instead of the O(V/2) average of a linear scan.

Allocates a CDF buffer on every call. For the hot decode loop, prefer sample_token_into which reuses a pre-allocated buffer.