Skip to main content

sample_boundary_token_dev

Function sample_boundary_token_dev 

Source
pub fn sample_boundary_token_dev(
    e: &Engine,
    logits: &CudaSlice<f32>,
    n_vocab: usize,
    sp: &SpecSampling,
    pen_hist: &[u32],
    sctr: &mut u32,
    site: &str,
) -> Result<u32, Box<dyn Error>>
Expand description

Draw a BOUNDARY token from the target distribution the request asked for (lane/sampled-spec-quality, Item 1) — the fix for “sampled spec emits an ARGMAX token at every burst boundary”.

WHY THIS EXISTS. A spec burst’s first emitted token is not produced by the accept walk: it comes off a logits row that already exists (the prime’s last row on a cold burst; the row after the last committed token on a continuation burst; the prefix-cache entry’s boundary row on a restored one). Pre-lane that token was argmax in BOTH sampling regimes, so a sampled stream took a greedy token once per burst — measured, not estimated, in research/spec-cache-20260818/SAMPLED-QUALITY.md. At temperature > 0 the customer asked for a sampled token, so this draws one.

THE PROGRAM IS THE FULL-ACCEPT BONUS’S PROGRAM, deliberately: penalize the row (over the session’s window), take this row’s OWN filter stats (the sampfix-20260805 law — stats from a neighbour row mis-scale every e0 and can wipe the row to token 0), gumbel-perturb with the session’s Philox stream at *sctr, argmax the perturbed row. Reusing the bonus’s composition means sample_check’s distributional oracle covers this draw too, and the boundary token is drawn from the same filtered/penalized p the accept walk targets.

THE STREAM IS THE SESSION’S, NOT A FRESH ONE. sctr is the caller’s live counter and is advanced by exactly one, so a boundary draw consumes the next value in the same Philox stream the accept walk uses — never a second, independently seeded stream (which would be a new distributional bug: two streams from one seed correlate wherever their counters collide). That also makes a restored session’s boundary draw at sctr == 0 bit-identical to the cold session’s own first draw from the same logits row, which is what preserves the sampled-hit lane’s per-seed hit==cold byte identity.