Module speculative 

Speculative decoding — draft + verify.

A small (“draft”) model generates N candidate tokens autoregressively. The big (“target”) model does ONE forward pass on prompt + drafts and produces N+1 logit distributions — one for every position at which a draft could be accepted, plus one bonus position after full acceptance. We then apply the DeepMind speculative sampling rule to decide which drafts survive, yielding 1..=N+1 tokens per target forward pass.

Original paper: Leviathan et al., 2023 — “Fast Inference from Transformers via Speculative Decoding” (https://arxiv.org/abs/2211.17192).

This module is algorithm-only: it operates on raw logit vectors and produces a list of accepted token ids. The engine/scheduler integration (draft-model loading, KV-cache management, iteration plumbing) is a separate layer and explicitly out of scope for this file — wiring it is the follow-up once the algorithm is locked down.

Structs

Speculation

Input to verify_speculation: logit vectors at each speculation position.

SpeculationOutcome

Result of one speculate+verify round.

SpeculativeDecodingConfig

Configuration for speculative decoding.

SpeculativeRunner

Drives one round of speculative decoding against a (draft, target) pair of ModelExecutors. Owns neither executor — the engine keeps them and hands references per call.

SpeculativeStepOutcome

Result of a single SpeculativeRunner::step.

Functions

verify_speculation

Execute the DeepMind speculative-sampling accept/reject loop.