Module speculative 

Prompt-lookup speculative decoding: propose several candidate next tokens by finding a repeat of the current context elsewhere in the token history (no separate draft model needed, unlike classic speculative decoding), then verify all candidates in a single Decoder::forward_batch call instead of one forward_token call per candidate.

This is the CPU-only, no-draft-model variant of speculative decoding (the same idea as vLLM’s “prompt lookup decoding”), chosen specifically because it needs no GPU and no second model to be useful – unlike tree-based speculative decoding with a real draft model, which needs real hardware to actually pay off.

Quality-neutrality is the property that matters most here

Speculative decoding is only worth having if it produces exactly the same output as plain greedy decode, just potentially faster. speculative_decode’s accept/reject protocol is designed so that every accepted token is one forward_batch would have produced anyway on its own path: a candidate is only kept if the model’s own argmax at that position agrees with it. This is checked directly by speculative_decode_matches_greedy_token_for_token, which runs both this module’s decode and a plain sequential forward_token loop against the same decoder and asserts the exact same token sequence comes out either way.

Structs

PromptLookupSpeculator

Proposes candidate continuation tokens by looking for the longest available match of the most recent ngram_size tokens earlier in history, and returning up to max_draft_len tokens that followed that earlier occurrence. Returns an empty vector if no match is found or history is too short to contain one.

SpeculativeDecodeResult

Result of a speculative decode run, with the counters that make its actual savings observable rather than just assumed.

Functions

speculative_decode

Runs greedy decoding for max_new_tokens steps, using speculator to propose candidate continuations and verifying them in batches. prompt_tokens is processed as a single prefill batch (one forward_batch call for the whole prompt, not one per prompt token – itself a real saving independent of speculation).