Module draft_model 

A second, smaller GGUF used as the drafter for speculative decoding.

crate::speculative already had the half that is hard to get right: the rejection rule, which makes speculation lossless at every temperature rather than only at --temp 0. What it did not have was a drafter worth running. The only implementation in the tree is crate::speculative::PromptLookupSpeculator, an n-gram match over the history with no model at all. It is free, and it helps on repetitive text, and it cannot carry a coding workload.

Why this is the item that moves the ceiling

Decode reads every weight in the model to emit one token, so

tokens/sec <= memory bandwidth / model bytes

is arithmetic, not engineering. A 17 GB checkpoint on a 960 GB/s card cannot pass about 56 tok/s however good the kernels are. Better kernels move an engine toward that number; they cannot move it past.

A draft model changes what is read per token instead of how fast it is read. A 2 GB drafter proposes k tokens, the target checks all k in ONE pass over its 17 GB, good guesses are kept and bad ones discarded, and the text is exactly what the target would have written alone.

The two things this has to get right

The draft KV must roll back. While proposing, the drafter advances its own cache over tokens the target has not accepted and may never accept. If those rows are left in place, the drafter’s context silently diverges from the target’s. Nothing errors: the accept rate just decays, which reads as “this drafter is bad” rather than “this drafter is desynchronised”. DraftModelSpeculator therefore truncates to synced at the top of every propose, and synced only ever counts tokens the caller’s history actually contains.

This is the repo’s dominant bug shape in its usual dress: two structures that must agree about one thing, here the target’s history and the drafter’s cache, with nothing enforcing it. What enforces it is that synced is derived from the history passed in on every call rather than remembered independently, so the drafter cannot hold an opinion about the history that the history disagrees with.

The vocabularies must match. See VocabMismatch.

Structs

DraftModelSpeculator

A Drafter backed by a second Decoder.

Enums

VocabMismatch

The draft and target checkpoints do not agree about token ids.