Struct DraftModelSpeculator 

pub struct DraftModelSpeculator { /* private fields */ }

A Drafter backed by a second Decoder.

Owns its own KV caches, entirely separate from the target’s. The two models run over the same token sequence but have different layer counts, head counts and head dimensions, so nothing about the two caches is shared.

Implementations

impl DraftModelSpeculator

pub fn keeps_host_kv(&self) -> bool

True when this drafter’s KV really lives in the host caches it owns.

A backend that keeps KV on the device leaves these at zero, and a drafter cannot roll back rows it cannot see. Callers check this after one warm-up rather than discovering it as a wrong accept rate.

pub fn synced_len(&self) -> usize

How many tokens of history this drafter’s KV currently holds. Exposed so a caller can assert the drafter kept up.

pub fn new( decoder: Decoder, target_config: &ModelConfig, sampling: SamplingParams, seed: u64, max_draft: usize, min_prob: f32, ) -> Result<Self, VocabMismatch>

Fails when the two checkpoints cannot be compared token for token. See VocabMismatch.

Trait Implementations

impl Drafter for DraftModelSpeculator

fn propose( &mut self, history: &[usize], _target_hidden: &[f32], max_len: usize, ) -> DraftBlock

Proposes at most max_len tokens to follow history. target_hidden is the target model’s final-layer hidden state for history’s last token, or empty when none is available yet (which a drafter that needs it must handle by proposing nothing). Takes &mut self because a drafter that is itself a model carries KV state across calls, and hiding that behind interior mutability would put a runtime borrow check on the hot path to buy nothing. A stateless drafter simply ignores it.