Struct SpeculativeDecoder 

pub struct SpeculativeDecoder { /* private fields */ }

Async speculative decoder.

Wraps a draft engine (generating spec_k candidates per step) and a target engine (verifying the candidates in a single batched forward pass). The two engines run with overlap via tokio.

Limitations

• Both engines must use the same tokenizer and vocabulary.
• The draft engine must be strictly smaller/faster than the target.
• The target engine must implement Rewindable (KV-cache based). For SSM targets use SpeculativeDecoder::new_n1 which forces N=1 mode.

Example

let decoder = SpeculativeDecoder::new(
    draft_engine,
    target_engine,
    AsyncSpecConfig::default(),
);
let stats = decoder.generate("hello", 128, |tok| print!("{tok}")).await?;

Implementations

impl SpeculativeDecoder

pub fn new( draft: InferenceEngine, target: InferenceEngine, config: AsyncSpecConfig, ) -> Self

Construct a new async speculative decoder.

Both engines must be loaded (i.e. is_loaded() is true) before generate is called.

pub fn new_n1( draft: InferenceEngine, target: InferenceEngine, config: AsyncSpecConfig, ) -> Self

Construct a decoder that always uses N=1 mode (for SSM targets).

pub fn stats(&self) -> &SpecStats

Return the accumulated statistics from all generate calls.

pub fn reset_stats(&mut self)

Reset statistics counters.

pub fn cancellation_token(&self) -> CancellationToken

Return a reference to the cancellation token for external cancellation.

pub async fn generate<F>( &mut self, prompt: &str, on_token: F, ) -> RuntimeResult<String>

where F: FnMut(&str) + Send + 'static,

Run async speculative generation for prompt, calling on_token for each decoded token.

Returns the full generated text and updates self.stats.

SSM fallback

If the target engine’s rewind() returns RewindError::NotSupported on the first call, the decoder automatically falls back to N=1 mode for the rest of the generation. SpecStats::n1_fallbacks is incremented.

Cancellation

The generation loop checks self.cancel after each speculation step. Callers can cancel by calling cancel.cancel() from another task.

Errors

Returns RuntimeError::ModelNotLoaded if either engine is not loaded. Returns RuntimeError::Cancelled if the cancellation token is triggered before the first token is produced.