Skip to main content

speculative_decode

Function speculative_decode 

Source
pub fn speculative_decode<D: Drafter + ?Sized>(
    decoder: &Decoder,
    prompt_tokens: &[usize],
    max_new_tokens: usize,
    kv_caches: &mut [KvCache],
    drafter: &D,
) -> SpeculativeDecodeResult
Expand description

Greedy speculative decode over a fresh KV cache, with prompt-lookup drafting. Thin wrapper over speculative_decode_with, kept for callers that want the original no-options shape.