Struct SpeculativeOptions 

pub struct SpeculativeOptions {
    pub max_new_tokens: usize,
    pub start_pos: usize,
    pub sampling: SamplingParams,
    pub seed: u64,
}

Everything speculative_decode_with needs beyond the model, the prompt and the drafter.

Fields

max_new_tokens: usize

start_pos: usize

Absolute position of the first prompt_tokens token in the KV cache: 0 for a fresh cache, cache.seq_len when resuming a warm one (a prefix-cache hit, or a second call continuing the first). Every position and every rollback length inside the decode loop is absolute, so a non-zero base is not a special case – see rolls_back_to_absolute_positions_on_a_warm_cache.

sampling: SamplingParams

The sampling configuration the target model would have used without speculation. Verification is lossless with respect to exactly these parameters (see accept_or_resample).

seed: u64

Trait Implementations

impl Clone for SpeculativeOptions

fn clone(&self) -> SpeculativeOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SpeculativeOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SpeculativeOptions

fn default() -> SpeculativeOptions

Returns the “default value” for a type.