Struct SpeculativeDecoder 

pub struct SpeculativeDecoder<D: DraftModel, T: TargetModel> { /* private fields */ }

Speculative decoder composing a draft model and a target model.

Trait bounds are intentionally deferred to the impl blocks rather than baked into the struct definition so callers can hold a SpeculativeDecoder whose inner models do not implement Debug.

Implementations

impl<D: DraftModel, T: TargetModel> SpeculativeDecoder<D, T>

pub fn new( draft: D, target: T, config: SpeculativeDecoderConfig, ) -> SpeculativeDecodingResult<Self>

Build a decoder. Returns an error if the draft and target disagree on vocabulary size, or if the config is invalid.

pub fn metrics(&self) -> &SpeculativeMetrics

Read-only access to the current metrics snapshot.

pub fn reset_metrics(&mut self)

Reset the metrics counters.

pub fn config(&self) -> &SpeculativeDecoderConfig

Read-only access to the configuration.

pub fn generate( &mut self, prefix: &[TokenId], max_tokens: usize, ) -> SpeculativeDecodingResult<Vec<TokenId>>

Run speculative decoding starting from prefix and producing at most max_tokens new tokens. The returned vector contains only the generated continuation, not the original prefix.

Uses an internally-seeded deterministic StdRng (seed 42). See Self::generate_with_rng for caller-controlled seeding.

pub fn generate_with_rng( &mut self, prefix: &[TokenId], max_tokens: usize, rng: &mut dyn SpecRng, ) -> SpeculativeDecodingResult<Vec<TokenId>>

Run speculative decoding with a caller-supplied RNG.

Trait Implementations

impl<D: DraftModel + Debug, T: TargetModel + Debug> Debug for SpeculativeDecoder<D, T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.