Skip to main content

TargetModel

tensorlogic_trustformers::speculative_decoding::traits

Trait TargetModel 

Source 
pub trait TargetModel {
    // Required methods
    fn vocab_size(&self) -> usize;
    fn verify(
        &self,
        prefix: &[TokenId],
        draft_tokens: &[TokenId],
    ) -> SpeculativeDecodingResult<TargetScores>;
}
Expand description

A model that, given a prefix and up to k draft continuations, returns per-position distributions (as log-probs) in a single forward pass.

Required Methods§

Source

fn vocab_size(&self) -> usize

Vocabulary cardinality the target emits log-probs over. Must match the draft’s vocab_size().

Source

fn verify( &self, prefix: &[TokenId], draft_tokens: &[TokenId], ) -> SpeculativeDecodingResult<TargetScores>

Score prefix concatenated with draft_tokens: return k + 1 distributions (the k draft-covered positions plus the bonus).

Implementors§