Skip to main content

TargetModel

ruvector_attention::attention::speculative

Trait TargetModel 

Source 
pub trait TargetModel: Send + Sync {
    // Required method
    fn verify_batch(
        &self,
        prefix: &[TokenId],
        draft_tokens: &[TokenId],
    ) -> Vec<Vec<(TokenId, f32)>>;
}
Expand description

Target model trait: the large, accurate model that verifies drafts.

Required Methods§

Source

fn verify_batch( &self, prefix: &[TokenId], draft_tokens: &[TokenId], ) -> Vec<Vec<(TokenId, f32)>>

Evaluates the target model on all draft positions in one forward pass.

Given the prefix and the draft tokens, returns the target model’s full probability distribution at each of the gamma + 1 positions (gamma verification positions plus one bonus position).

Each inner Vec<(TokenId, f32)> is a sparse probability distribution over the vocabulary (only tokens with nonzero probability need appear).

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§