Skip to main content

ModelEngine

Trait ModelEngine 

Source
pub trait ModelEngine: Send + Sync {
    // Required methods
    fn forward_batch(
        &mut self,
        tokens: &[TokenId],
    ) -> impl Future<Output = Result<Vec<Logits>, EngineError>> + Send;
    fn forward_token(
        &mut self,
        token: TokenId,
    ) -> impl Future<Output = Result<Logits, EngineError>> + Send;
}
Expand description

Abstract inference engine interface for draft and target models. Compatible with Candle, Burn, ONNX Runtime, or custom model runtimes.

Required Methods§

Source

fn forward_batch( &mut self, tokens: &[TokenId], ) -> impl Future<Output = Result<Vec<Logits>, EngineError>> + Send

Computes logits for a sequence of tokens in a single parallel forward pass.

Source

fn forward_token( &mut self, token: TokenId, ) -> impl Future<Output = Result<Logits, EngineError>> + Send

Computes logits for a single token step.

Dyn Compatibility§

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§