Skip to main content

InferenceEngine

el_runtime

Trait InferenceEngine 

Source 
pub trait InferenceEngine {
    // Required methods
    fn prefill(&mut self, tokens: &[Token]) -> Result<u32>;
    fn next_logits(&mut self, committed: &[Token]) -> Vec<i32>;
    fn eos_token(&self) -> Token;
}
Expand description

The inference engine adapter (RuntimeAcl). Implemented for real by Candle in the excluded adapter el-engine-candle (ADR-002).

Logits are integer milli-logits to keep the orchestrator deterministic and float-free; a real engine quantises its float logits at the ACL boundary.

Required Methods§

Source

fn prefill(&mut self, tokens: &[Token]) -> Result<u32>

Encode the (compressed) prompt; returns the resulting KV length.

Source

fn next_logits(&mut self, committed: &[Token]) -> Vec<i32>

Produce next-token logits given the committed context.

Source

fn eos_token(&self) -> Token

The end-of-sequence token id.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§