Skip to main content

InferenceEngine

el_runtime

Trait InferenceEngine 

Source 
pub trait InferenceEngine {
    // Required methods
    fn prefill(&mut self, tokens: &[Token]) -> Result<u32>;
    fn next_logits(&mut self, committed: &[Token]) -> Vec<i32>;
    fn eos_token(&self) -> Token;
    fn rollback(&mut self, keep_committed: u32) -> Result<()>;
}
Expand description

The inference engine adapter (RuntimeAcl). Implemented for real by Candle in the excluded adapter el-engine-candle (ADR-002).

Logits are integer milli-logits to keep the orchestrator deterministic and float-free; a real engine quantises its float logits at the ACL boundary.

Required Methods§

Source

fn prefill(&mut self, tokens: &[Token]) -> Result<u32>

Encode the (compressed) prompt; returns the resulting KV length.

Source

fn next_logits(&mut self, committed: &[Token]) -> Vec<i32>

Produce next-token logits given the committed context.

Source

fn eos_token(&self) -> Token

The end-of-sequence token id.

Source

fn rollback(&mut self, keep_committed: u32) -> Result<()>

Roll the engine’s internal state back so its context is exactly the prompt plus keep_committed generated tokens.

The ADR-012 control loop truncates the session’s committed output and KV descriptors on a safety backtrack. A stateful engine (one holding a real KV cache and position counters, e.g. a transformer) must mirror that truncation here — otherwise it keeps serving logits from the abandoned (unsafe) branch and never re-feeds the replacement tokens, so the rollback is silently a no-op at the engine level.

After Ok(()), the next next_logits call — passed a committed slice of length keep_committed — must produce logits consistent with that prefix. Returning Err makes the loop fail closed rather than resume on an inconsistent cache.

This method is required, with no default, deliberately: a default no-op would let a stateful adapter that forgot to override silently resume on a stale KV cache — a safety bug that fails open. Every engine must make the choice explicit. A stateless engine whose next_logits recomputes purely from committed implements it as Ok(()).

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§