Trait InferenceEngine 

pub trait InferenceEngine {
    // Required methods
    fn prefill(&mut self, tokens: &[Token]) -> Result<u32>;
    fn next_logits(&mut self, committed: &[Token]) -> Vec<i32>;
    fn eos_token(&self) -> Token;
    fn rollback(&mut self, keep_committed: u32) -> Result<()>;
    fn reset_cache(&mut self) -> Result<()>;

    // Provided method
    fn prefill_reuse(&mut self, full_context: &[Token]) -> Result<u32> { ... }
}

The inference engine adapter (RuntimeAcl). Implemented for real by Candle in the excluded adapter el-engine-candle (ADR-002).

Logits are integer milli-logits to keep the orchestrator deterministic and float-free; a real engine quantises its float logits at the ACL boundary.

Required Methods

fn prefill(&mut self, tokens: &[Token]) -> Result<u32>

Encode the (compressed) prompt; returns the resulting KV length.

fn next_logits(&mut self, committed: &[Token]) -> Vec<i32>

Produce next-token logits given the committed context.

fn eos_token(&self) -> Token

The end-of-sequence token id.

fn rollback(&mut self, keep_committed: u32) -> Result<()>

Roll the engine’s internal state back so its context is exactly the prompt plus keep_committed generated tokens.

The ADR-012 control loop truncates the session’s committed output and KV descriptors on a safety backtrack. A stateful engine (one holding a real KV cache and position counters, e.g. a transformer) must mirror that truncation here — otherwise it keeps serving logits from the abandoned (unsafe) branch and never re-feeds the replacement tokens, so the rollback is silently a no-op at the engine level.

After Ok(()), the next next_logits call — passed a committed slice of length keep_committed — must produce logits consistent with that prefix. Returning Err makes the loop fail closed rather than resume on an inconsistent cache.

This method is required, with no default, deliberately: a default no-op would let a stateful adapter that forgot to override silently resume on a stale KV cache — a safety bug that fails open. Every engine must make the choice explicit. A stateless engine whose next_logits recomputes purely from committed implements it as Ok(()).

fn reset_cache(&mut self) -> Result<()>

Return the engine to its pristine, pre-prefill state so the same loaded weights can serve a new conversation without being reloaded (ADR-018). After Ok(()), the next prefill must build a KV cache from scratch as if the engine had just been constructed.

This is distinct from rollback: rollback(keep) rewinds within a generation to a safe prefix of length keep (ADR-012); reset_cache discards the whole conversation. It is what lets a provider hold one resident model and reuse it across turns instead of re-reading the weights from disk every call.

Like rollback, it is required with no default: a stateful adapter that forgot to override would otherwise carry a stale cache into the next conversation. A stateless engine whose next_logits recomputes purely from committed implements it as Ok(()).

Provided Methods

fn prefill_reuse(&mut self, full_context: &[Token]) -> Result<u32>

Prefill full_context, reusing the KV already cached for its longest matching prefix and feeding only the divergent suffix — cross-turn incremental prefill (ADR-018 AC-3). Returns the resulting KV length.

This is the engine half of [InferenceSession::continue_prompt]: on a follow-up turn the whole conversation is re-rendered and re-tokenized, but a stateful engine that still holds the prior turn’s KV can skip re-encoding the unchanged prefix. The token-level prefix match against the live cache is the tokenizer-round-trip guard — if the re-tokenized context diverges from what was cached (decode→encode is not always identity), reuse stops at the divergence and the suffix is fed fresh.

Soundness contract. After Ok, the engine MUST be in the exact state a reset_cache() + prefill(full_context) would have left it — identical logits for any subsequent next_logits. Reuse is purely a compute optimisation; it must never change what the cache represents, so the runtime’s safety checks (which re-run over full_context every turn) see identical data.

Unlike rollback/reset_cache, a wrong implementation here is a correctness/perf regression, not a safety fail-open — so this has a safe default: discard the cache and re-prefill the whole context (no reuse). Stateful engines override it for the fast path; stateless engines (whose next_logits recomputes from committed) inherit the default unchanged.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl InferenceEngine for NullEngine