Struct QwenEngine 

pub struct QwenEngine { /* private fields */ }

A real Qwen2 transformer InferenceEngine.

Holds candle’s stateful KV cache. Within one generation it is fed incrementally (prefill, then one new token per next_logits call). The engine is loaded once and reused across conversations (ADR-018): candle exposes no public cache-clear, but its attention replaces the cache on a forward at index_pos == 0, so reset_cache evicts the previous conversation’s KV with a single benign position-0 forward — no engine reconstruction and no reload from disk between turns.

A within-generation safety backtrack (ADR-012) is supported via InferenceEngine::rollback: candle’s attention discards its cache when a forward runs at index_pos == 0, so we retain the prompt and replay it from position 0 to rebuild the cache for the safe prefix (the session then re-feeds the retained committed tokens). Float logits are quantised to integer milli-logits at the seam, exactly like CandleEngine, so the runtime stays float-free.

Implementations

impl QwenEngine

pub fn from_path(path: impl AsRef<Path>, eos: Token) -> Result<Self>

Load Qwen2 weights from a consumer-supplied GGUF file.

Trait Implementations

impl InferenceEngine for QwenEngine

fn reset_cache(&mut self) -> Result<()>

Release the current conversation’s KV while keeping the resident weights loaded (ADR-018) — the separation of conversation lifecycle from model lifecycle, and the engine half of InferenceSession::close / reset.

candle’s quantized_qwen2 owns its per-layer KV with no public clear API, but its attention ignores and replaces the cache on a forward at index_pos == 0. So one forward over a benign token (id 0) drops the prior (user) K/V tensors — freeing that memory and clearing the user’s data from the cache (PRD line 131) — without touching the weights. What remains is a single non-user token’s KV, itself overwritten by the next prefill or freed when the engine is dropped. Skipped when nothing has been cached yet (index_pos == 0), so a pristine or already-cleared engine does no work.

Distinct from rollback, which replays a retained prefix to rewind within a single generation. Fallible (it runs a forward); on error the caller (reset/close) leaves session state untouched and surfaces it.

fn prefill(&mut self, tokens: &[Token]) -> Result<u32>

Encode the (compressed) prompt; returns the resulting KV length.

fn next_logits(&mut self, committed: &[Token]) -> Vec<i32>

Produce next-token logits given the committed context.

fn eos_token(&self) -> Token

The end-of-sequence token id.

fn rollback(&mut self, _keep_committed: u32) -> Result<()>

Roll the engine’s internal state back so its context is exactly the prompt plus keep_committed generated tokens.