Struct QwenEngine 

pub struct QwenEngine { /* private fields */ }

A real Qwen2 transformer InferenceEngine.

Holds candle’s stateful KV cache. Within one generation it is fed incrementally (prefill, then one new token per next_logits call). The engine is loaded once and reused across conversations (ADR-018): candle exposes no public cache-clear, but its attention replaces the cache on a forward at index_pos == 0, so reset_cache evicts the previous conversation’s KV with a single benign position-0 forward — no engine reconstruction and no reload from disk between turns.

Across turns of the same conversation it also reuses the unchanged prefix’s KV: prefill_reuse feeds only the suffix the re-rendered conversation adds beyond cached, the live token sequence behind the cache (ADR-018 AC-3 cross-turn incremental prefill).

A within-generation safety backtrack (ADR-012) is supported via InferenceEngine::rollback: candle’s attention discards its cache when a forward runs at index_pos == 0, so we retain the prompt and replay it from position 0 to rebuild the cache for the safe prefix (the session then re-feeds the retained committed tokens). Float logits are quantised to integer milli-logits at the seam, exactly like CandleEngine, so the runtime stays float-free.

Implementations

impl QwenEngine

pub fn from_path(path: impl AsRef<Path>, eos: Token) -> Result<Self>

Load Qwen2 weights from a consumer-supplied GGUF file.

Trait Implementations

impl InferenceEngine for QwenEngine

fn reset_cache(&mut self) -> Result<()>

Release the current conversation’s KV while keeping the resident weights loaded (ADR-018) — the separation of conversation lifecycle from model lifecycle, and the engine half of InferenceSession::close / reset.

candle’s quantized_qwen2 owns its per-layer KV with no public clear API, but its attention ignores and replaces the cache on a forward at index_pos == 0. So one forward over a benign token (id 0) drops the prior (user) K/V tensors — freeing that memory and clearing the user’s data from the cache (PRD line 131) — without touching the weights. What remains is a single non-user token’s KV, itself overwritten by the next prefill or freed when the engine is dropped. Skipped when nothing has been cached yet (index_pos == 0), so a pristine or already-cleared engine does no work.

Distinct from rollback, which replays a retained prefix to rewind within a single generation. Fallible (it runs a forward); on error the caller (reset/close) leaves session state untouched and surfaces it.

fn prefill_reuse(&mut self, full_context: &[Token]) -> Result<u32>

Cross-turn incremental prefill (ADR-018 AC-3): reuse the KV already cached for the longest prefix full_context shares with the live cache, and feed only the divergent suffix at the live position — no reload, no whole-history re-prefill.

cached is the exact token sequence behind the current KV (length == index_pos). The token-level longest-common-prefix against it is the tokenizer-round-trip guard: a re-rendered+re-tokenized conversation that drifts from what was generated simply matches a shorter prefix and the rest is fed fresh. When full_context exactly extends the cache, only the new tail is forwarded (the fast path); otherwise — divergence, or a context shorter than the cache — candle cannot truncate its append-only cache, so we rebuild from position 0 (a forward at index_pos == 0 drops the old cache), which is never worse than the pre-ADR-018 full re-prefill.

Either branch leaves the engine in the same state a reset_cache() + prefill(full_context) would: the suffix is fed by the identical forward_one calls at the identical positions, so subsequent logits are bit-identical to a from-scratch prefill (the soundness contract).

fn prefill(&mut self, tokens: &[Token]) -> Result<u32>

Encode the (compressed) prompt; returns the resulting KV length.

fn next_logits(&mut self, committed: &[Token]) -> Vec<i32>

Produce next-token logits given the committed context.

fn eos_token(&self) -> Token

The end-of-sequence token id.

fn rollback(&mut self, _keep_committed: u32) -> Result<()>

Roll the engine’s internal state back so its context is exactly the prompt plus keep_committed generated tokens.