Struct QwenChatProvider 

pub struct QwenChatProvider { /* private fields */ }

A real local chat backend: a Qwen2 GGUF model + its tokenizer, driven through el_runtime::InferenceSession.

The model weights are loaded once at construction and kept resident (ADR-018): each chat renders the whole conversation to Qwen2.5 ChatML and reuses one persistent provenance-gated session — a follow-up turn reuses the cached KV prefix and prefills only the new suffix (continue_prompt, AC-3 cross-turn incremental prefill), while a fresh turn uses load_prompt — then generate (grammar mask → safety steer → guard + checkpointed rollback → greedy commit), instead of rebuilding the engine and re-reading the GGUF every turn. On-device safety (ADR-005 Lightweight tier + the ADR-012 control loop) is on by default; see with_safety. The resident model lives behind a Mutex, so the provider stays Send + Sync and concurrent chat calls serialize on the one conversation.

Implementations

impl QwenChatProvider

pub fn from_paths( model_path: impl AsRef<Path>, tokenizer_path: impl AsRef<Path>, ) -> Result<Self>

Load a Qwen2 GGUF model and its tokenizer.json from local paths.

pub fn with_safety(self, mode: SafetyMode) -> Self

Select the on-device safety tier (ADR-005). SafetyMode::Off disables the steerer and the ADR-012 control loop (the plain single-pass decode); SafetyMode::Lightweight (the default) runs the token-anchor guard + hard-ban steerer + checkpointed rollback. SecDecoding/Csd need model assets not shipped here and fall back to the Lightweight wiring.

pub fn with_extra_guard_words<I, S>(self, words: I) -> Self

where I: IntoIterator<Item = S>, S: AsRef<str>,

Add extra words to the chunk guard’s unsafe patterns (resolved to token ids via this model’s tokenizer). Primarily a test/demo hook: e.g. --guard-word banana lets you watch the ADR-012 rollback / fail-closed refusal fire on a benign word, without needing the model to emit genuinely harmful content. Guard-only — these are not added to the hard-ban list, so the trajectory loop (not silent suppression) is what engages.

pub fn with_expert_model(self, path: impl AsRef<Path>) -> Self

Enable model-backed contrastive steering (ADR-013) with a safety expert GGUF (same tokenizer/family as the chat model). Steering runs only inside the early-token window. Pointing this at the chat model itself gives ~zero contrast (a no-op); a safety-tuned Qwen GGUF gives real steering. No effect under --safety off.

pub fn with_steer_alpha(self, alpha_milli: i32) -> Self

Contrastive steering strength ×1000 (1000 = 1.0×). Only meaningful with with_expert_model.

pub fn end_session(&self) -> Result<()>

End the current conversation, releasing its KV / output / prompt / buffered events while keeping the model resident (ADR-018 separation of conversation and model lifecycles; the AC-4 explicit release / PRD line 131 “KV caches … cleared on session end”). The next chat starts a fresh conversation on the same loaded weights — no reload. A no-op if no conversation has started yet. To free the weights too, drop the provider (Rust ownership).

Trait Implementations

impl LlmProvider for QwenChatProvider

fn chat(&self, req: &ChatRequest) -> Result<ChatResponse>

Blocking, non-streaming chat completion.

fn chat_stream( &self, req: &ChatRequest, on_token: &mut dyn FnMut(ChatToken), ) -> Result<()>

Streaming chat: calls on_token for each fragment as it arrives. Returns when generation is complete or on error. The final call will have ChatToken::is_final == true.