Struct QwenChatProvider 

pub struct QwenChatProvider { /* private fields */ }

A real local chat backend: a Qwen2 GGUF model + its tokenizer, driven through el_runtime::InferenceSession.

Each chat call renders the whole conversation to Qwen2.5 ChatML, builds a fresh QwenEngine (candle has no public KV-cache reset), then runs the SDK’s standard provenance-gated session: load_prompt (prefill) → generate (grammar mask → safety steer → greedy commit). The provider holds no mutable session state, so it is Send + Sync without locking.

Implementations

impl QwenChatProvider

pub fn from_paths( model_path: impl AsRef<Path>, tokenizer_path: impl AsRef<Path>, ) -> Result<Self>

Load a Qwen2 GGUF model and its tokenizer.json from local paths.

Trait Implementations

impl LlmProvider for QwenChatProvider

fn chat(&self, req: &ChatRequest) -> Result<ChatResponse>

Blocking, non-streaming chat completion.

fn chat_stream( &self, req: &ChatRequest, on_token: &mut dyn FnMut(ChatToken), ) -> Result<()>

Streaming chat: calls on_token for each fragment as it arrives. Returns when generation is complete or on error. The final call will have ChatToken::is_final == true.