Struct Qwen3Runner 

pub struct Qwen3Runner { /* private fields */ }

Resolved Qwen3 runner — call Qwen3Runner::generate for streaming decode (F32 path), or Qwen3Runner::predict_logits for a single forward pass (works in both F32 and packed modes).

Implementations

impl Qwen3Runner

pub fn builder() -> Qwen3RunnerBuilder

pub fn config(&self) -> &Qwen3Config

pub fn device(&self) -> Device

pub fn disable_decode_compile_cache(&mut self)

Bypass the cached decode path; every generated token re-runs the full prefill graph from scratch. Slow (O(N²)) but a reference for numerical parity checks against the cached path.

pub fn predict_logits(&mut self, prompt_ids: &[u32]) -> Result<Vec<f32>, Error>

Generate n_new tokens after the given prompt. on_token is called once per generated id when stream(true) is set; otherwise the callback fires once at the end with the full vector. Returns the full generated id sequence.

The prompt is expected as raw token ids — tokenizer integration lives outside this module today (use the example binary for an end-to-end pipeline that wires tokenizers). Run a single prefill pass and return the last-position logits. Works in both F32 mode and packed-weights mode — in packed mode this is the only forward path supported today (streaming decode still goes through the F32 generator).

The prompt length must match the bucket the runner was built for (max_seq); shorter prompts are padded with the first token, longer prompts are truncated.

pub fn generate_packed( &mut self, prompt_ids: &[u32], n_new: usize, on_token: impl FnMut(u32), ) -> Result<Vec<u32>, Error>

Generate n_new tokens via repeated packed-mode prefills. Each step runs the full prefill graph against the growing token history (padded/truncated to max_seq), samples the next id, and appends it. Calls on_token per id.

Trade-off vs generate() on the F32 path: every token pays a full prefill instead of one decode step, so wall-clock throughput is ~max_seq × slower. Memory stays packed though — the only path that actually loads 14 B+ Q4_K_M GGUFs on a 32 GB Mac today. Tighter throughput needs the real bucketed decode-graph machinery (separate TODO; see CHANGELOG known-limitations).

pub fn generate( &mut self, prompt_ids: &[u32], n_new: usize, on_token: impl FnMut(u32), ) -> Result<Vec<u32>, Error>

pub fn generate_stoppable( &mut self, prompt_ids: &[u32], n_new: usize, on_token: impl FnMut(u32) -> bool, ) -> Result<Vec<u32>, Error>

Like [generate] but the callback can return false to stop sampling early (e.g. on EOS).

Trait Implementations

impl LmRunner for Qwen3Runner

fn family(&self) -> &'static str

Short family identifier ("qwen3", "llama32", "gemma").

fn vocab_size(&self) -> usize

LM head vocabulary size.

fn predict_logits(&mut self, prompt_ids: &[u32]) -> Result<Vec<f32>, Error>

Run prefill on prompt_ids and return last-token logits.

fn generate( &mut self, prompt_ids: &[u32], n_new: usize, on_token: &mut dyn FnMut(u32) -> bool, ) -> Result<Vec<u32>, Error>

Generate up to n_new tokens after prompt_ids using greedy (argmax) sampling. The default impl re-prefills on the full context each step — per-family runners should override with their cached decode fast path.

fn supports_multimodal(&self) -> bool

Whether this runner supports multimodal (image+text) generation.

fn generate_multimodal( &mut self, _prompt: &str, _rgb: &[u8], _img_w: usize, _img_h: usize, _tokenizer: Option<&Path>, _n_new: usize, _on_token: &mut dyn FnMut(u32) -> bool, ) -> Result<Vec<u32>, Error>

Multimodal generation: prefill with text where image markers are spliced with vision embeddings derived from rgb.