Trait LmRunner 

pub trait LmRunner: Send {
    // Required methods
    fn family(&self) -> &'static str;
    fn vocab_size(&self) -> usize;
    fn predict_logits(&mut self, prompt_ids: &[u32]) -> Result<Vec<f32>, Error>;

    // Provided methods
    fn generate(
        &mut self,
        prompt_ids: &[u32],
        n_new: usize,
        on_token: &mut dyn FnMut(u32) -> bool,
    ) -> Result<Vec<u32>, Error> { ... }
    fn supports_multimodal(&self) -> bool { ... }
    fn generate_multimodal(
        &mut self,
        _prompt: &str,
        _rgb: &[u8],
        _img_w: usize,
        _img_h: usize,
        _tokenizer: Option<&Path>,
        _n_new: usize,
        _on_token: &mut dyn FnMut(u32) -> bool,
    ) -> Result<Vec<u32>, Error> { ... }
}

Minimal per-family runner interface used by auto_dispatch and the rlx-text / skill integration.

Implementations must be Send so the boxed trait can move across threads (e.g. when a server runs inference on a worker pool). Sync is intentionally not required — runners hold mutable per-call compile / cache state.

Required Methods

fn family(&self) -> &'static str

Short family identifier ("qwen3", "llama32", "gemma").

fn vocab_size(&self) -> usize

LM head vocabulary size.

fn predict_logits(&mut self, prompt_ids: &[u32]) -> Result<Vec<f32>, Error>

Run prefill on prompt_ids and return last-token logits.

Provided Methods

fn generate( &mut self, prompt_ids: &[u32], n_new: usize, on_token: &mut dyn FnMut(u32) -> bool, ) -> Result<Vec<u32>, Error>

Generate up to n_new tokens after prompt_ids using greedy (argmax) sampling. The default impl re-prefills on the full context each step — per-family runners should override with their cached decode fast path.

on_token returns true to continue, false to stop.

fn supports_multimodal(&self) -> bool

Whether this runner supports multimodal (image+text) generation.

fn generate_multimodal( &mut self, _prompt: &str, _rgb: &[u8], _img_w: usize, _img_h: usize, _tokenizer: Option<&Path>, _n_new: usize, _on_token: &mut dyn FnMut(u32) -> bool, ) -> Result<Vec<u32>, Error>

Multimodal generation: prefill with text where image markers are spliced with vision embeddings derived from rgb.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors