Trait LmRunner 

pub trait LmRunner: Send {
    // Required methods
    fn family(&self) -> &'static str;
    fn vocab_size(&self) -> usize;
    fn predict_logits(&mut self, prompt_ids: &[u32]) -> Result<Vec<f32>, Error>;

    // Provided methods
    fn generate(
        &mut self,
        prompt_ids: &[u32],
        n_new: usize,
        on_token: &mut dyn FnMut(u32) -> bool,
    ) -> Result<Vec<u32>, Error> { ... }
    fn prefill_logits(&mut self, _prompt_ids: &[u32]) -> Result<Vec<f32>, Error> { ... }
    fn decode_logits(&mut self, _token: u32) -> Result<Vec<f32>, Error> { ... }
    fn prefill_logits_reusing(
        &mut self,
        prompt: &[u32],
        _snap: &SessionSnapshot,
        _reuse_len: usize,
    ) -> Result<Vec<f32>, Error> { ... }
    fn export_session(&self) -> Option<SessionSnapshot> { ... }
    fn restore_session(&mut self, _snap: &SessionSnapshot) -> bool { ... }
    fn supports_multimodal(&self) -> bool { ... }
    fn generate_multimodal(
        &mut self,
        _prompt: &str,
        _rgb: &[u8],
        _img_w: usize,
        _img_h: usize,
        _tokenizer: Option<&Path>,
        _n_new: usize,
        _on_token: &mut dyn FnMut(u32) -> bool,
    ) -> Result<Vec<u32>, Error> { ... }
}

Minimal per-family runner interface used by auto_dispatch and the rlx-text / skill integration.

Implementations must be Send so the boxed trait can move across threads (e.g. when a server runs inference on a worker pool). Sync is intentionally not required — runners hold mutable per-call compile / cache state.

Required Methods

fn family(&self) -> &'static str

Short family identifier ("qwen3", "llama32", "gemma").

fn vocab_size(&self) -> usize

LM head vocabulary size.

fn predict_logits(&mut self, prompt_ids: &[u32]) -> Result<Vec<f32>, Error>

Run prefill on prompt_ids and return last-token logits.

Provided Methods

fn generate( &mut self, prompt_ids: &[u32], n_new: usize, on_token: &mut dyn FnMut(u32) -> bool, ) -> Result<Vec<u32>, Error>

Generate up to n_new tokens after prompt_ids using greedy (argmax) sampling. The default impl re-prefills on the full context each step — per-family runners should override with their cached decode fast path.

on_token returns true to continue, false to stop.

fn prefill_logits(&mut self, _prompt_ids: &[u32]) -> Result<Vec<f32>, Error>

Prefill prompt_ids, seed the decode KV cache, and return the last-position logits [vocab]. Together with [decode_logits] this gives a host-driven decode loop: the caller owns sampling, logit bias, log-probs, and stop detection (used by the HTTP server). The default reports unsupported so existing runners keep compiling.

fn decode_logits(&mut self, _token: u32) -> Result<Vec<f32>, Error>

Feed one token, advance the KV cache, and return the next-position logits [vocab]. See LmRunner::prefill_logits. Default: unsupported.

fn prefill_logits_reusing( &mut self, prompt: &[u32], _snap: &SessionSnapshot, _reuse_len: usize, ) -> Result<Vec<f32>, Error>

Prefill prompt reusing a session snapshot that already covers its first reuse_len tokens — only the suffix is processed. Returns the last-position logits. The default ignores the snapshot and does a full prefill, so callers always get correct logits even without reuse.

fn export_session(&self) -> Option<SessionSnapshot>

Snapshot this runner’s decode session (KV cache + token history) for prompt-cache reuse. None ⇒ the family doesn’t support it.

fn restore_session(&mut self, _snap: &SessionSnapshot) -> bool

Restore a previously exported session so generation resumes from a cached prefix. Returns true if applied.

fn supports_multimodal(&self) -> bool

Whether this runner supports multimodal (image+text) generation.

fn generate_multimodal( &mut self, _prompt: &str, _rgb: &[u8], _img_w: usize, _img_h: usize, _tokenizer: Option<&Path>, _n_new: usize, _on_token: &mut dyn FnMut(u32) -> bool, ) -> Result<Vec<u32>, Error>

Multimodal generation: prefill with text where image markers are spliced with vision embeddings derived from rgb.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors