Trait PersistentDecodeModel 

pub trait PersistentDecodeModel {
    // Required methods
    fn decode_n_layers(&self) -> usize;
    fn build_decode_token_data(
        &self,
        device: &Device,
        cached_len: usize,
        tokens: &[u32],
        session: &DecodeSession,
        cache: &KvCache,
        rope_inv_freq: Option<&[f64]>,
    ) -> Result<DecodeTokenData>;
}

The per-model variation points of one persistent-decode rebind step.

Deliberately one data hook plus a layer count, rather than a hook per difference. The three ways the Llama and Phi copies differed were: an extra rope_inv_freq argument, an extra cache_dtype + device-offset argument pair, and the SymEnv construction. The first two are inputs to token-data construction and collapse into the hook; the third dissolved — see crate::inference_context::DecodeSession::per_token_sym_env, whose doc carries the measurement and the do-not-delete warning.

Required Methods

fn decode_n_layers(&self) -> usize

Transformer layers whose K/V versions are bumped after a step.

fn build_decode_token_data( &self, device: &Device, cached_len: usize, tokens: &[u32], session: &DecodeSession, cache: &KvCache, rope_inv_freq: Option<&[f64]>, ) -> Result<DecodeTokenData>

Recompute this token’s device-resident data Consts (token ids, RoPE cos/sin, mask, and — only on the device-offset path — the KV-write offset).

This is where the per-model path divergence lives and stays. An impl reads session.offset_node().is_some() if it needs to; the driver does not, so it cannot accidentally normalise the two models onto one path.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors