Enum KvLayout 

pub enum KvLayout {
    Gqa {
        n_kv_heads: usize,
        head_dim: usize,
    },
    MlaLatent {
        kv_lora_rank: usize,
        qk_rope_head_dim: usize,
    },
    MlaExpanded {
        n_heads: usize,
        k_head_dim: usize,
        v_head_dim: usize,
    },
}

How one layer’s KV cache is shaped. Which variant applies is a property of the decoder that will run, not of the architecture name – see KvLayout::MlaLatent’s doc comment for the one place that distinction bites.

Variants

Gqa

Multi-head / grouped-query attention: one K vector and one V vector of n_kv_heads * head_dim per token, per layer. MHA is just the n_kv_heads == n_heads case – there is no separate variant for it, and the halving GQA buys shows up entirely in n_kv_heads.

Fields

n_kv_heads: usize

head_dim: usize

MlaLatent

MLA in its absorbed form: the cache holds only the compressed latent plus the decoupled RoPE slice, kv_lora_rank + rope_dim scalars per token per layer, and K/V are reconstructed from it on the fly. One vector, not two – there is no * 2 here.

ferrox does not run this form today. mla::mla_forward_token (and therefore kimi_decoder, glm_dsa, glm52_decoder) caches the expanded per-head K and V, so a real ferrox MLA run costs KvLayout::MlaExpanded. This variant is what the absorbed form would cost, and is the right number to plan against only once a decoder actually caches the latent.

Fields

kv_lora_rank: usize

qk_rope_head_dim: usize

MlaExpanded

MLA as ferrox actually caches it: per-head K of qk_nope_head_dim + qk_rope_head_dim and per-head V of v_head_dim, both materialised (mla::mla_forward_token’s k_cache/v_cache). K and V head dims differ, which is exactly why this cannot reuse the Gqa arm.

Fields

n_heads: usize

k_head_dim: usize

v_head_dim: usize

Implementations

impl KvLayout

pub fn elems_per_token_per_layer(self) -> u64

Cached scalars one token contributes to one layer.

pub fn describe(self) -> String

One-line description of the arithmetic, for the report a user reads when they want to know why they got the context they got.

Trait Implementations

impl Clone for KvLayout

fn clone(&self) -> KvLayout

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for KvLayout

impl Debug for KvLayout

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for KvLayout

impl PartialEq for KvLayout

fn eq(&self, other: &KvLayout) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for KvLayout