Struct MlaConfig 

pub struct MlaConfig {
    pub num_heads: usize,
    pub q_lora_rank: usize,
    pub kv_lora_rank: usize,
    pub qk_nope_head_dim: usize,
    pub qk_rope_head_dim: usize,
    pub v_head_dim: usize,
    pub use_output_gate: bool,
    pub rope: Option<MlaRopeConfig>,
}

Gated MLA (multi-head latent attention) hyperparameters, verified against Kimi K3’s real config.json text_config block and the real KimiMLAAttention reference implementation (modeling_kimi_linear.py).

Fields

num_heads: usize

q_lora_rank: usize

kv_lora_rank: usize

qk_nope_head_dim: usize

qk_rope_head_dim: usize

v_head_dim: usize

use_output_gate: bool

Kimi K3’s addition on top of standard DeepSeek-style MLA: attn_output *= sigmoid(g_proj(hidden_states)) before o_proj.

rope: Option<MlaRopeConfig>

None reproduces Kimi K3’s real, confirmed behavior: no rotary embedding at all (mla.rs’s module doc comment; the real KimiMLAAttention.forward asserts use_nope and never calls a rotary function). Some is for architectures whose decoupled q_rot/k_rot slices genuinely are position-rotated – e.g. GLM-5.2, whose real config.json (zai-org/GLM-5.2) sets rope_interleave: true for its main attention (confirmed against llama.cpp PR #25407’s LLAMA_ROPE_TYPE_NORM rope call on q_pe/k_pe in src/models/glm-dsa.cpp).

Trait Implementations

impl Clone for MlaConfig

fn clone(&self) -> MlaConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for MlaConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.