Struct AttnWeights 

pub struct AttnWeights {
    pub q_proj: WeightMatrix,
    pub k_proj: WeightMatrix,
    pub v_proj: WeightMatrix,
    pub o_proj: WeightMatrix,
    pub norm_weight: Vec<f32>,
    pub q_norm: Option<Vec<f32>>,
    pub k_norm: Option<Vec<f32>>,
    pub q_bias: Option<Vec<f32>>,
    pub k_bias: Option<Vec<f32>>,
    pub v_bias: Option<Vec<f32>>,
    pub post_attn_norm: Option<Vec<f32>>,
    pub post_ffn_norm: Option<Vec<f32>>,
}

Fields

q_proj: WeightMatrix

k_proj: WeightMatrix

v_proj: WeightMatrix

o_proj: WeightMatrix

norm_weight: Vec<f32>

q_norm: Option<Vec<f32>>

OLMoE-style QK-RMSNorm (attn_q_norm/attn_k_norm GGUF tensors), applied to the whole q_proj/k_proj output (width n_heads*head_dim / n_kv_heads*head_dim) before RoPE – confirmed against OlmoeAttention.forward in transformers/models/olmoe/modeling_olmoe.py (q_norm(q_proj(x)), k_norm(k_proj(x)), both plain whole-vector RMSNorm, not per-head). None for every model that doesn’t ship these tensors – absent, not zero/identity-weighted, so existing presets/fixtures are byte-for-byte unaffected.

Qwen3 / Gemma3 ship the same tensor names with length head_dim (per-head). Which style is used is selected by ModelConfig::qk_norm_style (refined at load from weight length).

k_norm: Option<Vec<f32>>

q_bias: Option<Vec<f32>>

Qwen2/Qwen2-MoE-family QKV attention bias (attn_{q,k,v}.bias GGUF tensors, real config.qkv_bias), added elementwise to the corresponding projection’s output before QK-norm/RoPE – confirmed against the real transformers source (Qwen2MoeAttention.__init__: q_proj = nn.Linear(..., bias= config.qkv_bias), same for k_proj/v_proj; o_proj has no bias). Found as a real, previously-unhandled architecture gap: ferrox’s generic GGUF loader silently ignored these real tensors entirely, producing fluent-but-wrong output on a real downloaded Qwen1.5-MoE checkpoint (same failure class as OLMoE’s missing QK-norm). None for every model that doesn’t ship these tensors.

k_bias: Option<Vec<f32>>

v_bias: Option<Vec<f32>>

post_attn_norm: Option<Vec<f32>>

Gemma 2+/3 post-attention RMSNorm (blk.N.post_attention_norm.weight / llama.cpp attn_post_norm). Applied to attention output before the residual add. None for Llama/Qwen/OLMoE.

post_ffn_norm: Option<Vec<f32>>

Gemma 2+/3 post-FFN RMSNorm (blk.N.post_ffw_norm.weight).