Struct AttnWeights 

pub struct AttnWeights {

    pub q_proj: WeightMatrix,
    pub k_proj: WeightMatrix,
    pub v_proj: WeightMatrix,
    pub o_proj: WeightMatrix,
    pub norm_weight: NormOp,
    pub q_norm: Option<Vec<f32>>,
    pub k_norm: Option<Vec<f32>>,
    pub q_bias: Option<Vec<f32>>,
    pub k_bias: Option<Vec<f32>>,
    pub v_bias: Option<Vec<f32>>,
    pub post_attn_norm: Option<Vec<f32>>,
    pub post_ffn_norm: Option<Vec<f32>>,
    pub output_gate: Option<AttnGate>,
    pub sinks: Option<Vec<f32>>,
    pub attn_sub_norm: Option<Vec<f32>>,
    pub o_scale: Option<f32>,
    pub o_bias: Option<Vec<f32>>,
    pub shortconv: Option<ShortConv>,
    pub ssm: Option<SsmBlock>,
    pub q_gate_interleaved: bool,
}

Fields

q_proj: WeightMatrix

k_proj: WeightMatrix

v_proj: WeightMatrix

o_proj: WeightMatrix

norm_weight: NormOp

The PRE-attention norm, or NormOp::None for the post-norm-only topology (olmo2 / exaone4), which projects Q/K/V straight off the raw residual. See crate::norm.

q_norm: Option<Vec<f32>>

OLMoE-style QK-RMSNorm (attn_q_norm/attn_k_norm GGUF tensors), applied to the whole q_proj/k_proj output (width n_heads*head_dim / n_kv_heads*head_dim) before RoPE – confirmed against OlmoeAttention.forward in transformers/models/olmoe/modeling_olmoe.py (q_norm(q_proj(x)), k_norm(k_proj(x)), both plain whole-vector RMSNorm, not per-head). None for every model that doesn’t ship these tensors – absent, not zero/identity-weighted, so existing presets/fixtures are byte-for-byte unaffected.

Qwen3 / Gemma3 ship the same tensor names with length head_dim (per-head). Which style is used is selected by ModelConfig::qk_norm_style (refined at load from weight length).

k_norm: Option<Vec<f32>>

q_bias: Option<Vec<f32>>

Qwen2/Qwen2-MoE-family QKV attention bias (attn_{q,k,v}.bias GGUF tensors, real config.qkv_bias), added elementwise to the corresponding projection’s output before QK-norm/RoPE – confirmed against the real transformers source (Qwen2MoeAttention.__init__: q_proj = nn.Linear(..., bias= config.qkv_bias), same for k_proj/v_proj; o_proj has no bias). Found as a real, previously-unhandled architecture gap: frink’s generic GGUF loader silently ignored these real tensors entirely, producing fluent-but-wrong output on a real downloaded Qwen1.5-MoE checkpoint (same failure class as OLMoE’s missing QK-norm). None for every model that doesn’t ship these tensors.

k_bias: Option<Vec<f32>>

v_bias: Option<Vec<f32>>

post_attn_norm: Option<Vec<f32>>

Gemma 2+/3 post-attention RMSNorm (blk.N.post_attention_norm.weight / llama.cpp attn_post_norm). Applied to attention output before the residual add. None for Llama/Qwen/OLMoE.

post_ffn_norm: Option<Vec<f32>>

Gemma 2+/3 post-FFN RMSNorm (blk.N.post_ffw_norm.weight).

output_gate: Option<AttnGate>

The learned output gate, blk.N.attn_gate.weight, applied to the attention output before o_proj (afmoe, laguna, step35). See crate::attn_gate for the two axes it varies on and the one input it always reads. None for every architecture whose graph has no such op; a file carrying the tensor on one of those is refused as unconsumed rather than gated.

sinks: Option<Vec<f32>>

blk.N.attn_sinks.weight, one learned logit per query head that joins every softmax and contributes nothing to the output (ggml_soft_max_add_sinks, llama-graph.cpp:2600).

Used to live on the gpt-oss side table alone, which spelled the rule as “arch is gpt-oss”. Four llama.cpp graphs pass this tensor into build_attn – openai-moe.cpp:115, mimo2.cpp:177, dflash.cpp, deepseek4.cpp – through the SAME build_attn_mha path, so the rule is “the tensor is present”. gpt-oss requires it (openai-moe.cpp:44) and loader.rs still refuses a gpt-oss file without one; the fused Metal launches refuse any layer that has one, by the exhaustive destructure in Decoder::metal_attn_view.

attn_sub_norm: Option<Vec<f32>>

BitNet’s blk.N.attn_sub_norm.weight, [hidden_dim]: an RMSNorm on the attention output BEFORE o_proj (bitnet.cpp:101-106), the other side of that matmul from post_attn_norm. Loaded only for a model whose ModelConfig::block_sub_norms says so (crate::sub_norms); the fused Metal launches refuse the model through metal_can_serve_model and the layer through the exhaustive destructure in Decoder::metal_attn_view.

o_scale: Option<f32>

blk.N.attn_output.scale, the {1} companion build_lora_mm multiplies the attention branch by right after wo (llama-graph.cpp:1492-1494; talkie writes it on every export, crate::weight_scales). The fused Metal launches refuse a layer that has one through the destructure in metal_attn_view.

o_bias: Option<Vec<f32>>

blk.N.attn_output.bias, added right after wo (and after o_scale, the order build_attn has). Loaded for the architectures whose graph creates the tensor (crate::proj_bias::ATTN_OUT_BIAS_CREATORS), which includes gpt-oss, whose bias used to live on its side table; None everywhere else, where a present tensor is refused as unread. The fused Metal launches refuse a layer that has one through the destructure in metal_attn_view.

shortconv: Option<ShortConv>

LFM2’s short convolution, Some on exactly the layers whose shape is AttnShape::ShortConv (crate::shortconv); the four projections above are empty on such a layer. The host bodies branch on the SHAPE and reach this field through it; the fused Metal launches refuse the model (a short-conv model is never uniform) and the layer (the destructure in metal_attn_view).

ssm: Option<SsmBlock>

The state-space block (crate::ssm_block): Some on exactly the layers whose shape is AttnShape::Mamba1 / Mamba2, and on every GQA layer of a ModelConfig::parallel_ssm model (crate::mamba2::PARALLEL_WITH_ATTENTION); the same rules as shortconv otherwise.

q_gate_interleaved: bool

attn_q is 2 * n_heads * head_dim wide, each head’s [q, gate] interleaved, and sigmoid(gate) multiplies the attention output before wo (crate::attn_gate::Q_INTERLEAVED_GATE_ARCHS). The projection stays one matrix; the three host bodies split its output. The fused Metal launches refuse the layer.