Enum QkNormStyle 

pub enum QkNormStyle {
    WholeVector,
    PerHead,
    PerHeadScalar,
    PerHeadDistinct,
}

How attn_q_norm / attn_k_norm weights are applied (when present).

Variants

WholeVector

OLMoE: one RMSNorm over the full Q/K projection width.

PerHead

Qwen3 / Gemma3: RMSNorm per head with weight length head_dim.

PerHeadScalar

Talkie (talkie.cpp:26,82-91): RMSNorm per head, then ONE learned scalar per head for Q (attn_q_norm is {1, n_head}), and the same per-head RMSNorm with NO weight for K (:90, build_norm(Kcur, nullptr, ...)); there is no attn_k_norm tensor. Decided by architecture (PER_HEAD_SCALAR_QK_GAIN), not by the weight’s length: a file whose n_head == head_dim would make the length ambiguous. Applied after RoPE, as the graph does.

PerHeadDistinct

PLaMo-2 (plamo2.cpp:92-93,163,166): RMSNorm per head with a DISTINCT weight per head – attn_q_norm is {head_dim, n_head} and attn_k_norm {head_dim, n_head_kv}, and build_norm over the 3-d {head_dim, n_head, n_tokens} view norms each head and multiplies by that head’s row. The weight is n_heads * head_dim long, the same length as QkNormStyle::WholeVector’s, which is why it is decided by architecture (PER_HEAD_DISTINCT_QK_NORM) and not by the length rule.

Trait Implementations

impl Clone for QkNormStyle

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for QkNormStyle

impl Debug for QkNormStyle

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for QkNormStyle

fn default() -> Self

Returns the “default value” for a type.

impl Eq for QkNormStyle

impl PartialEq for QkNormStyle

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for QkNormStyle