Struct ModernBertLayerWeights 

pub struct ModernBertLayerWeights<T> {
    pub qkv_weight: T,
    pub output_weight: T,
    pub attn_norm_weight: Option<T>,
    pub mlp_wi_weight: T,
    pub mlp_wo_weight: T,
    pub mlp_norm_weight: T,
    pub is_global: bool,
}

Weights for one ModernBERT encoder layer.

All projections are bias-free. The QKV weight is a fused [3*hidden, hidden] matrix. The MLP Wi is [2*intermediate, hidden] and chunks into value + gate halves for the GeGLU activation.

Fields

qkv_weight: T

Fused Q+K+V projection weight [3*hidden, hidden] – no bias.

output_weight: T

Attention output projection weight [hidden, hidden] – no bias.

attn_norm_weight: Option<T>

Pre-attention LayerNorm weight [hidden] – None for layer 0 (identity).

mlp_wi_weight: T

Gated MLP input weight [2*intermediate, hidden] – chunks to value[intermediate] + gate[intermediate] for GeGLU.

mlp_wo_weight: T

MLP output projection weight [hidden, intermediate] – no bias.

mlp_norm_weight: T

Pre-MLP LayerNorm weight [hidden].

is_global: bool

Whether this layer uses global (full) or local (sliding window) attention.