Struct ModernBertWeights 

pub struct ModernBertWeights<T> {
    pub tok_embeddings: T,
    pub emb_norm_weight: T,
    pub final_norm_weight: T,
    pub zero_bias: T,
    pub layers: Vec<ModernBertLayerWeights<T>>,
    pub num_heads: usize,
    pub head_dim: usize,
    pub hidden_dim: usize,
    pub intermediate_dim: usize,
    pub layer_norm_eps: f32,
    pub local_window: usize,
}

Full ModernBERT model weights, generic over tensor type.

Includes embedding table, per-layer encoder weights, final norm, and model geometry. The tensor type T becomes Driver::Tensor when loaded onto a specific backend.

Fields

tok_embeddings: T

Word embedding table [vocab_size, hidden].

emb_norm_weight: T

Post-embedding LayerNorm weight [hidden] (no bias).

final_norm_weight: T

Final LayerNorm weight [hidden] applied before pooling (no bias).

zero_bias: T

A zero-filled tensor [hidden] used as dummy bias for LayerNorm calls.

The Driver::layer_norm API requires a bias tensor; ModernBERT has none, so we pass this zero buffer instead.

layers: Vec<ModernBertLayerWeights<T>>

Per-layer encoder weights.

num_heads: usize

Number of attention heads (12 for modernbert-embed-base).

head_dim: usize

Dimension per attention head (hidden / num_heads, 64).

hidden_dim: usize

Hidden dimension (768 for modernbert-embed-base).

intermediate_dim: usize

MLP intermediate dimension (1152 for modernbert-embed-base).

layer_norm_eps: f32

Layer normalization epsilon (1e-5).

local_window: usize

Sliding window size for local attention layers (128).