Struct MoeWeights 

pub struct MoeWeights {

    pub router: WeightMatrix,
    pub experts: ExpertBacking,
    pub shared_experts: Vec<ExpertWeights>,
    pub shared_expert_gate: Option<Vec<f32>>,
    pub norm_weight: NormOp,
    pub exp_probs_bias: Option<Vec<f32>>,
    pub ffn_sub_norm: Option<Vec<f32>>,
    pub down_scale: Option<f32>,
    pub dense_bias: Option<DenseBias>,
    pub exps_norm: Option<Vec<f32>>,
    pub parallel_sum_scale: Option<f32>,
    pub parallel: Option<ParallelNorm>,
    pub activation_counts: Vec<AtomicU64>,
}

Fields

router: WeightMatrix

experts: ExpertBacking

shared_experts: Vec<ExpertWeights>

shared_expert_gate: Option<Vec<f32>>

Qwen2-MoE-specific: when present, the shared experts’ combined output is scaled by sigmoid(shared_expert_gate . x) before being added to the routed output, instead of added unconditionally – confirmed against the real transformers source (Qwen2MoeSparseMoeBlock.forward: shared_expert_output = F.sigmoid(self.shared_expert_gate(hidden_states)) * shared_expert_output) and llama.cpp’s real qwen2moe.cpp (ffn_gate_inp_shexp dotted against the hidden state, sigmoid, multiplied into the shared-expert branch before the final add). Real on-disk shape is [hidden_dim] (a Linear(hidden_dim, 1, bias=false)’s weight, flattened – ggml’s real create_tensor call declares it as {n_embd}, not a 2D matrix), so this is a plain owned vector dotted with the normed hidden state directly, not a WeightMatrix. None for every other architecture (DeepSeek-V3’s shared experts, for one real confirmed contrast, add unconditionally with no gate at all).

norm_weight: NormOp

The PRE-FFN norm, or NormOp::None for the post-norm-only topology (olmo2 / exaone4), which runs the FFN on the raw post-attention residual. See crate::norm.

exp_probs_bias: Option<Vec<f32>>

DeepSeek-V3’s aux-loss-free expert-selection bias, on disk as blk.{N}.exp_probs_b.bias (llama.cpp’s LLM_TENSOR_FFN_EXP_PROBS_B – note the on-disk name has no ffn_ prefix, llama-arch.cpp:416). It is added to the selection score only: the top-k is taken over gating(logit) + bias[expert], while each winner’s combine weight comes from the unbiased gating(logit) (build_moe_ffn: “leave probs unbiased as it’s later used to get expert weights”). Biasing the weight too would silently skew every routed contribution away from what the router learned.

None for every checkpoint that does not ship the tensor. When it is present, the GPU MoE fast paths refuse the layer rather than route without it – their kernels have no bias input.

ffn_sub_norm: Option<Vec<f32>>

BitNet’s blk.N.ffn_sub_norm.weight, [ffn_dim]: an RMSNorm on silu(gate) * up BEFORE down (bitnet.cpp:135-140), inside the dense FFN. Some only for a model whose ModelConfig::block_sub_norms says so (crate::sub_norms), and only on a dense layer: no MoE graph has this site. Read by Decoder::run_dense_expert and Decoder::dense_ffn_batch, the two dense FFN bodies; the fused kernels never see it because metal_can_serve_model refuses the model.

down_scale: Option<f32>

blk.N.ffn_down.scale, the {1} companion multiplied onto the dense FFN’s output right after down (crate::weight_scales); refused on a routed layer, whose experts carry their own.

dense_bias: Option<DenseBias>

The dense FFN’s blk.N.ffn_{up,gate,down}.bias (crate::proj_bias), on a dense layer whose architecture’s graph creates them and whose file carries at least one; None otherwise. Applied by run_dense_expert and dense_ffn_batch, whose fused Metal launch has no bias site and is fenced on it.

exps_norm: Option<Vec<f32>>

Arctic’s blk.N.ffn_norm_exps.weight, [hidden_dim]: the SECOND per-layer norm, applied to the layer INPUT to make the routed branch’s operand (arctic.cpp:45,136-139; RouterInput::NormedLayerInput). REQUIRED on a routed layer of such a model, None everywhere else. Read by Decoder::router_operand, the one constructor of the operand.

parallel_sum_scale: Option<f32>

The factor on ffn_out + moe_out for a layer whose dense FFN is summed with its experts (crate::parallel_dense_ffn): Some only when the loader filled shared_experts from the dense names AND the row scales the sum (grok.cpp:180, sqrt(2)/2). Applied by the FFN bodies to the whole branch output before the post-FFN norm.

parallel: Option<ParallelNorm>

Some when this layer is a PARALLEL residual, `x + attn(norm(x))

• ffn(norm(x)), and under which norm the FFN reads the layer input (crate::parallel_residual): the vector attention read (SharedNorm, and then norm_weightisNormOp::Nonebecause there is no tensor) or its ownffn_norm of the layer input (TwoNorms). Noneis the sequential layer,ffn_norm(h)over the post-attention residual. Read by ONE constructor,Decoder::branch_inputs`, before attention runs.

activation_counts: Vec<AtomicU64>

How many times each routed expert (index into experts) has been selected by route_top_k across every forward_token/ forward_batch call so far. Real observed hotness, not a placeholder – feeds placement_plan below, which is what PlacementPlan::from_budget needs to prioritize actually-hot experts for GPU residency instead of guessing by index.

Implementations

impl MoeWeights

pub fn n_experts(&self) -> usize

pub fn expert_bytes(&self, e: usize) -> usize

This routed expert’s weight byte footprint, from resident matrices or the stored layout – identical numbers either way, so residency planning is backing-independent.

pub fn with_expert<R>(&self, e: usize, f: impl FnOnce(&ExpertWeights) -> R) -> R

Runs f against expert e’s weights, materializing them from the store first when this layer is store-backed. The lease (and therefore the cache entry’s pin) lives exactly as long as f’s borrow.

pub fn placement_plan(&self, vram_budget_bytes: u64) -> PlacementPlan

A real VRAM-budget-and-hotness-driven placement plan for this layer’s routed experts, built from each expert’s actual resident byte size (WeightMatrix::resident_bytes() summed across its gate/up/down matrices, so it reflects the real quantization format in use, not an estimate) and the activation counts observed so far. See frink_moe::PlacementPlan::from_budget.