Struct MoeLayerConfig 

pub struct MoeLayerConfig {
    pub n_experts: usize,
    pub n_experts_active: usize,
    pub n_shared_experts: usize,
    pub hidden_dim: usize,
    pub expert_ffn_dim: usize,
    pub gating: GatingFunction,
    pub norm_topk_prob: bool,
    pub expert_group_count: Option<usize>,
    pub expert_group_used_count: Option<usize>,
    pub expert_weights_scale: f32,
}

Static per-layer MoE configuration. One of these is built per layer from a ModelConfig preset (ferrox-models).

Fields

n_experts: usize

n_experts_active: usize

n_shared_experts: usize

hidden_dim: usize

expert_ffn_dim: usize

gating: GatingFunction

Which function converts router logits into selection scores. See GatingFunction’s doc comment: this is not a stylistic choice, it’s an evidence-backed architectural detail that differs by model family.

norm_topk_prob: bool

Only meaningful for GatingFunction::Softmax (the Sigmoid path has its own separate, always-renormalized convention – see route_top_k_sigmoid‘s doc comment). Whether the top-k selected experts’ softmax weights get renormalized to sum to one after selection. Mixtral’s real routing does this (routing_weights /= routing_weights.sum(...) in its reference implementation) and it’s the right default for any architecture that doesn’t document otherwise – but it is a real, per-model choice, not a law of nature: OLMoE’s real config.json sets norm_topk_prob: false, confirmed against OlmoeTopKRouter.forward in transformers/models/olmoe/modeling_olmoe.py (router_top_value /= router_top_value.sum(...) only runs if self.norm_topk_prob) and against llama.cpp’s real hardcoded build_moe_ffn(..., false, ..., LLAMA_EXPERT_GATING_FUNC_TYPE_SOFTMAX, ...) call for LLM_ARCH_OLMOE in src/models/olmoe.cpp (GGUF carries no metadata key for this – it’s an architecture-hardcoded fact in the reference implementation, not something read from the file). Getting this wrong silently produces a real, wrong generation: caught by comparing ferrox’s real OLMoE output directly against llama.cpp loading the identical GGUF file (llama.cpp answered “Paris” for “the capital of France is”; ferrox, with this bug, answered something else entirely).

expert_group_count: Option<usize>

Optional DeepSeek-V3 / GLM-family expert grouping (expert_group_count / expert_group_used_count in GGUF, i.e. n_group / topk_group in the HF configs). None means flat top-k over all experts (Llama / OLMoE / Qwen2-MoE). Grouped routing is selected at load time; the hot path reads these fields as data.

expert_group_used_count is the number of groups that survive the group filter – it is not a per-group expert quota. All n_experts_active experts are then chosen by one global top-k over the surviving groups, so a token may (and normally does) put several experts in the same group. See route_top_k_grouped_biased for the exact rule and for what reading this field as “experts per group” silently does instead.

expert_group_used_count: Option<usize>

expert_weights_scale: f32

llama.cpp’s expert_weights_scale hparam (GGUF {arch}.expert_weights_scale, LLM_KV_EXPERT_WEIGHTS_SCALE): a constant every routed expert’s combine weight is multiplied by after the optional top-k renormalisation (build_moe_ffn: weights = ggml_scale(ctx0, weights, w_scale)). 1.0 when the checkpoint does not carry the key, which is a real no-op rather than a guess – llama.cpp skips the scale for both 0.0 and 1.0. DeepSeek-V3-lineage MoE recipes (dots1, bailingmoe2, hunyuan-moe, …) set it to values like 2.5, and ignoring it scales every routed contribution wrong.

Trait Implementations

impl Clone for MoeLayerConfig

fn clone(&self) -> MoeLayerConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for MoeLayerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.