Crate ferrox_moe 

ferrox-moe: sparse Mixture-of-Experts routing, a shared-expert path, and a CPU/GPU expert-placement scheduler.

The placement design mirrors the pattern popularized by ik_llama.cpp (tensor-name-regex overrides deciding which experts live on GPU vs CPU RAM, e.g. --cpu-moe / -ncmoe) and by llama.cpp’s layer-split conventions, adapted here to a config-driven Rust scheduler rather than copied CLI-flag parsing code. See docs/THIRD_PARTY_NOTICES.md.

Structs

ExpertBias

One expert’s gate/up/down bias vectors.

ExpertWeights

One expert’s gate/up/down weight matrices. Each may be plain f32 (synthetic/test weights) or still-quantized bytes loaded straight from a GGUF file (real checkpoints) – WeightMatrix::apply dispatches to the right kernel either way.

MoeLayerConfig

Static per-layer MoE configuration. One of these is built per layer from a ModelConfig preset (ferrox-models).

PlacementPlan

A CPU/GPU placement plan for a layer’s experts.

ResidencyPlan

The output of PlacementPlan::plan_layers_against_global_budget: one per-layer PlacementPlan view over a single, globally-accounted device budget. device_bytes_planned <= vram_budget_bytes holds by construction across ALL layers combined – the property the old per-layer planning could not provide.

RoutingDecision

Router output for one token: which experts fire, and their (already-normalized) combination weights.

Enums

ExpertPlacement

Where a given expert’s weights currently live. GpuDevice-placed experts only actually execute on a GPU under --features cuda and/or --features metal (see run_expert_placed); without a GPU feature the CPU path executes regardless. Device id is meaningful for CUDA; Metal currently uses the system default device and ignores the id.

GatingFunction

Top-k softmax router over per-expert logits, as used by DeepSeek / GLM / Kimi-style MoE layers (a linear gate scores every expert, top-k experts are kept, and their scores are renormalized to sum to one). Which function converts a router’s raw per-expert logits into selection scores, before top-k selection and normalization.

Constants

SWIGLU_OAI_ALPHA

gpt-oss’s SwiGLU sigmoid steepness (llama-graph.cpp, LLM_FFN_SWIGLU_OAI_MOE: constexpr float alpha = 1.702f).

SWIGLU_OAI_LIMIT

gpt-oss’s SwiGLU clamp (constexpr float limit = 7.0f, same site).

Functions

combine_expert_outputs

Combines routed + shared expert outputs for one token.

route_hash

DeepSeek V4’s real hash-based first-layer MoE routing: for the first hash_layer_count layers, which experts fire is not learned top-k/sigmoid/sqrt-softplus selection at all – it’s a direct token-id-to-expert-id lookup table (ffn_gate_tid2eid, GGUF shape [n_expert_used, n_vocab]; real per-layer dispatch in src/models/deepseek4.cpp: selected_experts = ggml_get_rows(ctx0, layer.ffn_gate_tid2eid, res->t_inp_tokens), with exp_probs_b (the selection-bias tensor) set to nullptr for these layers specifically because there is no learned selection to bias – the expert ids are fixed by the table, not chosen by a score).

route_top_k

Top-k router over per-expert logits, dispatching to softmax, sigmoid, or sqrt-softplus scoring per gating. See GatingFunction’s doc comment for why this distinction is real and evidence-backed, not a stylistic choice.

route_top_k_biased

llama.cpp’s build_moe_ffn selection, with the DeepSeek-V3 aux-loss-free bias applied to the selection score only.

route_top_k_grouped

Mixtral-/DeepSeek-style grouped top-k: split logits into n_groups contiguous expert groups, pick the k_per_group highest within each group (by the same score as flat route_top_k), then optionally keep only the global top-total_k across groups.

route_top_k_sigmoid

sigmoid-then-renormalize top-k routing: score every expert with sigmoid(logit) (independently per expert, not a joint softmax distribution), pick the top-k by that score, then renormalize just the selected experts’ sigmoid scores to sum to one. This is the DeepSeek-V3 / GLM4-MoE convention found in ik_llama.cpp’s real GGUF hparams-loading source.

route_top_k_sigmoid_with_bias

Sigmoid top-k routing with a real “aux-loss-free” per-expert bias term added only for top-k selection (topk_method: "noaux_tc" in Kimi K3’s real config.json, KimiMoEGate.forward in modeling_kimi_linear.py, adapted from DeepSeek-V3’s own MoE gate – the same convention, not Kimi-specific): the selection scores are sigmoid(logit) + bias[expert], but the weight each selected expert’s output gets multiplied by uses the raw, unbiased sigmoid(logit) – getting this backwards (biasing the weight itself, not just the selection) would silently skew routed-expert contribution away from what the router actually learned. Weights are renormalized to sum to 1 (if k>1) then multiplied by scaling_factor (Kimi K3’s routed_scaling_factor, 1.0 in its real config, i.e. a no-op there, but a real multiplier for any other model using this same convention with a different value).

route_top_k_softmax

softmax-then-top-k routing (the Mixtral/older-DeepSeek convention: softmax over every expert first, then select the top-k of those probabilities – not “top-k logits, then softmax just those”; the two are mathematically different since softmax’s denominator would only sum the selected subset in the latter). norm_topk_prob controls whether the selected top-k probabilities are then renormalized to sum to one – true is the right default for any architecture that doesn’t document otherwise (Mixtral does this), but it is a real per-model choice: see MoeLayerConfig::norm_topk_prob’s doc comment for why OLMoE specifically needs false.

route_top_k_softmax_weight

gpt-oss routing: pick the top-k experts by their raw router logits, then softmax over just those k.

route_top_k_sqrtsoftplus

sqrt-softplus top-k routing, DeepSeek V4’s real non-hash-routed MoE layers (ffn_exp_probs_b present, i.e. every layer at or past hash_layer_count): score every expert with sqrt(softplus(logit)) (independently per expert, like route_top_k_sigmoid’s sigmoid – not a joint softmax distribution), pick the top-k by that score, then (if norm_topk_prob) renormalize the selected scores to sum to one. See GatingFunction::SqrtSoftplus for the real citation. DeepSeek V4’s real non-hash MoE layers additionally add a learned bias (ffn_exp_probs_b) to the selection score only – see route_top_k_sqrtsoftplus_with_bias for that variant; this plain version is the bias-free building block, analogous to route_top_k_sigmoid vs route_top_k_sigmoid_with_bias.

route_top_k_sqrtsoftplus_with_bias

sqrt-softplus top-k routing with a selection-only bias term, mirroring route_top_k_sigmoid_with_bias but for DeepSeek V4’s real GatingFunction::SqrtSoftplus scoring: selection uses sqrt(softplus(logit)) + bias[expert], but each selected expert’s combine weight uses the raw, unbiased sqrt(softplus(logit)). Weights are renormalized to sum to one (if k>1 and renormalize), then multiplied by scaling_factor (DeepSeek V4’s real expert_weights_norm/expert_weights_scale hparams, read directly in load_arch_hparams).

run_expert

Runs one token’s hidden state through a single expert’s SwiGLU FFN.

run_expert_oai

run_expert for gpt-oss: per-expert biases on all three matmuls and swiglu_oai in place of SwiGLU.

run_expert_placed

swiglu_oai

gpt-oss’s clamped SwiGLU.