Module config 

Architecture configs. Prefer GGUF / config.json over preset defaults. Unconfirmed preset fields must be listed in best_effort_fields. What actually runs: docs/MODELS.md.

Structs

KdaConfig

KDA (Kimi Delta Attention) hyperparameters, verified against Kimi K3’s real config.json linear_attn_config block and the real gated delta-rule reference implementation in fla-org/flash-linear-attention’s fla/ops/kda/naive.py (the exact recurrence: decay state by exp(g), then add a rank-1 beta * k ⊗ (v - kᵀS) correction, then read o = qᵀS) and fla/ops/kda/gate.py (the lower-bounded gate: g = gate_lower_bound * sigmoid(exp(A_log) * (raw_g + dt_bias)), and beta = sigmoid(raw_beta)).

KimiHybridAttention

Kimi K3’s real attention topology, transcribed from the published huggingface.co/moonshotai/Kimi-K3/config.json’s linear_attn_config block. kda_layers/full_attn_layers are kept exactly as published – 1-indexed (layer 1 is the model’s first transformer layer), not ferrox’s usual 0-indexed layers slice – so a caller wiring this into Decoder must subtract 1 before indexing.

MlaConfig

Gated MLA (multi-head latent attention) hyperparameters, verified against Kimi K3’s real config.json text_config block and the real KimiMLAAttention reference implementation (modeling_kimi_linear.py).

MlaRopeConfig

RoPE parameters for the decoupled q_rot/k_rot slices of an MLA attention layer that does apply rotation (unlike Kimi K3 – see MlaConfig::rope’s doc comment). Always the interleaved convention (ferrox_core::attention::apply_rope_interleaved) for every real architecture confirmed so far to use this (GLM-5.2’s rope_interleave: true); a separate split-half variant isn’t wired in here since no confirmed real user of it exists yet.

ModelConfig

Enums

AttentionKind

Which attention mechanism a model uses. Gqa (grouped-query attention + RoPE, uniform across every layer) is the only variant ferrox-core/ferrox-models::decoder actually implement today – it’s what every preset runs through, including the two whose real published attention differs (DeepSeek V4 Pro’s CSA/HCA, Kimi K3’s hybrid KDA/Gated-MLA). KimiHybrid exists so Kimi K3’s real, cited attention hyperparameters are captured accurately rather than silently discarded, even though Decoder itself still runs the GQA path for every layer (the dedicated Kimi decoder is the one consumer of the hybrid variant today).

FfnActivation

Dense / expert FFN non-linearity used by the generic decoder.

LayerAttentionKind

Which concrete attention mechanism a single 0-indexed layer uses – the resolved answer Decoder needs per layer once it dispatches on AttentionKind instead of always running GQA (see ModelConfig::layer_attention_kind; only the dedicated Kimi decoder actually dispatches on it today).

RopeLayout

Which RoPE pairing convention a model uses. Confirmed against llama.cpp’s llama_model_rope_type (src/llama-model.cpp): Norm is adjacent-pair / GPT-J (LLAMA_ROPE_TYPE_NORM); Neox is split-half / GPT-NeoX (LLAMA_ROPE_TYPE_NEOX). Getting this wrong silently produces fluent-but-wrong logits (the real Llama-3.1-8B early-stop bug: ferrox applied NeoX to a Norm architecture).

Constants

MODEL_LEVEL_TENSORS_READ_BY_CONFIG

Model-level (not per-layer) tensors ModelConfig::from_gguf reads.

Functions

deepseek_v4_pro

DeepSeek V4 Pro structural sketch only — CSA/HCA is not on this GQA Decoder path. Real primitives live under deepseek_v4_attention / hyper_connections and are not assembled into a served decoder yet. Hparams (~1.6T / ~49B active) are placeholders for smoke/bench.

glm_5_2

GLM-5.2 (Z.ai) structural sketch only — not a supported real inference path. Real DSA lives in glm_dsa / glm52_decoder and is not wired into Decoder / ferrox-server. This preset drives smoke/bench with synthetic GQA weights only (~744B / ~40B active hparams as published placeholders).

kimi_k3

Kimi K3 structural sketch only for the generic GQA Decoder. Real checkpoint work uses the dedicated Kimi stack (kimi_loader / KimiEngine); slice-verified, not a full end-to-end run. Do not treat this preset as a runnable Kimi substitute.

test_dense_fixture

Matches the generated on-disk fixture exactly (hidden_dim, head counts, ffn_dim, vocab, rope_theta, eps). Used by ferrox inspect-run and the cross-validation test in crates/ferrox-models/tests/gguf_roundtrip.rs to prove the real GGUF loader + forward pass produce the same numbers as an independent NumPy reference implementation reading the same file.

test_mixed_fixture

Matches the generated on-disk mixed-topology fixture: 3 layers, the first of which is an ordinary dense FFN and the remaining two are genuine MoE (3 experts, top-1 routing, 1 shared expert each). Used to verify the “leading dense layers” loading path (ModelConfig::layer_is_dense) against a real file – the pattern found in DeepSeek-2/3-family models via ik_llama.cpp’s source (LLM_KV_LEADING_DENSE_BLOCK_COUNT), which was previously only documented, not implemented or tested.

test_moe_fixture

Matches the generated on-disk multi-expert MoE fixture: 4 experts, top-2 routing, 1 shared expert, packed 3D expert tensors. Used to verify the previously- untested multi-expert loading path (split_expert_tensor in ferrox-models::loader) against a real file, the same way test_dense_fixture verifies the single-expert path.