Crate ferrox_models 

ferrox-models: GGUF decoder, architecture registry, and structural presets for frontier stacks (GLM / DeepSeek V4 / Kimi).

Unconfirmed preset fields go in best_effort_fields and must be overwritten from real config.json / GGUF metadata. Status of what actually runs: docs/MODELS.md. Presets glm_5_2 / deepseek_v4_pro / kimi_k3 are sketches for synthetic tests — not real-checkpoint support. Dedicated primitives live in glm52_*, deepseek_v4_*, kimi_* modules.

Re-exports

pub use capability::architecture_catalog;

pub use capability::coverage_report_markdown;

pub use capability::resolve_architecture;

pub use capability::resolve_profile;

pub use capability::ArchPath;

pub use capability::ArchProfile;

pub use capability::ArchScope;

pub use capability::DecoderFamily;

pub use capability::MemoryKind;

pub use capability::QkNormStyle;

pub use config::deepseek_v4_pro;

pub use config::glm_5_2;

pub use config::kimi_k3;

pub use config::FfnActivation;

pub use config::ModelConfig;

pub use config::RopeLayout;

pub use decoder::Decoder;

pub use device_budget::BudgetBackend;

pub use device_budget::DeviceBudget;

pub use engine::DeepseekV4Engine;

pub use engine::Engine;

pub use engine::Glm52Engine;

pub use engine::KimiEngine;

pub use engine::MlaDenseFfn;

pub use engine::MlaEngine;

pub use engine::MlaLayerFfn;

pub use engine::MlaLayerWeights;

pub use engine::MlaMoeFfn;

pub use engine::MlaMoeRuntime;

pub use engine::TextTokenizer;

pub use engine_factory::ensure_generic_decoder;

pub use engine_factory::load_gemma4_engine_from_path;

pub use engine_factory::load_glm52_engine_from_path;

pub use engine_factory::load_mla_engine_from_path;

pub use engine_factory::select_engine_kind;

pub use engine_factory::EngineSelectError;

pub use engine_factory::SelectedEngineKind;

pub use engine_factory::ServedEngine;

pub use execution_plan::ExecutionPlan;

pub use execution_plan::FusedOpCaps;

pub use execution_plan::MemoryPlan;

pub use execution_plan::PlanGeometry;

pub use gemma4_engine::Gemma4Engine;

pub use gemma4_engine::Gemma4Hparams;

pub use gemma4_engine::GEMMA4_ARCHES;

pub use kv_budget::Ceiling;

pub use kv_budget::ContextCap;

pub use kv_budget::ContextFit;

pub use kv_budget::KvBudget;

pub use kv_budget::KvBudgetError;

pub use kv_budget::KvElem;

pub use kv_budget::KvLayout;

pub use kv_budget::KvShape;

pub use kv_budget::SlidingWindow;

pub use kv_budget::CTX_AUTO_GRANULARITY;

pub use loader::LoadError;

pub use output_projection::grouped_output_projection;

pub use prefix_cache::PrefixCache;

pub use prefix_cache::PrefixCacheStats;

pub use prefix_cache::PrefixMatch;

pub use sampling::Sampler;

pub use sampling::SamplingParams;

pub use speculative::speculative_decode;

pub use speculative::PromptLookupSpeculator;

pub use speculative::SpeculativeDecodeResult;

pub use tensor_role::TensorRole;

pub use tokenizer::ByteTokenizer;

pub use tokenizer::GgufBpeTokenizer;

pub use tokenizer::GgufSpmTokenizer;

pub use tokenizer::GgufUnigramTokenizer;

pub use tokenizer::TokenizerLoadError;

Modules

block_residual

Kimi K3’s cross-layer “block residual” mixing mechanism (_apply_attn_res in the real modeling_kimi_linear.py), a real architectural feature discovered by reading KimiDecoderLayer.forward in full rather than assuming a standard pre-norm residual: every layer blends its running residual stream with a growing set of saved checkpoints from earlier layers in the same block, using a learned RMSNorm-projected softmax score to weight the blend – structurally a tiny self-attention over {checkpoints..., current}.

capability

Explicit architecture capability registry for the generic GGUF path.

chat_template

Chat-template rendering driven by the GGUF’s own tokenizer.chat_template Jinja2 string.

config

Architecture configs. Prefer GGUF / config.json over preset defaults. Unconfirmed preset fields must be listed in best_effort_fields. What actually runs: docs/MODELS.md.

decoder

Generic decoder-only transformer forward pass, assembled from a ModelConfig. Each layer is: RMSNorm -> GQA attention (+RoPE) -> residual -> RMSNorm -> MoE FFN (router + routed experts + shared experts) -> residual. This is the standard decoder block shape shared by the LLaMA/DeepSeek/GLM/Kimi family of open-weight models.

deepseek_v4_decoder

A dedicated decoder skeleton for DeepSeek V4’s real architecture, separate from ferrox-models::decoder::Decoder (the generic GQA path every other preset uses) — analogous to glm52_decoder.rs / kimi_decoder.rs: composes the already-independently-tested mHC, CSA/HCA compression + attention, derope, grouped output projection, and sqrtsoftplus MoE primitives into one synthetic forward pass.

device_budget

How many bytes the selected backend will let this process hold – the right-hand side of crate::kv_budget’s inequality.

engine

A trait-based abstraction over the two structurally different but text-in/text-out-shaped forward passes this crate has: Decoder (GQA+RoPE, used by GLM-5.2/DeepSeek V4 Pro and every real GGUF checkpoint) and Kimi K3’s dedicated hybrid KDA/Gated-MLA stack (kimi_decoder). This lets ferrox-server share one generic generation loop across both engines (see ferrox-server::generate::generate_engine) for the actual sampling/stop-sequence logic, rather than hand-duplicating it – while keeping GGUF-only features (the KV block pool, PrefixCache) as Decoder-specific code layered on top, not forced into this trait. The reason: Kimi’s KDA state is a fixed-size recurrent matrix that collapses history irreversibly, so it cannot support the same restore/truncate operations KvCache can – unifying those too would mean either a leaky abstraction or silently pretending Kimi supports something it doesn’t.

engine_factory

Load-time engine selection (llama.cpp llama_model_* factory analogue).

execution_plan

Load-time execution / memory plans (llama.cpp graph-params analogue).

gdn

Qwen-style Gated Delta Net (GDN) — linear-attention / SSM recurrent primitive for hybrid arches (qwen35, qwen35moe, qwen3next, …).

gemma4_engine

Gemma-4 dedicated text engine (E2B / E4B-style GGUF).

gemma4_gguf_loader

Gemma-4 GGUF → crate::gemma4_engine::Gemma4Engine.

glm52_decoder

A dedicated decoder for GLM-5.2’s real architecture, separate from ferrox-models::decoder::Decoder (the generic GQA path every other preset uses) – analogous to kimi_decoder.rs’s role for Kimi K3: composes the already-independently-tested glm_dsa attention module with a standard SwiGLU dense/MoE FFN into a real forward pass, without touching the existing GQA decoder.

glm52_gguf_loader

Loads ferrox-models::glm52_decoder weights from a GLM-5.2 GGUF checkpoint, via ferrox_gguf::TensorSource – the same trait ferrox-models::loader’s generic GQA path and kimi_gguf_loader’s Kimi K3 path both use. Follows kimi_gguf_loader.rs’s dedicated-loader pattern (a hand-written loader for an architecture whose MLA+DSA structure doesn’t fit the generic GQA loader), not loader.rs’s generic path.

glm_dsa

GLM-5.2’s real DSA (DeepSeek Sparse Attention) attention layer: RoPE-carrying MLA (ferrox_models::mla’s math, inlined here rather than reused directly – see below) plus the lightning indexer (ferrox_core::attention::lightning_indexer_topk) selecting which causal positions are visible, then sparse attention restricted to exactly those (ferrox_core::attention::causal_mla_attention_sparse).

hf_pull

Hugging Face Hub download helper (hf download when the CLI is installed).

hybrid_engine

Hybrid attn+SSM engine stub (Jamba / LFM2 / Nemotron-H / Qwen3.5 GDN / …).

hybrid_gguf_loader

Qwen3.5 / Qwen3-Next hybrid GGUF → GDN weight loader skeleton (P3).

hyper_connections

DeepSeek V4’s “mHC” (multi-stream Hyper-Connection) residual mixing: instead of one residual stream, the model carries hc (hc_mult, real reference value 4) parallel streams per token, and each sub-layer (attention, FFN) is preceded by a learned, per-token gated merge of those streams into one input, followed by a Sinkhorn- normalized mix back into all hc streams.

kda

Kimi K3’s KDA (Kimi Delta Attention): a gated delta-rule linear attention mechanism used on the majority of Kimi K3’s layers (69 of 93, per AttentionKind::KimiHybrid), interleaved with Gated MLA (ferrox_models::mla) on the remainder.

kimi_decoder

A dedicated decoder for Kimi K3’s real hybrid architecture, separate from ferrox-models::decoder::Decoder (which every GQA-only preset – GLM-5.2, DeepSeek V4 Pro, the test fixtures – uses and which ferrox-cli, ferrox-server, prefix_cache, and speculative all depend on): this module composes the already-independently-tested kda, mla, latent_moe, block_residual, and ferrox_core::situ_and_mul pieces into a real forward pass, without touching any of that existing, production-quality GQA code path (a shared, polymorphic Decoder was judged too risky to attempt without a way to verify it end to end).

kimi_generate

Ties kimi_tokenizer::KimiTokenizer, kimi_decoder::kimi_forward_token, and sampling::Sampler into a real text-in/text-out generation loop for Kimi K3 – the piece that turns “a decoder that can run one forward pass given weights and a token id” into something a CLI or server can actually use for a prompt. Mirrors the shape of ferrox-server::generate’s loop (encode prompt, decode incrementally, sample each step, stop at max tokens or EOS, decode the output ids) but against kimi_decoder’s separate per-token state/forward-pass API rather than ferrox_core::cache::KvCache.

kimi_gguf_loader

Loads ferrox-models::kimi_decoder weights from a real Kimi K3 GGUF checkpoint (as opposed to kimi_loader, which reads the real safetensors checkpoint) – via ferrox_gguf::TensorSource, the same trait ferrox-models::loader’s GQA-model GGUF path uses.

kimi_loader

Loads ferrox-models::kimi_decoder weights from Kimi K3’s real safetensors checkpoint (via ferrox-safetensors::ShardedSafetensors), using the exact real tensor names/shapes/dtypes fetched live from huggingface.co/moonshotai/Kimi-K3 (a real shard header, not guessed) – confirmed to match this crate’s kda/mla/latent_moe/ kimi_decoder struct field names and shapes exactly.

kimi_tokenizer

Kimi K3’s real tokenizer: tiktoken-style rank-based BPE, loaded from a real tiktoken.model file (base64-encoded byte sequence + rank per line – the standard OpenAI tiktoken vocab format, confirmed against a real downloaded Kimi K3 tiktoken.model and its real tokenization_kimi.py/tokenizer_config.json), not from GGUF metadata like GgufBpeTokenizer/GgufSpmTokenizer – Kimi K3 ships as safetensors, with its own real tokenizer format, distinct from both existing tokenizers in this module.

kimi_validate

Validate a Kimi K3 checkpoint directory without loading all shards.

kv_budget

Pre-load KV budget arithmetic: answer “will this fit” before allocating anything, from terms that are all exact in the GGUF header.

latent_moe

Kimi K3’s “latent MoE” block (KimiSparseMoeBlock in the real modeling_kimi_linear.py), a real architectural detail beyond a standard top-k MoE FFN, discovered by reading the real source rather than assuming the more common DeepSeek-V3-style MoE this crate’s ferrox_moe::route_top_k was originally written for: routed experts operate on a down-projected latent space (routed_expert_hidden_size = 3584, half of hidden_size = 7168 in Kimi K3’s real config — use_latent_moe/latent_moe_use_norm are real, confirmed-active config fields, not a rare/optional path), not the full hidden dimension. Real per-layer flow:

llama4_engine

Llama 4 dedicated stack stub — not generic GQA.

loader

Loads a real Decoder from an on-disk GGUF file, using the llama.cpp-style tensor naming convention (token_embd.weight, blk.N.attn_q.weight, blk.N.ffn_gate.weight or, for MoE, blk.N.ffn_gate_exps.weight, output_norm.weight, output.weight). Until this module existed, ferrox could only run correctly-shaped random weights.

minimax_engine

MiniMax M2/M3 dedicated stack stub — not generic GQA.

mla

DeepSeek-style Multi-head Latent Attention (MLA): low-rank Q/KV compression, with an optional sigmoid output gate (Kimi K3’s real addition) and an optional RoPE rotation of the decoupled q_rot/ k_rot slices (MlaConfig::rope; GLM-5.2’s real addition – see below). Transcribed directly from real reference code, not guessed or derived by analogy:

mla_gguf_loader

DeepSeek-2 / Mistral-4 GGUF → crate::engine::MlaEngine.

mmproj

Detect mmproj-*.gguf vision towers next to a main GGUF (P7).

output_projection

DeepSeek V4’s grouped attention output projection (wo_a/wo_b, o_group_count groups): a real, structural replacement for the usual single dense wo output projection, transcribed directly from the real, merged reference implementation (llama.cpp PR #24162, src/models/deepseek4.cpp::graph::build_attention, the tail end after attn_derope).

prefix_cache

KV-prefix caching: when a new request’s tokens share a leading subsequence with a previously processed request, skip recomputing the KV state for that shared prefix entirely, restoring it from a stored snapshot instead of running forward_batch over tokens that were already processed.

recurrent_engine

Recurrent engine stub (Mamba / RWKV).

residency_report

Dry-run residency planning: what a checkpoint would cost to run, computed from its GGUF header alone – no weights loaded, no allocation. This is the “plan before you allocate” half of the residency workstream; the enforcement half at decode time is the bounded expert store (ferrox_core::expert_store) plus the global device plan (ferrox_moe::ResidencyPlan).

sampling

Token sampling from a decoder’s output logits: temperature, top-k, top-p (nucleus), and repetition penalty, on top of the greedy argmax ferrox previously always used unconditionally (see crate::speculative, which still uses plain greedy argmax directly since its quality-neutrality proof depends on exactly matching greedy decode – sampling is a deliberately separate, opt-in path).

speculative

Prompt-lookup speculative decoding: propose several candidate next tokens by finding a repeat of the current context elsewhere in the token history (no separate draft model needed, unlike classic speculative decoding), then verify all candidates in a single Decoder::forward_batch call instead of one forward_token call per candidate.

t5_engine

T5 encoder-decoder engine stub.

tensor_role

Semantic tensor roles mirroring llama.cpp llm_tensor / LLM_TN.

tokenizer

A real, reversible byte-level tokenizer: each UTF-8 byte maps to token id byte as u32 (vocabulary 0..256). This is not a full BPE/tokenizer.json implementation – GLM-5.2, DeepSeek V4 Pro, and Kimi K3 each ship their own trained BPE vocabulary alongside their weights, and none of those vocab files are guessable or available in this environment (see docs/MODELS.md) – but unlike the previous placeholder (byte % vocab_size, which was lossy and could not decode back to the original text), this tokenizer is exact and round-trips perfectly. It is the honest “smallest real thing that works” rather than a fake stand-in.

vision

Kimi K3’s MoonViT-V2 vision encoder: patch embedding -> N transformer encoder layers (RMSNorm, self-attention with 2D RoPE, gated-MLP-free plain MLP2 feed-forward) -> a patch-merger projector into the text decoder’s embedding space. Genuinely new territory for ferrox, which has been text-only until now.

vl_engine

Multimodal / VL serve stub (Qwen2-VL / CogVLM / Gemma4-VL / …).