Expand description
A dedicated decoder skeleton for DeepSeek V4’s real architecture,
separate from ferrox-models::decoder::Decoder (the generic GQA path
every other preset uses) — analogous to glm52_decoder.rs /
kimi_decoder.rs: composes the already-independently-tested mHC,
CSA/HCA compression + attention, derope, grouped output projection,
and sqrtsoftplus MoE primitives into one synthetic forward pass.
Synthetic weights only — not a real checkpoint path. No GGUF
loader, no Engine wiring, no claim of oracle-correct output against
a production DeepSeek V4 file. This module exists to prove the real
primitives compose into a finite forward pass end-to-end on tiny dims,
the same rigor glm52_decoder.rs applies before a real loader lands.
Real per-layer flow (simplified to one layer here, transcribed from
llama.cpp PR #24162 deepseek4.cpp’s layer loop):
attn_in = mHC_pre(hc_streams, attn_hc_pre)
attn_out = rms_norm(attn_in) |> HCA/CSA attention |> derope |> grouped wo_a/wo_b
hc_streams = mHC_post(attn_out, ...)
ffn_in = mHC_pre(hc_streams, ffn_hc_pre)
ffn_out = rms_norm(ffn_in) |> MoE (sqrtsoftplus routing)
hc_streams = mHC_post(ffn_out, ...)
hidden = mHC_head(hc_streams)
logits = output_head(rms_norm(hidden))Deliberately not implemented in this skeleton (real, cited scope
for later slices): incremental DSV4 KV/compressor state
(llama-kv-cache-dsv4.cpp), CSA’s coff=2 dual-role projection,
hash-based first-layer MoE selection, and multi-layer stacking.
Structs§
- Deepseek
V4Attn Weights - One layer’s attention-side weights (synthetic, tiny-dim fixtures only).
- Deepseek
V4Csa Weights - A CSA layer’s extras: the doubled role projection and the Lightning
Indexer. Present only on a
LayerCompressor::Csalayer, and its absence there is a configuration error rather than a default. - Deepseek
V4Decode State - Deepseek
V4Decoder Config - Deepseek
V4Decoder Layer Weights - Deepseek
V4Decoder Weights - Deepseek
V4Layer State - Minimal per-layer state: raw K/V for the SWA window plus optional compressed entries. No incremental DSV4 cache yet — callers append one token at a time and pool when the layer’s compressor has enough raw positions for its next block.
- Deepseek
V4Moe FfnWeights - MoE FFN weights for one layer (
sqrtsoftplusgating, no hash routing).
Functions§
- deepseek_
v4_ forward_ token - One decode step through the single synthetic layer, then final norm +
output projection.
token_idindexes the embedding table.