Module deepseek_v4_decoder

Expand description

A dedicated decoder skeleton for DeepSeek V4’s real architecture, separate from ferrox-models::decoder::Decoder (the generic GQA path every other preset uses) — analogous to glm52_decoder.rs / kimi_decoder.rs: composes the already-independently-tested mHC, CSA/HCA compression + attention, derope, grouped output projection, and sqrtsoftplus MoE primitives into one synthetic forward pass.

Synthetic weights only — not a real checkpoint path. No GGUF loader, no Engine wiring, no claim of oracle-correct output against a production DeepSeek V4 file. This module exists to prove the real primitives compose into a finite forward pass end-to-end on tiny dims, the same rigor glm52_decoder.rs applies before a real loader lands.

Real per-layer flow (simplified to one layer here, transcribed from llama.cpp PR #24162 deepseek4.cpp’s layer loop):

attn_in  = mHC_pre(hc_streams, attn_hc_pre)
attn_out = rms_norm(attn_in) |> HCA/CSA attention |> derope |> grouped wo_a/wo_b
hc_streams = mHC_post(attn_out, ...)
ffn_in   = mHC_pre(hc_streams, ffn_hc_pre)
ffn_out  = rms_norm(ffn_in) |> MoE (sqrtsoftplus routing)
hc_streams = mHC_post(ffn_out, ...)
hidden   = mHC_head(hc_streams)
logits   = output_head(rms_norm(hidden))

Deliberately not implemented in this skeleton (real, cited scope for later slices): incremental DSV4 KV/compressor state (llama-kv-cache-dsv4.cpp), CSA’s coff=2 dual-role projection, hash-based first-layer MoE selection, and multi-layer stacking.

Structs§

DeepseekV4AttnWeights: One layer’s attention-side weights (synthetic, tiny-dim fixtures only).
DeepseekV4DecodeState
DeepseekV4DecoderConfig
DeepseekV4DecoderLayerWeights
DeepseekV4DecoderWeights
DeepseekV4LayerState: Minimal per-layer state: raw K/V for the SWA window plus optional compressed entries. No incremental DSV4 cache yet — callers append one token at a time and optionally pool when hca_compress_ratio raw positions are available.
DeepseekV4MoeFfnWeights: MoE FFN weights for one layer (sqrtsoftplus gating, no hash routing).

Functions§

deepseek_v4_forward_token: One decode step through the single synthetic layer, then final norm + output projection. token_id indexes the embedding table.