Expand description
Qwen-style Gated Delta Net (GDN) — linear-attention / SSM recurrent
primitive for hybrid arches (qwen35, qwen35moe, qwen3next, …).
Distinct from Kimi KDA (kda.rs): GDN uses a fused QKV projection,
a single depthwise ssm_conv1d over the concatenated QKV channels,
per-head ssm_alpha / ssm_beta gates, and decay
exp(softplus(α + ssm_dt) · ssm_a) (GGUF ssm_a is typically
-exp(A_log)). KDA is not a drop-in for this graph.
§GGUF tensor name mapping (per layer L)
| Role | GGUF name |
|---|---|
| Fused Q‖K‖V | blk.{L}.attn_qkv.weight |
| Output / z gate | blk.{L}.attn_gate.weight |
| Depthwise causal conv | blk.{L}.ssm_conv1d.weight |
| Decay bias | blk.{L}.ssm_dt.bias (alt: ssm_dt) |
| Decay scale | blk.{L}.ssm_a |
| Input gate β | blk.{L}.ssm_beta.weight |
| Forget raw α | blk.{L}.ssm_alpha.weight |
| Output RMSNorm | blk.{L}.ssm_norm.weight |
| Output projection | blk.{L}.ssm_out.weight |
Legacy qwen3next may pack β/α into ssm_ba or fuse QKV+z into
ssm_in; this module implements the split qwen35 layout only.
GGUF weight load skeleton: crate::hybrid_gguf_loader. Serve still
fail-closed — factory HybridEngine::reject.
Structs§
- GdnConfig
- Tiny-config dims for the Qwen35-style GDN step (equal K/V heads).
- GdnState
- Fixed-size recurrent + short-conv state (unlike growing KV).
- GdnWeights
- Weights matching the qwen35 GGUF layout (see module docs).
Functions§
- gdn_
forward_ token - One decode step. Assumes
num_k_heads == num_v_headsandhead_k_dim == head_v_dim == cfg.head_dim.