Skip to main content

Module gdn

Module gdn 

Source
Expand description

Qwen-style Gated Delta Net (GDN) — linear-attention / SSM recurrent primitive for hybrid arches (qwen35, qwen35moe, qwen3next, …).

Distinct from Kimi KDA (kda.rs): GDN uses a fused QKV projection, a single depthwise ssm_conv1d over the concatenated QKV channels, per-head ssm_alpha / ssm_beta gates, and decay exp(softplus(α + ssm_dt) · ssm_a) (GGUF ssm_a is typically -exp(A_log)). KDA is not a drop-in for this graph.

§GGUF tensor name mapping (per layer L)

RoleGGUF name
Fused Q‖K‖Vblk.{L}.attn_qkv.weight
Output / z gateblk.{L}.attn_gate.weight
Depthwise causal convblk.{L}.ssm_conv1d.weight
Decay biasblk.{L}.ssm_dt.bias (alt: ssm_dt)
Decay scaleblk.{L}.ssm_a
Input gate βblk.{L}.ssm_beta.weight
Forget raw αblk.{L}.ssm_alpha.weight
Output RMSNormblk.{L}.ssm_norm.weight
Output projectionblk.{L}.ssm_out.weight

Legacy qwen3next may pack β/α into ssm_ba or fuse QKV+z into ssm_in; this module implements the split qwen35 layout only.

GGUF weight load skeleton: crate::hybrid_gguf_loader. Serve still fail-closed — factory HybridEngine::reject.

Structs§

GdnConfig
Tiny-config dims for the Qwen35-style GDN step (equal K/V heads).
GdnState
Fixed-size recurrent + short-conv state (unlike growing KV).
GdnWeights
Weights matching the qwen35 GGUF layout (see module docs).

Functions§

gdn_forward_token
One decode step. Assumes num_k_heads == num_v_heads and head_k_dim == head_v_dim == cfg.head_dim.