Expand description
CPU f32 reference for Qwen3.5 hybrid decode (the arch behind NuExtract-3,
Qwen3_5ForConditionalGeneration): 24 gated-DeltaNet linear-attention layers + 8 full-attention
layers (every 4th), SwiGLU MLP, tied lm_head. This is a pure-CPU decoder — no wgpu/Vulkan — and
it is the spec the GPU kernels are checked against, itself validated op-for-op against the
transformers gold continuation (see bin/qwen35-cpu).
Reuses the already-validated gated-delta recurrence crate::deltanet::DeltaNetRef::core for the
linear layers; the full-attention layer (per-head QK-RMSNorm, PARTIAL RoPE on the first
head_dim·partial_rotary_factor dims, GQA, and the Qwen3.5 attention OUTPUT GATE
o · sigmoid(gate)) is implemented here. mRoPE collapses to standard RoPE for text-only input
(all three position axes equal the token position), which is the decode regime this serves.
Numerics that matter for parity: the standard RMSNorms are Gemma-style x·(1+w) (weights stored
as the delta from 1); only the gated DeltaNet norm uses a plain w (handled inside DeltaNetRef).