Module hyper_connections 

DeepSeek V4’s “mHC” (multi-stream Hyper-Connection) residual mixing: instead of one residual stream, the model carries hc (hc_mult, real reference value 4) parallel streams per token, and each sub-layer (attention, FFN) is preceded by a learned, per-token gated merge of those streams into one input, followed by a Sinkhorn- normalized mix back into all hc streams.

Transcribed directly from the real, merged reference implementation (llama.cpp PR #24162, src/models/deepseek4.cpp: build_hc_pre/build_hc_post/build_hc_head/build_hc_sinkhorn/ build_hc_weighted_sum, read line-by-line), not derived by analogy – this closes the “mHC’s exact math was not read” gap from earlier research. The real implementation asserts hc == 4 in build_hc_pre (the mix-tensor offset layout is hardcoded to that split), so this module does too rather than silently pretending to support other values.

Not yet wired into a DeepSeek V4 decoder (no such decoder exists in ferrox yet) – this is the residual-mixing primitive on its own, analogous to how mla.rs/block_residual.rs exist as standalone, tested modules before Kimi K3’s decoder consumed them.

Structs

HyperConnectionHeadWeights

Weights for the final output merge (build_hc_head): same structure as the pre half of HyperConnectionPreWeights, but only ever produces the hc-wide merge gate (no post/comb, since there is no further sub-layer to re-inject into).

HyperConnectionPreWeights

Weights for the pre-sub-layer merge (build_hc_pre): projects the flattened, RMS-normed hc streams to (2 + hc) * hc mix logits, split into pre (hc), post (hc), and comb (hc*hc).

Constants

HC_MULT

The real reference implementation’s only supported hyper-connection multiplicity; build_hc_pre’s mix-tensor offsets are hardcoded to this split (GGML_ASSERT(hc == 4) in the real source).

Functions

head

build_hc_head: the final collapse of hc streams into one output vector before unembedding – structurally the pre-gate half of pre, with no post/comb since nothing follows it.

post

build_hc_post: re-injects the sub-layer’s single output back into all hc streams, each scaled by its post gate and mixed with the original (pre-merge) residual streams via the Sinkhorn comb matrix: `out[dst] = sub_layer_out * post[dst] + sum_src(residual[src]

pre

build_hc_pre: merges hc residual streams into one sub-layer input, plus the post gate and Sinkhorn-normalized comb matrix needed by post afterward. Returns (merged_input, post_gate, comb_matrix).

sinkhorn

build_hc_sinkhorn: comb[dst][src], real algorithm – softmax over dst (per fixed src), +eps, one row-normalization (each dst row sums to 1 over src), then sinkhorn_iters - 1 rounds of [column-normalize (each src column sums to 1 over dst), row-normalize].

weighted_sum

build_hc_weighted_sum: per-token weighted sum of the hc stream vectors (each n_embd-wide), sum_h(x[h] * weights[h]).