Module matmul 

Functions

geglu

Elementwise gated FFN combine: gelu(gate) * up (Gemma GeGLU).

gelu

GELU (tanh approximation) used by Gemma GeGLU FFNs.

layer_norm

Plain (non-RMS) LayerNorm – ggml’s LLM_NORM (as opposed to LLM_NORM_RMS, what rms_norm implements): subtract the mean, divide by the standard deviation, then apply an elementwise affine * weight + bias. GLM-5.2’s real DSA lightning indexer normalizes its compressed key through exactly this (indexer_k_norm carries both a weight and a bias GGUF tensor – confirmed against llama.cpp PR #23346/#25407’s real create_tensor(tn(LLM_TENSOR_INDEXER_K_NORM, "weight"|"bias", i), ...) calls and the build_norm(indexer_k, ..., LLM_NORM, il) call site, LLM_NORM being ggml’s plain-LayerNorm op, distinct from every other norm in this codebase so far, which are all RMSNorm).

matmul_f32

Row-major matmul: a is [m, k], b_t is [n, k] (i.e. already transposed, which is how GGUF stores weight matrices for a y = W x projection). Output is [m, n]. Parallelized over output rows with rayon, matching the “each output row is independent” decomposition used across llama.cpp / ggml’s matmul kernels – except for the single-token decode case (m == 1, the common case for this path, since WeightMatrix::F32 is only used for small tensors like embeddings/synthetic weights), where parallelizing over m would give exactly one chunk regardless of thread count, i.e. no parallelism at all no matter how large n is. That case instead parallelizes over n (output features) directly, since out is exactly n elements long when m == 1 and needs no layout transpose to do so.

rms_norm

RMSNorm as used by LLaMA-family and DeepSeek/GLM/Kimi-family decoders: x_normalized = x / sqrt(mean(x^2) + eps) * weight

rms_norm_per_head

Per-head RMSNorm (Qwen3 / Gemma3 attn_q_norm / attn_k_norm): weight has length head_dim and is reused for every head in x (layout [n_heads, head_dim] row-major).

silu

SiLU / swish activation: x * sigmoid(x). Used by the SwiGLU-style gated MLP and MoE expert feed-forward blocks in this family of models.

situ_and_mul

Kimi K3’s situ activation (hidden_act: "situ" in its real config.json, registered as ACT2FN["situ"] -> SituAndMul in modeling_kimi_linear.py): beta*tanh(gate/beta)*sigmoid(gate) * linear_beta*tanh(up/linear_beta). Not SiLU/SwiGLU – a real, non-obvious fact confirmed by reading Kimi K3’s actual reference source and config (activation_situ_beta=4.0, activation_situ_linear_beta=25.0) rather than assuming the more common SwiGLU convention every other model in this codebase uses.

softcap_inplace

Soft-cap used by Gemma 2+ attention / final logits: softcap * tanh(x / softcap).

swiglu

Elementwise gated FFN combine: silu(gate) * up, the standard SwiGLU pairing used inside both dense and MoE-expert feed-forward blocks.