Module ops 

The elementwise and per row ops of the compat graph, in FP32.

Statistics and transcendental functions are taken in f64 and rounded once, so each result is within an ulp or so of the exact value. PyTorch computes them in f32 with its own vectorized approximations, and the difference to it is at that level either way.

Structs

Rope

Rotary position tables for one base, as Hugging Face builds them for the default rope type.

Functions

add

x += y.

exp_neg

e^y for y <= 0 without calls, so softmax loops vectorize. Below -87 it gives about 1.6e-38 instead of going on down to zero.

geglu

ModernBERT’s gated MLP input: u is rows of 2i, the first half goes through GELU and is multiplied by the second half.

gelu

The exact GELU, x/2 (1 + erf(x/√2)), which is what nn.GELU() and ModernBERT’s "gelu" compute.

layer_norm

LayerNorm over rows of d: (x - mean) / sqrt(var + eps) * w + b, with the biased variance PyTorch uses.