# Benchmarking
The measurement protocol, and the five ways these numbers went wrong before they
went right. This page is longer than the architecture page on purpose: on this
hardware, at these sizes, **how you time it changes the answer more than what
you changed in the kernel.**
## The protocol
| `bench_gemm_tnnt_tune` | Paired, interleaved TN/NT kernel A/B. Use this for kernel comparisons. |
| `bench_gemm_tile_tune` | Broad tile/BK ladder. Blocked-style timing — see pitfall 1. |
| `bench_gemm_sweep` | Cross-runtime lane (f32 exact / tf32 / bf16), JSON out. |
| `bench/paired_cross_runtime.py` | Alternates the tessl and PyTorch/MLX lanes round by round. |
The A/B rig is 92 measurement-only kernels and is **not** in the default
metallib — linking it took the shipped artifact from 0.22 MB to 1.09 MB. Opt in:
```bash
TESSL_GEMM_TUNE=1 cargo build --release --bins
```
Both tuning binaries `exit(2)` with that exact command when the variants are
absent, rather than printing a page of `skip(pipe)` and exiting 0.
## Pitfall 1 — a noise floor wider than the effect
The tile-tune rig times a baseline block and a variant block minutes apart, so
GPU clock drift lands entirely in the ratio. Run four times, it showed the
**production kernel measured against itself** ranging **0.92×–1.46×**.
Any single-run conclusion from that rig is inside its own noise.
The paired lanes interleave baseline and candidate iteration by iteration
(A,B,A,B,… alternating which goes first) and report the ratio of per-round
medians across repeated rounds. That brought the spread to roughly ±5%.
> [!NOTE]
> Both places above named `bench_gemm_coop_ab` until 2026-09-01. No such binary
> exists — the crate builds five, and the paired A/B lane is
> `bench_gemm_tnnt_tune`. A first pass at this corrected only the *Reproducing*
> block at the end of this document and left these two, which is why the note
> there could truthfully say it was fixed while two live references survived.
## Pitfall 2 — a harness that contaminates its own baseline
The A/B harness allocated a fresh output tensor per candidate. Those piled up
across the run, and with nine candidates the **baseline kernel** drifted
0.268 → 0.665 ms *inside a single shape block*. Read naively, that is a 2.5×
candidate win invented entirely by allocation order.
Fixed by allocating once per shape. The harness now also prints its own baseline
spread across rows and flags
```
EXCEEDS 10%: ratios above are not comparable
```
because this failure is otherwise invisible — every number still looks like a
number. One shape (2048×768×768) still trips that guard and is excluded rather
than quoted.
## Pitfall 3 — cross-process drift
Running the Rust sweep once and the Python sweep once, minutes apart, puts all
the drift between them into the ratio. Two back-to-back runs of the *identical*
benchmark disagreed by **16–21% on the PyTorch lane alone** — larger than most
differences being reported.
`bench/paired_cross_runtime.py` alternates the lanes round by round and reports
the median of per-round ratios plus the spread. **Single-run cross-runtime
numbers should not be quoted**, including earlier ones from this project: a
"1.07× ahead of PyTorch on bf16" claim did not survive the paired protocol and
became 1.00×.
## Pitfall 4 — stale binaries
`cargo test --lib` rebuilds the library but **not** the binaries. Two rounds of
"the fix isn't working" were a stale `bench_gemm_sweep`. Worse, a conclusion was
drawn from those numbers — that in-kernel register pressure cost ~25% — and it
was never actually tested.
Rebuild binaries explicitly before benchmarking:
```bash
cargo build --release --bins
```
## Pitfall 5 — traversal order
The tune rig used row-major tile traversal while production uses Morton/Z-order
on square power-of-two grids. The control read 0.71–0.88× of production *despite
identical geometry*. A rig must mirror `tile_from_linear`, or every variant is
penalised on square shapes.
## The dispatch floor
Under this submit-and-wait protocol both tessl and PyTorch sit on a **~0.25 ms
per-GEMM floor**. tessl's wall time is flat from 4 MFLOP to 2416 MFLOP:
| 64×64 | 4 | 0.266 | 0.299 |
| 512×512 | 268 | 0.234 | 0.244 |
| 1536×1536 | 2416 | 0.267 | 0.396 |
| 2048×2048 | 4295 | 0.360 | 0.547 |
A 600× increase in work at constant wall time is not a GEMM measurement. Below
roughly 2 GFLOP these benchmarks measure submit-and-wait latency, so ratios there
— including bf16 `square_512` at 0.84× — say nothing about kernel throughput.
Real workloads batch many dispatches into one command buffer and do not pay this
per-GEMM.
## Reproducing
```bash
# Rust lanes, human table plus JSON to stdout. BENCH_SHAPES overrides the
# built-in ladder; BENCH_WARMUP and BENCH_ITERS default to 10 and 50.
cargo run --release --bin bench_gemm_sweep
BENCH_SHAPES="2048x2048x2048,4096x4096x1024" cargo run --release --bin bench_gemm_sweep
# Paired cross-runtime (needs torch and/or mlx importable)
python3 bench/paired_cross_runtime.py --rounds 5 --lanes torch,mlx
# Tile-geometry and TN/NT A/B lanes. These live behind TESSL_GEMM_TUNE because
# the 92 measurement kernels are excluded from the default metallib.
TESSL_GEMM_TUNE=1 cargo build --release --bins
cargo run --release --bin bench_gemm_tile_tune
cargo run --release --bin bench_gemm_tnnt_tune
# Bit-exact parity of TensorOps against the reference SIMD path
cargo run --release --bin probe_gemm_parity
```
> [!NOTE]
> This block previously named a binary `bench_gemm_coop_ab` and an environment
> variable `BENCH_ROUNDS`. Neither exists; the crate builds four binaries and
> they are the ones listed above. The command failed with "no bin target named
> `bench_gemm_coop_ab`" for anyone who tried it.
## Measured — M5 Pro, 2026-08-31
`cargo run --release --bin bench_gemm_sweep`, medians over 50 iterations after
10 warmup, single run rather than paired, so treat these as the machine's shape
rather than a cross-runtime claim:
| 512³ | 796 | 1,284 | 1,358 | 912 |
| 1024³ | 4,167 | 7,291 | 8,410 | 2,380 |
| 2048³ | 6,431 | 13,383 | 21,220 | 2,735 |
| 4096³ | 6,292 | 13,624 | **26,642** | 2,254 |
| qkv_proj (2048×768×768) | 4,262 | 7,639 | 9,209 | 2,348 |
| mlp_up (8192×3072×768) | 6,127 | 16,293 | 24,776 | 2,639 |
| mlp_down (8192×768×3072) | 6,403 | 15,301 | 25,288 | 2,334 |
| tall_k1024 (4096×4096×1024) | 6,180 | 16,096 | 25,126 | 2,732 |
GFLOP/s. Three things in that table are worth reading rather than skimming.
**bf16 TensorOps is 11.8× the portable simdgroup fallback at 4096³** (26,642 vs
2,254) and 4.2× exact f32. That ratio is the entire argument for the
cooperative-destination path.
**At 512³ the simdgroup fallback beats TensorOps f32** — 912 against 796. Not a
regression: at that size the whole GEMM is inside the dispatch floor described
above, so the comparison measures submit-and-wait latency, not the kernels.
**f32 exact flattens at ~6,400 GFLOP/s from 2048³ onward** while bf16 keeps
climbing to 4096³. Exact f32 uses the 32×32 single-simdgroup kernel with no
register accumulator, so it is bandwidth-bound where the cooperative kernels are
not.
## The `nn` kernels — M5 Pro, 2026-08-31
```bash
cargo run --release --bin bench_nn_kernels
```
These kernels are small enough that the dispatch floor above dominates them
entirely. A 1×4096 RMSNorm moves 32 KB; issued alone with a `synchronize()` it
reports ~620 µs, which is the driver, not the shader. So each kernel is timed
twice and both columns are printed:
* **batched** — `set_async_encode(true)`, 64 dispatches accumulated into one
command buffer, one `synchronize()`, divided by 64. What a decode loop pays.
* **solo** — `set_async_encode(false)`, one dispatch, one synchronize.
`async_encode` defaults to **off**, so the solo column is what a caller gets
without asking for anything. The gap between the columns is what the packed
encode path is worth, and at `mlp_silu n=4096` it is 4.1 µs against 203 µs —
**49×**.
| `rms_norm_f32` | 1×4096 | 24.2 | 388.6 | 2.0 |
| `rms_norm_f32` | 512×4096 | 55.1 | 254.5 | 305.0 |
| `rms_norm_f32` | 2048×4096 | 310.5 | 512.7 | 216.2 |
| `mlp_silu` | n=4096 | 4.1 | 203.2 | 11.9 |
| `mlp_gelu_tanh` | n=4096 | 4.0 | 199.7 | 12.2 |
| `mlp_silu` | n=1M | 30.0 | 213.0 | 419.2 |
| `mlp_gelu_tanh` | n=1M | 30.9 | 216.4 | 406.7 |
| `mlp_silu` | n=8M | 440.1 | 582.4 | 228.8 |
| `softmax_rows_f32` | 512×4096 | 79.8 | 259.2 | 210.4 |
| `softmax_rows_f32` | 2048×8192 | 588.2 | 796.2 | 228.2 |
| `row_sum_f32` | 2048×8192 | 276.3 | 502.9 | **242.9** |
| `row_max_f32` | 2048×8192 | 277.7 | 435.3 | 241.7 |
| `gemv_q8` | 4096×4096 | 95.8 | 259.2 | 197.4 |
| `gemv_q4` [row] | 4096×4096 | 81.6 | 261.9 | 128.8 |
| `gemv_q4` [tiled] | 4096×4096 | 131.8 | 305.1 | 79.8 |
| `gemv_q8` | 11008×4096 | 307.9 | 786.8 | 164.9 |
| `gemm_i8_dequant` | 512³ | 18.9 | 210.6 | — |
| `gemm_i8_dequant` | 2048³ | 343.4 | 526.7 | — |
The two integer GEMMs are compute-bound rather than bandwidth-bound: 14,221 and
**50,032 GFLOP/s**. The latter is 1.9× the bf16 TensorOps peak in the table
above, which is the expected shape for int8 on these accelerators.
The `n=1M` row at 419 GB/s is the honest ceiling estimate for this machine —
above it the working set stops fitting and `n=8M` settles at 229 GB/s, which is
where the large reductions also land. Read ~240 GB/s as "saturating memory" for
these shapes.
### RMSNorm was one thread per row — fixed 2026-08-31
This benchmark's first run found `rms_norm_f32` peaking at **87 GB/s** where
`row_sum_f32` reached **243 GB/s** on identical traffic, and taking 404 µs to
move 32 KB at 1×4096.
Not an artifact. The kernel's own first line said "One thread per row", and
`dispatch_1d(rt, &p, rows)` launched exactly `rows` threads, each walking its
row serially twice — once to accumulate the sum of squares, once to scale. At
the decode shape `rows = 1`, so the whole kernel ran on a single GPU thread.
RMSNorm runs twice per transformer layer on every token, making decode the worst
point on that curve.
All three variants — `rms_norm_f32`, `rms_norm_bf16`,
`rms_norm_residual_add_f32` — now use one threadgroup per row with the sum of
squares reduced as a tree, which is the pattern `reduce.metal` already used:
| 1×4096 (decode) | 404.3 µs | **24.2 µs** | **16.7×** | 0.1 → 2.0 |
| 512×4096 | 556.2 µs | **55.1 µs** | **10.1×** | 30.2 → 305.0 |
| 2048×4096 | 770.7 µs | **310.5 µs** | **2.5×** | 87.1 → 216.2 |
The 512×4096 case now reaches 305 GB/s, above the large reductions, because its
working set still fits cache where a 2048-row sweep does not.
The reduction reassociates where the serial loop did not, so results differ in
the low bits. That is why the new tests compare against an f64 reference rather
than a matching f32 accumulation — the old `rms_norm_ref` summed in f32 in the
same order the serial kernel did, which agreed with the kernel's rounding
instead of measuring it.
Two things this exposed beyond the kernel itself. The existing tests used `dim`
of 16 to 64, and `reduce_tptg` hands a 4096-wide row 1024 threads, so every
lane's strided loop ran exactly once — **deleting the loop entirely left all
three tests green**. `rms_norm_sums_rows_wider_than_one_threadgroup` and its
sibling now cover 4096, a ragged 3000, and 8192, and both kill that mutation.
And `REDUCE_TREE` moved into `kernels/reduce_tree.h` so the two callers share
one definition; `build.rs` had to start tracking `.h` for `rerun-if-changed`,
without which editing the shared reduction would leave both dependents stale in
the metallib while the tests reported a pass.
### Q8 GEMV had the same defect, plus one of its own — fixed 2026-08-31
`gemv_q8` was also one thread per row, and for a kernel whose entire cost is
streaming weights that was wrong twice over. Parallelism was capped at `rows`,
and adjacent threads read addresses `cols` bytes apart, so a simdgroup's 32
loads touched 32 different cache lines and nothing coalesced.
It now uses one simdgroup per four output rows with lanes striding K — the
geometry the MLX Q4 simd GEMVs already used, reusing the existing
`simd_gemv_threadgroups` and `SIMD_TPTG` host helpers rather than inventing a
second convention. Lanes take four bytes each through a `char4` load, so a
simdgroup covers 128 contiguous bytes per instruction instead of 32.
| 4096×4096 | 274.5 µs | **95.8 µs** | **2.9×** | 68.9 → 197.4 |
| 11008×4096 | 383.6 µs | **307.9 µs** | **1.25×** | 132.4 → 164.9 |
The tall case gains least and is the honest limit of this change. Vectorising to
`char4` moved 4096² from 161.6 to 197.4 GB/s but left 11008×4096 flat at ~163,
so that shape is bound by something the access pattern does not control —
plausibly the 5.6 MB of scale/zero pairs it also streams, which are read
uniformly by all 32 lanes of a simdgroup. Recorded as measured rather than
explained away; the next thing to try is loading them once per simdgroup.
The `char4` path needs `group_size % 4 == 0`, so a scalar fallback stays for
other group widths. Both paths and the ragged row tail are covered by
`gemv_q8_covers_the_row_tail_and_the_scalar_fallback` — the pre-existing test
used rows = 24 and group = 16, a multiple of the 8 rows a threadgroup owns and a
group width divisible by four, so it exercised neither. Verified: forcing
`xv = 0` in the fallback and removing the `row < rows` writeback guard both left
it green. The new test seeds `y` past `rows` with a sentinel, because a tail
threadgroup writing rows it does not own is otherwise invisible.
### The sweep that found a correctness bug — 2026-08-31
After RMSNorm and Q8 GEMV, the remaining `dispatch_1d` call sites in `src/nn.rs`
were swept for the same shape: a thread per *row* with a serial loop inside,
rather than a thread per element.
Seven were clean. `mlp_silu`, `mlp_gelu_tanh`, `mlp_gelu_tanh_bf16`,
`scale_f32_inplace`, `kv_store_timestep`, `kv_ring_densify`, `softcap_logits`
and `embed_lookup_q4` are genuinely elementwise — zero inner loops — so one
thread per element is the right grid.
The MLX Q4 row variants (`gemv_q4_mlx`, `_wide`, `_tiled`) are one thread per
row deliberately: they take an **f32** activation vector, and their own doc
comment points at `gemv_q4_mlx_simd` for the bandwidth-limited decode shape,
which reads bf16 and halves the traffic. A faster sibling already exists; these
are a different input contract, not an oversight.
`gemv_q4` was the one left, and benchmarking it turned up something worse than a
slow kernel:
| 4096×4096 | 128.8 GB/s | *3,077 GB/s* | 79.8 GB/s |
| 11008×4096 | 143.9 GB/s | *1,217 GB/s* | 81.6 GB/s |
3,077 GB/s is not a number this machine can produce, and that is the whole tell.
`gemv_q4_tiled` indexes its output row by `threadgroup_position_in_grid`, so it
needs one threadgroup per row; it was being dispatched with
`rows.div_ceil(128)` groups — the geometry the *other* kernel needs. It wrote
the first `rows / 128` rows and left the rest of `y` untouched. No error, no
partial-write signal, just whatever was in the buffer before.
Measured directly: at 512 rows, `nn::gemv_q4(.., tiled = true)` wrote **4 rows
and left 508** holding their sentinel.
Nothing caught it. `promoted_kernels.rs` checks the pipeline name exists;
`nn_adversarial.rs` checks the error paths; neither runs the kernel for a
number. `gemv_q4_tiled_writes_every_row_and_agrees_with_the_row_kernel` now
does, at 512×256 and a ragged 300×128, seeding `y` per variant so the row
kernel's output cannot mask what the tiled one failed to write.
With the grid corrected, the tiled variant is **slower** than the row variant at
both shapes. Its apparent speed was entirely the work it was skipping.
This benchmark's own check was too weak to catch it: it required only that
*some* output element be non-zero, and 4 correct rows satisfied that. It now
seeds every output with a sentinel immediately before the kernel runs and fails
if any live element survives — verified by re-breaking the dispatch, which
reports "4064 of 4096 output elements were never written (first at 32)". Seeding
per call rather than per allocation matters: the two `gemv_q4` arms share one
`y`, and a buffer seeded once would be filled by the first arm before the second
was checked.
Every table in this repository is reproducible with the commands above, on an
M5 Pro. On different silicon expect different constants — see Status in the
README.