# Verification
Three layers, each of which has been shown to **fail** on an injected fault. A
check that has never failed is not known to work.
## 1. Static audit
```bash
python3 scripts/audit_gemm_tiles.py
```
Cross-checks, mechanically, the two relationships Rust's type system cannot
express:
- every Rust `TileGeom` against the `constexpr int SM`/`SN` compiled into the
kernel it dispatches — a mismatch means the host launches the wrong
threadgroup count and leaves output tiles unwritten;
- each cooperative kernel's `constexpr int BKC` against Rust's `COOP_BKC` — a
drift there lets the host admit K values whose tail the kernel's
`k + BKC <= K` loop silently drops.
It also **fails on any `*_coop` kernel that is not explicitly pinned**. Those
kernels are dispatched through a variable, never `pipeline("literal")`, so a
scanner cannot discover them; an unpinned one would escape the audit entirely
while appearing to pass.
Paths resolve from the script's own location, so it runs from any directory and
from inside an extracted `.crate`. Verified against three injected faults: a
tile drift, a BKC drift, and an unpinned kernel.
## 2. Adversarial shape sweep
Hand-picked shapes across every dispatch path: degenerate (1×1×1), primes,
one-off tile boundaries (63/65/127/129/257), exact tile multiples, extreme
aspect ratios, and shapes straddling each clause of the cooperative gate.
Output buffers are pre-seeded with a `1e30` sentinel, so a tile the kernel fails
to write is **caught** rather than read as a plausible number. Results are
checked against an f64 CPU reference; accumulating paths are checked for
`C0 + A@B` rather than `A@B`.
## 3. Randomized shape fuzz
```bash
# 160 cases, part of the ordinary suite
cargo test --release --lib -- --test-threads=1 --nocapture gemm_fuzz_quick
# 2500-case soak, #[ignore]d so it stays out of the default run
cargo test --release --lib -- --ignored --test-threads=1 --nocapture gemm_fuzz_deep
# replay a failing seed
STRESS_SEED=0xdeadbeef cargo test --release --lib -- --test-threads=1 gemm_fuzz_quick
```
Deterministic and seeded, so a failure prints the seed and shape and reproduces
exactly.
> [!CAUTION]
> Everything this section said before 2026-08-31 was wrong, and wrong in the
> way this document exists to prevent. It documented a test named
> `gemm_randomized_shape_fuzz` and two environment variables `GEMM_FUZZ_SEED`
> and `GEMM_FUZZ_CASES`. None of the three exist. The command it told you to run
> therefore matched no test and printed:
>
> ```
> running 0 tests
> test result: ok. 0 passed; 0 failed; 89 filtered out
> ```
>
> A verification command that runs nothing and reports `ok` is worse than no
> command, because it converts an unexamined kernel into a documented green
> tick. It also claimed the fuzzer "asserts its own coverage — every kernel
> `gemm` can select must be chosen for at least 1% of cases or the run fails".
> No such assertion is implemented. Per-kernel coverage accounting would be
> worth having; until it exists, the fuzzer checks correctness on the shapes it
> happens to draw and nothing more.
**Malformed env values panic rather than falling back.** A seed parsed with
`.ok()` and silently discarded means a soak across eight seeds re-runs one seed
eight times and reports success every time. `STRESS_SEED` accepts hex or
decimal and refuses anything else loudly.
## 4. Hostile input across the `nn` surface
```bash
cargo test --release --test nn_adversarial -- --test-threads=1
```
Every entry point in `tessl::nn` is driven with undersized buffers, degenerate
dimensions, non-finite scalars, and dimension products chosen to overflow. Each
case asserts three things, and the third is the one that matters: the call
returns `Err`, it does not panic, and `take_dispatch_count()` is still zero.
Without that third assertion a kernel that validated *after* encoding would pass
while still having submitted work.
The checks are load-bearing rather than decorative. Compiling `nn` with the
`require` capacity checks disabled does not produce a clean failure — the suite
hangs the GPU past a 120-second timeout, against 0.06 s with them in place.
## 5. Numeric coverage of the promoted kernels
`promoted_kernels.rs` asserts each of the 44 promoted entry points resolves out
of tessl's own metallib. That is a gate on the *move*, not on correctness: a
kernel can resolve, dispatch, and return wrong numbers.
It did. `gemv_q4_tiled` resolved, had adversarial coverage of its error paths,
and wrote 4 rows of 512 because the host handed it the other Q4 kernel's grid.
Giving every promoted kernel a numeric test found six defects in total — two
grid mismatches, one undocumented weight layout, uninitialised threadgroup
scratch across half of every attention query block, a NaN in the online softmax,
and an output-width validation that made `out_bf16` unusable.
All 44 now have one, in `nn_kernels.rs`, `reductions.rs`, `nn_wiring.rs`,
`promoted_numeric.rs`, `attention.rs`, `qkv_rope.rs` and `q4_interleaved.rs`.
Where a family is selected by an enum or a bool, every arm is exercised: the
three Q4 MLX row variants share one reference, and both `Q4MlxLayout` packings
are checked against each other as well as against the dense reference.
## Fault injection
The suite is only worth its green tick if it can go red. Six faults injected
into the cooperative kernels, six caught:
| `BKC` 128 → 256 (K tail silently dropped) | fuzz + sweep |
| tile `SM` 64 → 128 (rows unwritten) | fuzz + sweep |
| accumulator store removed | sentinel check |
| accumulator seeded non-zero | reference check |
| every other K block skipped | reference check |
| column offset off by one | sentinel check |
Two earlier "faults" were caught by nothing — and both were bad injections, not
gaps: one was a no-op (`if (K != 99999u)` is always true) and one perturbed the
result by less than the declared tolerance. They are recorded here because
"the test didn't catch it" and "there was nothing to catch" look identical in a
log.
## Current state
```
audit PASS, 0 mismatches, fires on 3 injected faults
fault tests 6 of 6 caught
lib tests 89 (88 passing, 1 #[ignore]d deep soak)
integration 139 across 18 files
doc tests 1
total 228 passing
```
Measured with `cargo test --release -- --test-threads=1`, which is mandatory
rather than tuning: GPU tests share default command encoders across threads.