Module int4 

W4A8 int4 weights for the microdecoder — packed two-per-byte, unpacked in registers.

Why the microdecoder, and why now

Doctrine #2 sends int4 to the microdecoder FIRST, and the measurement now agrees. Its 5-layer body is re-read fifteen times per frame — the single largest repeated read in the model — so halving its weight bytes attacks the one place cache residency is plausibly winnable: roughly 79 MB of Q8 becomes ~40 MB of Q4, which is the difference between spilling to DRAM every depth step and staying resident across all fifteen.

Until 2026-08-10 this was a rounding error: the codec was 92% of browser frame time and the talker+microdecoder 7.9%. After the packed GEMM and the kernel team took the codec down 13x, the split is codec 65% / talker+micro 33% — so this now targets a third of the frame.

The quantization contract, and how it differs from Q8

Symmetric, per-output-channel, ties-to-even — the same shape as crate::int8::quantize_row_q8, with one deliberate asymmetry preserved: the most negative representable value is never emitted. Q8 excludes -128 and keeps [-127, 127]; Q4 excludes -8 and keeps [-7, 7]. That symmetry is what makes -w exactly representable whenever w is, so negating a row negates its quantization exactly, and it keeps the accumulator’s worst case symmetric.

The cost is real and must not be glossed: 15 levels instead of 255. Quantization error is ~17x larger per weight, which is precisely why doctrine #2 gates this behind BOTH a per-ISA speed test that includes unpack cost AND a blind-listening equivalence test. This module ships the arithmetic, not the decision. Nothing routes to it until those gates are run.

Packing

Two nibbles per byte, low nibble first, along k. A row of odd length pads its final high nibble with the BIASED zero (8), not a raw 0.

That distinction matters and is easy to get backwards. A raw 0 nibble decodes to 0 - 8 = -8, the largest negative weight in the range — so zero-initialized padding is not neutral, it is maximally non-neutral. Today nothing reads past k and it would not matter, but the entire reason to store int4 is a future SIMD unpack that processes whole BYTES, and such a kernel would silently fold that -8 into the last accumulator. Padding with the biased zero makes the pad decode to 0.0 and keeps any whole-byte kernel correct by construction.

Storing the nibble biased by +8 (so [-7, 7] becomes [1, 15]) makes unpacking a shift-and-mask with no sign extension, and the bias cancels exactly in the dot product — see dot_i32_q4, where it becomes a single correction term computed from the activation sum.

Structs

QuantizedMatrixQ4

A weight matrix quantized to symmetric int4, packed two values per byte.

Functions

dot_i32_q4

Exact i32 dot product of an int8 activation row against one packed int4 weight row.

linear_q4

W4A8 linear: out[m, n] = x[m, k] @ weight^T, scales applied once per element.