Module quant

Expand description

Int8 block quantization (Q8: blocks of 32 weights sharing one f32 scale).

Structs§

Offsets
PageVec: A zero-initialised, page-aligned buffer whose allocation is a whole number of pages, so GPU APIs can map it without copying.
PairAct: Quantized activations interleaved per token pair for 2x2 int8 matrix-multiply instructions (SMMLA): pair p holds tokens 2p and 2p + 1 (an odd count repeats the last token). q: per 8 values, 8 of the first token then 8 of the second (2 * n_in bytes per pair); s (per block of 32) and h (per half block) as [first, second, first, second].
Q4Matrix
QAct: Quantized activations [t, n_in]: int8 values and one scale per block.
QBuf: Storage for quantized data: either owned (quantized at load time) or borrowed from a pre-quantized checkpoint that outlives the matrix (the model borrows its ModelFiles).
QMatrix: A quantized [n_out, n_in] matrix: int8 values plus one scale per 32-element block.

QKind: Quantized storage formats, as recorded in and read from pre-quantized checkpoints.

PAGE: Page size assumed for GPU zero-copy wrapping (16 KiB covers Apple Silicon; a multiple of 4 KiB).
QBLOCK

dequantize_rows: Dequantizes rows r0..r1 of m into out (f32, row-major, (r1-r0) * n_in).
dequantize_rows_q4: Dequantizes rows r0..r1 of m into out (f32 row-major).
f16_to_f32
from_mlx_q4: Repacks an MLX affine 4-bit matrix (w = scale * q + bias per group of group elements, eight nibbles per little-endian u32, low nibble first) into Q4_0 layout plus per-group offsets: scale * q + bias = scale * (q - 8) + (bias + 8 * scale). The nibbles carry over unchanged.
mlx_dequant_row: Dequantizes one row of an MLX affine matrix (any bit width dividing 32) into out.
quantize_act
quantize_bf16: Quantizes a bf16 row-major matrix. Symmetric round-to-nearest, scale = absmax / 127 per block.
quantize_q4