Expand description
Group-wise 4-bit quantization support (GGUF q4_0-style layout).
Phase 2 ships the dequantize op as portable burn tensor ops (nibble
extraction via remainder/div — no bitwise ops or custom kernels
required), so it runs on any backend. A fused dequant-matmul CubeCL
kernel (dequantize tiles inside the matmul, avoiding a materialized f32
weight) is future work; until then dequantize_q4 materializes the
f32 weight and callers use the normal matmul path.
§Packed layout (GGUF q4_0)
Weights are grouped into blocks of group_size (32) values. Each block
stores 16 bytes: byte j’s low nibble is value j of the block and
its high nibble is value j + 16. The dequantized value is
(nibble - 8) * scale (symmetric quantization with an implicit zero
point of 8). Scales are stored per block as f32 on device (f16 scales
are widened at load time by the format adapter).
Constants§
- DEFAULT_
Q4_ GROUP_ SIZE - Default quantization group size (GGUF
q4_0).
Functions§
- dequantize_
q4 - Dequantizes packed 4-bit weights to f32.