Expand description
Precision / storage dtypes for the fabric. Compute stays f32 (WebGPU-baseline has no shader-f16),
but weights can LIVE on the GPU in half precision and be dequantized on-device — half the memory,
and the path real fp16/bf16 checkpoints take. Half is a packed storage tensor (2 values per u32
word); dequant() expands to a compute Tensor, Tensor::to_half() packs one down.
Structs§
- Half
- A half-precision tensor stored packed (2×16-bit per 32-bit word) in GPU memory.
- QRow
- Per-row (per-output-channel) quantized 2D matrix at
bits∈ {4,8}, packed 32/bits per word, with one scale per row — more accurate than a single per-tensor scale, and int4 is 1/8 the memory. - QTensor
- A per-tensor symmetric int8-quantized tensor (4 values packed per u32) plus its scale.
- Ternary
- A ternary-weight matrix (BitNet b1.58 family): weights ∈ {−1,0,+1} packed 16 per u32 (2 bits each), with a per-output-channel scale (absmean). The matmul is effectively multiply-free — each weight just adds, subtracts, or skips an activation. 1.58 bits/weight ≈ 1/16 the memory of f32.