pub enum DType {
Show 16 variants
F32,
F16,
BF16,
I8,
I32,
Q4_0,
Q4_1,
Q5_0,
Q5_1,
Q8_0,
Q2_K,
Q3_K,
Q4_K,
Q5_K,
Q6_K,
Q8_K,
}Expand description
The element type of a tensor.
This enumerates every numeric format the Kopitiam Runtime can hold in a
tensor, including the block-quantized formats used by GGUF weights. It
deliberately does not say how a format is computed on — a kernel may
dequantize a DType::Q4_0 block to f32 before multiplying, or it may
have a fused quantized path. That is kopitiam-kernels’ business; this
type only describes what the bytes mean.
§Why block-quantized formats are dtypes and not a separate concept
It is tempting to model quantized weights as “a f32 tensor with a
compression scheme attached”. That is the wrong abstraction: a Q4_0
tensor genuinely cannot be indexed elementwise without decoding a whole
block, so pretending it is a float tensor produces an API that lies. By
making quantization a property of the element type, every consumer is
forced to ask “can I index this?” (see DType::is_quantized) instead
of finding out at runtime.
Variants§
F32
IEEE 754 single precision.
F16
IEEE 754 half precision.
BF16
bfloat16: same exponent range as f32, fewer mantissa bits. The
usual format for model weights that must survive f32 dynamic range
without paying for 32 bits.
I8
Signed 8-bit integer.
I32
Signed 32-bit integer, mostly for token ids and indices.
Q4_0
4-bit block quantization, symmetric, 32 elements per block plus one
f16 scale.
Q4_1
4-bit block quantization, asymmetric: 32 elements per block plus an
f16 scale and an f16 minimum.
Q5_0
5-bit block quantization, symmetric, 32 elements per block.
Q5_1
5-bit block quantization, asymmetric, 32 elements per block.
Q8_0
8-bit block quantization, symmetric, 32 elements per block plus one
f16 scale.
Q2_K
ggml “K-quant” 2-bit format. Superblock of 256 elements: 16 sub-block
scales+mins packed 4 bits each, 2-bit quants, an f16 super-scale and
an f16 super-min. 84 bytes per superblock.
Q3_K
ggml “K-quant” 3-bit format. Superblock of 256: a high-bit mask, 2-bit
low quants, 16 six-bit sub-block scales packed into 12 bytes, and one
f16 super-scale. 110 bytes per superblock.
Q4_K
ggml “K-quant” 4-bit format (the Q4_K_M workhorse). Superblock of
256: f16 super-scale, f16 super-min, 8 six-bit sub-block scales and
8 six-bit sub-block mins packed into 12 bytes, then 4-bit quants. 144
bytes per superblock.
Q5_K
ggml “K-quant” 5-bit format. Like Self::Q4_K plus a 32-byte
high-bit field promoting each 4-bit quant to 5 bits. 176 bytes per
superblock.
Q6_K
ggml “K-quant” 6-bit format. Superblock of 256: 4-bit low quants, 2-bit
high quants, 16 signed i8 sub-block scales and one f16 super-scale.
210 bytes per superblock.
Q8_K
ggml “K-quant” 8-bit format. Superblock of 256: an f32 scale, 256
signed i8 quants and 16 i16 group sums. 292 bytes per superblock.
Used by ggml only as an intermediate activation type for K-quant dot
products, never as an on-disk weight format; supported here for
completeness. f32 (not f16) scale, unlike every other quant.
Implementations§
Source§impl DType
impl DType
Sourcepub const fn block_size(self) -> usize
pub const fn block_size(self) -> usize
Number of tensor elements encoded per storage block.
Non-quantized types are trivially one element per “block”. Quantized
types pack Self::block_size elements into Self::block_bytes
bytes, which is why a quantized tensor’s element count must be a
multiple of this.
Sourcepub const fn block_bytes(self) -> usize
pub const fn block_bytes(self) -> usize
Bytes occupied by one block of Self::block_size elements.
The quantized numbers below are the on-disk GGUF block layouts:
Q4_0 is a 2-byte f16 scale plus 32 4-bit weights (16 bytes) = 18;
Q4_1 adds a 2-byte minimum = 20; Q5_0 adds a 4-byte high-bit field to
Q4_0 = 22; Q5_1 likewise on Q4_1 = 24; Q8_0 is a 2-byte scale plus 32
bytes = 34.
The K-quant super-block sizes below are the sizeof(block_qX_K)
static-asserted in ggml’s ggml-common.h (MIT), with QK_K = 256
and K_SCALE_SIZE = 12:
- Q2_K =
2*f16 + QK_K/16 + QK_K/4= 4 + 16 + 64 = 84 - Q3_K =
f16 + QK_K/8 + QK_K/4 + 12= 2 + 32 + 64 + 12 = 110 - Q4_K =
2*f16 + 12 + QK_K/2= 4 + 12 + 128 = 144 - Q5_K =
2*f16 + 12 + QK_K/2 + QK_K/8= 4 + 12 + 128 + 32 = 176 - Q6_K =
f16 + QK_K/16 + 3*QK_K/4= 2 + 16 + 192 = 210 - Q8_K =
f32 + QK_K + QK_K/16*i16= 4 + 256 + 32 = 292
Sourcepub const fn is_quantized(self) -> bool
pub const fn is_quantized(self) -> bool
Whether this type packs multiple elements into a shared block with its own scale, and therefore cannot be indexed one element at a time.
Sourcepub const fn is_float(self) -> bool
pub const fn is_float(self) -> bool
Whether this type is a floating-point format that kernels can compute on directly.
Sourcepub const fn storage_bytes(self, elements: usize) -> Option<usize>
pub const fn storage_bytes(self, elements: usize) -> Option<usize>
Storage bytes needed to hold elements values of this type.
Returns None when elements is not a whole number of blocks, which
for a quantized type is not a representable tensor rather than a
rounding question — surfacing it as None keeps callers from
silently allocating a buffer that cannot hold what they asked for.