pub enum DType {
F32,
F16,
BF16,
I8,
I32,
Q4_0,
Q4_1,
Q5_0,
Q5_1,
Q8_0,
}Expand description
The element type of a tensor.
This enumerates every numeric format the Kopitiam Runtime can hold in a
tensor, including the block-quantized formats used by GGUF weights. It
deliberately does not say how a format is computed on — a kernel may
dequantize a DType::Q4_0 block to f32 before multiplying, or it may
have a fused quantized path. That is kopitiam-kernels’ business; this
type only describes what the bytes mean.
§Why block-quantized formats are dtypes and not a separate concept
It is tempting to model quantized weights as “a f32 tensor with a
compression scheme attached”. That is the wrong abstraction: a Q4_0
tensor genuinely cannot be indexed elementwise without decoding a whole
block, so pretending it is a float tensor produces an API that lies. By
making quantization a property of the element type, every consumer is
forced to ask “can I index this?” (see DType::is_quantized) instead
of finding out at runtime.
Variants§
F32
IEEE 754 single precision.
F16
IEEE 754 half precision.
BF16
bfloat16: same exponent range as f32, fewer mantissa bits. The
usual format for model weights that must survive f32 dynamic range
without paying for 32 bits.
I8
Signed 8-bit integer.
I32
Signed 32-bit integer, mostly for token ids and indices.
Q4_0
4-bit block quantization, symmetric, 32 elements per block plus one
f16 scale.
Q4_1
4-bit block quantization, asymmetric: 32 elements per block plus an
f16 scale and an f16 minimum.
Q5_0
5-bit block quantization, symmetric, 32 elements per block.
Q5_1
5-bit block quantization, asymmetric, 32 elements per block.
Q8_0
8-bit block quantization, symmetric, 32 elements per block plus one
f16 scale.
Implementations§
Source§impl DType
impl DType
Sourcepub const fn block_size(self) -> usize
pub const fn block_size(self) -> usize
Number of tensor elements encoded per storage block.
Non-quantized types are trivially one element per “block”. Quantized
types pack Self::block_size elements into Self::block_bytes
bytes, which is why a quantized tensor’s element count must be a
multiple of this.
Sourcepub const fn block_bytes(self) -> usize
pub const fn block_bytes(self) -> usize
Bytes occupied by one block of Self::block_size elements.
The quantized numbers below are the on-disk GGUF block layouts:
Q4_0 is a 2-byte f16 scale plus 32 4-bit weights (16 bytes) = 18;
Q4_1 adds a 2-byte minimum = 20; Q5_0 adds a 4-byte high-bit field to
Q4_0 = 22; Q5_1 likewise on Q4_1 = 24; Q8_0 is a 2-byte scale plus 32
bytes = 34.
Sourcepub const fn is_quantized(self) -> bool
pub const fn is_quantized(self) -> bool
Whether this type packs multiple elements into a shared block with its own scale, and therefore cannot be indexed one element at a time.
Sourcepub const fn is_float(self) -> bool
pub const fn is_float(self) -> bool
Whether this type is a floating-point format that kernels can compute on directly.
Sourcepub const fn storage_bytes(self, elements: usize) -> Option<usize>
pub const fn storage_bytes(self, elements: usize) -> Option<usize>
Storage bytes needed to hold elements values of this type.
Returns None when elements is not a whole number of blocks, which
for a quantized type is not a representable tensor rather than a
rounding question — surfacing it as None keeps callers from
silently allocating a buffer that cannot hold what they asked for.