Enum DType 

pub enum DType {
    F32,
    F16,
    BF16,
    I8,
    I32,
    Q4_0,
    Q4_1,
    Q5_0,
    Q5_1,
    Q8_0,
    Q2_K,
    Q3_K,
    Q4_K,
    Q5_K,
    Q6_K,
    Q8_K,
}

The element type of a tensor.

This enumerates every numeric format the Kopitiam Runtime can hold in a tensor, including the block-quantized formats used by GGUF weights. It deliberately does not say how a format is computed on — a kernel may dequantize a DType::Q4_0 block to f32 before multiplying, or it may have a fused quantized path. That is kopitiam-kernels’ business; this type only describes what the bytes mean.

Why block-quantized formats are dtypes and not a separate concept

It is tempting to model quantized weights as “a f32 tensor with a compression scheme attached”. That is the wrong abstraction: a Q4_0 tensor genuinely cannot be indexed elementwise without decoding a whole block, so pretending it is a float tensor produces an API that lies. By making quantization a property of the element type, every consumer is forced to ask “can I index this?” (see DType::is_quantized) instead of finding out at runtime.

Variants

F32

IEEE 754 single precision.

F16

IEEE 754 half precision.

BF16

bfloat16: same exponent range as f32, fewer mantissa bits. The usual format for model weights that must survive f32 dynamic range without paying for 32 bits.

I8

Signed 8-bit integer.

I32

Signed 32-bit integer, mostly for token ids and indices.

Q4_0

4-bit block quantization, symmetric, 32 elements per block plus one f16 scale.

Q4_1

4-bit block quantization, asymmetric: 32 elements per block plus an f16 scale and an f16 minimum.

Q5_0

5-bit block quantization, symmetric, 32 elements per block.

Q5_1

5-bit block quantization, asymmetric, 32 elements per block.

Q8_0

8-bit block quantization, symmetric, 32 elements per block plus one f16 scale.

Q2_K

ggml “K-quant” 2-bit format. Superblock of 256 elements: 16 sub-block scales+mins packed 4 bits each, 2-bit quants, an f16 super-scale and an f16 super-min. 84 bytes per superblock.

Q3_K

ggml “K-quant” 3-bit format. Superblock of 256: a high-bit mask, 2-bit low quants, 16 six-bit sub-block scales packed into 12 bytes, and one f16 super-scale. 110 bytes per superblock.

Q4_K

ggml “K-quant” 4-bit format (the Q4_K_M workhorse). Superblock of 256: f16 super-scale, f16 super-min, 8 six-bit sub-block scales and 8 six-bit sub-block mins packed into 12 bytes, then 4-bit quants. 144 bytes per superblock.

Q5_K

ggml “K-quant” 5-bit format. Like Self::Q4_K plus a 32-byte high-bit field promoting each 4-bit quant to 5 bits. 176 bytes per superblock.

Q6_K

ggml “K-quant” 6-bit format. Superblock of 256: 4-bit low quants, 2-bit high quants, 16 signed i8 sub-block scales and one f16 super-scale. 210 bytes per superblock.

Q8_K

ggml “K-quant” 8-bit format. Superblock of 256: an f32 scale, 256 signed i8 quants and 16 i16 group sums. 292 bytes per superblock. Used by ggml only as an intermediate activation type for K-quant dot products, never as an on-disk weight format; supported here for completeness. f32 (not f16) scale, unlike every other quant.

Implementations

impl DType

pub const fn block_size(self) -> usize

Number of tensor elements encoded per storage block.

Non-quantized types are trivially one element per “block”. Quantized types pack Self::block_size elements into Self::block_bytes bytes, which is why a quantized tensor’s element count must be a multiple of this.

pub const fn block_bytes(self) -> usize

Bytes occupied by one block of Self::block_size elements.

The quantized numbers below are the on-disk GGUF block layouts: Q4_0 is a 2-byte f16 scale plus 32 4-bit weights (16 bytes) = 18; Q4_1 adds a 2-byte minimum = 20; Q5_0 adds a 4-byte high-bit field to Q4_0 = 22; Q5_1 likewise on Q4_1 = 24; Q8_0 is a 2-byte scale plus 32 bytes = 34.

The K-quant super-block sizes below are the sizeof(block_qX_K) static-asserted in ggml’s ggml-common.h (MIT), with QK_K = 256 and K_SCALE_SIZE = 12:

• Q2_K = 2*f16 + QK_K/16 + QK_K/4 = 4 + 16 + 64 = 84
• Q3_K = f16 + QK_K/8 + QK_K/4 + 12 = 2 + 32 + 64 + 12 = 110
• Q4_K = 2*f16 + 12 + QK_K/2 = 4 + 12 + 128 = 144
• Q5_K = 2*f16 + 12 + QK_K/2 + QK_K/8 = 4 + 12 + 128 + 32 = 176
• Q6_K = f16 + QK_K/16 + 3*QK_K/4 = 2 + 16 + 192 = 210
• Q8_K = f32 + QK_K + QK_K/16*i16 = 4 + 256 + 32 = 292

pub const fn is_quantized(self) -> bool

Whether this type packs multiple elements into a shared block with its own scale, and therefore cannot be indexed one element at a time.

pub const fn is_float(self) -> bool

Whether this type is a floating-point format that kernels can compute on directly.

pub const fn storage_bytes(self, elements: usize) -> Option<usize>

Storage bytes needed to hold elements values of this type.

Returns None when elements is not a whole number of blocks, which for a quantized type is not a representable tensor rather than a rounding question — surfacing it as None keeps callers from silently allocating a buffer that cannot hold what they asked for.

Trait Implementations

impl Clone for DType

fn clone(&self) -> DType

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for DType

impl Debug for DType

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Display for DType

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Eq for DType

impl Hash for DType

fn hash<__H>(&self, state: &mut __H)

where __H: Hasher,

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for DType

fn eq(&self, other: &DType) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for DType