Skip to main content

Module quantization

Module quantization 

Source
Expand description

§GGUF Quantization Engine

Provides pure-Rust, zero-dependency de-quantization and high-performance dot product kernels for standard GGUF quantization formats:

  • Q8_0: 8-bit quantized weights with FP16 block scales (34 bytes per 32 elements).
  • Q4_0: 4-bit quantized weights with FP16 block scales (18 bytes per 32 elements).

Structs§

QuantizedLinearModel
A memory-efficient quantized linear projection model (Q8_0 or Q4_0). Reduces LM head weight memory by up to 75%–87% compared to FP32.

Enums§

QuantType
Quantization format type.

Constants§

QK_K
Number of elements per GGML quantization block.

Functions§

dequantize_q4_0
Dequantizes a Q4_0 byte slice into an FP32 output buffer.
dequantize_q8_0
Dequantizes a Q8_0 byte slice into an FP32 output buffer.
f16_to_f32
Converts an IEEE 754 half-precision (FP16) 16-bit integer to a single-precision (FP32) float.
f32_to_f16
Converts a single-precision float (FP32) into an IEEE 754 half-precision (FP16) integer.
quantize_q8_0_block
Quantizes an FP32 slice of 32 elements into a Q8_0 block (34 bytes).
vec_dot_q4_0
Computes the dot product between a Q4_0 quantized weight row and an FP32 vector directly without full dequantization.
vec_dot_q8_0
Computes the dot product between a Q8_0 quantized weight row and an FP32 vector directly without full dequantization.
vec_dot_q8_0_scalar
Fallback scalar implementation of Q8_0 dot product.