Expand description
§GGUF Quantization Engine
Provides pure-Rust, zero-dependency de-quantization and high-performance dot product kernels for standard GGUF quantization formats:
- Q8_0: 8-bit quantized weights with FP16 block scales (34 bytes per 32 elements).
- Q4_0: 4-bit quantized weights with FP16 block scales (18 bytes per 32 elements).
Structs§
- Quantized
Linear Model - A memory-efficient quantized linear projection model (Q8_0 or Q4_0). Reduces LM head weight memory by up to 75%–87% compared to FP32.
Enums§
- Quant
Type - Quantization format type.
Constants§
- QK_K
- Number of elements per GGML quantization block.
Functions§
- dequantize_
q4_ 0 - Dequantizes a Q4_0 byte slice into an FP32 output buffer.
- dequantize_
q8_ 0 - Dequantizes a Q8_0 byte slice into an FP32 output buffer.
- f16_
to_ f32 - Converts an IEEE 754 half-precision (FP16) 16-bit integer to a single-precision (FP32) float.
- f32_
to_ f16 - Converts a single-precision float (FP32) into an IEEE 754 half-precision (FP16) integer.
- quantize_
q8_ 0_ block - Quantizes an FP32 slice of 32 elements into a Q8_0 block (34 bytes).
- vec_
dot_ q4_ 0 - Computes the dot product between a Q4_0 quantized weight row and an FP32 vector directly without full dequantization.
- vec_
dot_ q8_ 0 - Computes the dot product between a Q8_0 quantized weight row and an FP32 vector directly without full dequantization.
- vec_
dot_ q8_ 0_ scalar - Fallback scalar implementation of Q8_0 dot product.