Expand description
Byte-to-f32 dequantization dispatched on GgmlDType, for callers holding a raw GGUF
tensor byte slice and its declared dtype (the common shape at a model-loader’s from_gguf
site).
Centralizes what used to be independently hand-rolled in 10 fuel-transformers model
files (dequant_bytes_to_f32 + cpu_dequant_q4_0_bytes, copy-pasted identically across
lazy_quantized_{llama,qwen2,qwen3,qwen3_moe,gemma3,glm4,lfm2,phi3,smollm3,t5}.rs), each
wiring only F32/F16/BF16/Q4_0 and erroring on the other 10 GgmlDType variants —
including Q6_K, llama.cpp’s ordinary choice for output.weight/token_embd.weight even
in an otherwise-Q4_0 checkpoint, so a real-world GGUF file routinely hit the catch-all.
The Q4_0 arm those files DID wire was also a hand re-derivation of the block math,
bypassing GgmlType::to_float entirely — so the tested implementation was not the one
executing.
14 of the 15 GgmlDType variants are wired here. The one exclusion: Q8_1 declines with
a typed error citing GAP-125, because BlockQ8_1::to_float is unimplemented!() upstream
(a real numerics gap in this crate, not a dispatch-layer one — wiring it here would only
trade an Err for a panic).
⚠️ This centralization is not purely subtractive. The ten hand-rolled implementations it
replaces were each numerically incomplete but memory-safe; the first version of the single
function replacing them introduced a real soundness bug (an unaligned &[u8] cast to
&[T] for a T needing alignment 2+ — see [cast_and_dequant]’s doc), caught before merge
by review, not by any test in the original diff. Fixed with an alignment check, an
alternate copy-to-Vec<T> path for the misaligned case, and a Miri-verified born-red.
Functions§
- dequant_
ggml_ bytes - Dequantize a raw GGUF tensor byte slice of the given
GgmlDTypetof32.