Crate fuel_quantized 

Backend-agnostic quantized (ggml/gguf block-format) numerics.

This crate owns the format-side of fuel’s quantization stack:

• k_quants — BlockQX block types, the GgmlType trait, and scalar reference impls of every fmt fuel supports.
• [avx] / [neon] — cfg-gated CPU SIMD vec_dot helpers used by GgmlType::vec_dot impls in k_quants. They live here (rather than in fuel-cpu-backend) because the orphan rule pins them to the crate that defines BlockQX.
• utils — quantization-time helpers shared by k_quants impls.
• cpu — CPU-side adapters that bridge Vec<BlockQX> to the backend-agnostic fuel_backend_contract::quantized::DynQuantizedStorage trait, plus the QuantizedDeviceKernels impl on fuel_cpu_backend::CpuBackendDevice.

Per-backend (CUDA, Metal, future Vulkan) quantized fast paths live in their own crates and implement the same trait pair from fuel-core-types.

Re-exports

pub use cpu::QuantizedType;

pub use cpu::cpu_from_data;

pub use cpu::cpu_zeros;

pub use dequant::dequant_ggml_bytes;

pub use k_quants::BlockQ2K;

pub use k_quants::BlockQ3K;

pub use k_quants::BlockQ4_0;

pub use k_quants::BlockQ4_1;

pub use k_quants::BlockQ4K;

pub use k_quants::BlockQ5_0;

pub use k_quants::BlockQ5_1;

pub use k_quants::BlockQ5K;

pub use k_quants::BlockQ6K;

pub use k_quants::BlockQ8_0;

pub use k_quants::BlockQ8_1;

pub use k_quants::BlockQ8K;

pub use k_quants::GgmlType;

pub use k_quants::K_SCALE_SIZE;

pub use k_quants::QK_K;

pub use k_quants::QK4_0;

pub use k_quants::QK4_1;

pub use k_quants::QK5_0;

pub use k_quants::QK5_1;

pub use k_quants::QK8_0;

pub use k_quants::QK8_1;

pub use k_quants::blocks_as_bytes;

pub use k_quants::blocks_from_bytes_cow;

pub use k_quants::matmul;

pub use k_quants::matmul_f16;

Modules

awq_repack

Host repack of a HuggingFace *-AWQ (GEMM-version) checkpoint into the buffers baracuda’s W4A16 AWQ kernel reads (ROADMAP row 31).

cpu

Format-agnostic CPU helpers for quantized storage.

dequant

Byte-to-f32 dequantization dispatched on GgmlDType, for callers holding a raw GGUF tensor byte slice and its declared dtype (the common shape at a model-loader’s from_gguf site).

iq4

IQ4_NL and IQ4_XS block formats (GAP-211 B1).

iq_grid_types

IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ1_S and IQ1_M block formats (GAP-211 B2): the grid-table IQ quantizations. DECODE-ONLY.

iq_grids

Lookup tables for the grid-based IQ formats (GAP-211 B2).

k_quants

tq_mx_types

TQ1_0, TQ2_0 (ternary) and MXFP4, NVFP4 (4-bit float) block formats (GAP-211 C). DECODE-ONLY.

utils

Enums

GgmlDType

The ggml block-format dtype tag. Mirrors llama.cpp’s ggml_type @9d57ce456c for the subset fuel supports (the classic 15 — F32/F16/BF16 plus Q4_0..Q8_1 and Q2K..Q8K, plus every IQ family: IQ4_NL/IQ4_XS (GAP-211 B1) and the decode-only IQ1_S/IQ1_M/IQ2_XXS/IQ2_XS/IQ2_S/IQ3_XXS/IQ3_S (GAP-211 B2); block geometry and the dequant formula verified against ggml-common.h and ggml-quants.c at that commit), plus the decode-only TQ1_0/TQ2_0/MXFP4/NVFP4 (GAP-211 C); NOT the other ggml types, which an unsupported-tensor GGUF load reports by name instead).

Traits

DynQuantizedStorage

Object-safe per-backend quantized storage. Each backend (CPU, CUDA, Metal, …) supplies a concrete type that implements this trait; fuel-core holds them as Box<dyn DynQuantizedStorage> and dispatches without naming backends.

QuantizedDeviceKernels

Backend device → quantized-kernel constructors. fuel-core looks this up per Device to allocate fresh QStorage of a given dtype, or to load pre-quantized bytes.