Module weight_matrix 

WeightMatrix: a weight matrix that may live either as plain f32 (small dims, embeddings, synthetic test weights) or as raw Q8_0/Q4_0 block bytes loaded straight from a GGUF file, with no f32 expansion at load time. This is what lets ferrox load a multi-billion-parameter checkpoint without first blowing it up 4x in RAM: the loader (ferrox-models) hands tensors over still quantized, and every matmul call here dispatches to the fused dequant+dot kernels in ferrox-quant.

Enums

BatchActs

A batch of activations quantized once for reuse across several WeightMatrix::apply_batch_with_acts calls that read the same input (q/k/v on one normed batch; gate/up on another). Build with WeightMatrix::quantize_batch_acts. Q8_0/Q4_0 matrices consume BatchActs::Q8; the K-quants consume BatchActs::Q8K.

QuantKind

WeightBytes

Backing storage for a quantized weight matrix’s raw bytes: either an owned buffer (synthetic/test weights, or any tensor that had to be copied for some other reason) or a zero-copy view into a shared memory-mapped GGUF file. This is the fix for the “loader read everything into a fresh Vec” inefficiency: a real checkpoint’s resident memory should be the mmap itself, not a second copy of it, which is how llama.cpp’s mmap-based loader both avoid doubling a multi-hundred-gigabyte checkpoint’s memory footprint.

WeightMatrix

Functions

active_backend

The backend dense matmuls will actually use in this process, decided by the same cached env/probe reads dispatch uses. CUDA wins when both are compiled in, matching [WeightMatrix::apply_gpu]’s order.

cpu_int_dot_enabled

Whether CPU Q8_0 / Q4_0 / Q4_K / Q5_K / Q6_K matvec should quantize the activation to int8 and use the integer vec_dot path. Q4_K additionally lazy-repacks into interleaved block_q4_Kx8 for 8-wide GEMV; Q8_0 into block_q8_0x4 and Q4_0 into block_q4_0x4 for 4-wide GEMV.

cpu_int_dot_kind_supported

Which quant kinds take the CPU integer vec_dot path (activation quantized to Q8/Q8_K, int8xint8 dots) rather than the much slower f32 dequant-dot. cols matters: the K-quant kernels need a whole number of 256-element super-blocks, the legacy ones 32-element blocks.

cuda_matvec_kind_supported

Which quant kinds have a CUDA matvec kernel.

default_cpu_int_dot_on^⚠

Sets FERROX_CPU_INT_DOT=1 unless the caller already expressed a preference. Call from a binary’s startup, before any worker threads exist. See cpu_int_dot_enabled for why the default lives here rather than in the getter.

metal_matvec_kind_name

Which quant kinds have a Metal matvec kernel, as the kernel name [ferrox_metal::gpu::matvec_launch_meta] resolves.

metal_mul_mm_kind_supported

Which quant kinds have a Metal batched simdgroup GEMM (*_mul_mm_sg), the prefill path. A kind with a matvec but no GEMM still runs on Metal — as batch separate matvecs over the same weights, which is the 13.7x shape.

quant_kind_for

Maps a GGUF tensor’s on-disk dtype to the QuantKind a WeightMatrix uses to pick a fused dequant+dot kernel, or None for a dtype with no quantized kernel (F32, or one not implemented at all).