Module repack 

Interleaved Q4_K / Q5_K × Q8_K and Q8_0 × Q8 GEMV (llama.cpp repack layouts).

• Q4_K: packs 8 rows into block_q4_Kx8 (make_block_q4_Kx8).
• Q5_K: packs 8 rows into block_q5_Kx8 (make_block_q5_Kx8).
• Q8_0: packs 4 rows into block_q8_0x4 (make_block_q8_0x4) with 4-byte interleave for NEON SDOT ggml_gemv_q8_0_4x4_q8_0.
• Q4_0: packs 4 rows into block_q4_0x4 (make_block_q4_0x4) with 4-byte interleave + XOR 0x88888888 for ggml_gemv_q4_0_4x4_q8_0.

Gated on FERROX_CPU_INT_DOT, which ferrox and ferrox-server turn on by default (=0 opts out); off in the library so golden cross-validation stays reference-exact.

Structs

Q8ActsX4

A quad of up to Q8K_ACTS_X4_NC Q8_0 activations, pre-interleaved into the layout llama.cpp’s ggml_quantize_mat_q8_0_4x8 writes into block_q8_0x4 (arch/arm/repack.cpp): every 32-element block’s qs in 8-byte runs, plus the per-block per-row scales. Consumed by the i8mm 4x8 GEMMs; prepared once per matmul, same hoist as Q8KActsX4.

Q8KActsX4

Constants

Q4_0X4_BLOCK_BYTES

Bytes per interleaved block_q4_0x4 (4 × f16 d + 64 qs).

Q4_0X4_GEMM_NC

How many activations one gemm_q4_0x4_group pass keeps in flight.

Q4_0X4_INTERLEAVE

qs interleave width for ggml_gemv_q4_0_4x4_q8_0 (NEON SDOT). The DotProd-only default; q4_0x4_interleave picks 8 on i8mm hosts.

Q4_0X4_NROWS

Number of Q4_0 rows packed into one interleaved block.

Q4_KX8_BLOCK_BYTES

Bytes per interleaved block_q4_Kx8 (8 × f16 d + 8 × f16 dmin + 96 scales + 1024 qs).

Q4_KX8_GEMM_NC

How many activations one gemm_q4_kx8_group pass keeps in flight.

Q4_KX8_NROWS

Number of Q4_K rows packed into one interleaved block.

Q5_KX8_BLOCK_BYTES

Bytes per interleaved block_q5_Kx8 (8 × f16 d + 8 × f16 dmin + 96 scales + 256 qh + 1024 qs).

Q5_KX8_GEMM_NC

How many activations one gemm_q5_kx8_group pass keeps in flight.

Q5_KX8_NROWS

Number of Q5_K rows packed into one interleaved block.

Q6_KX8_BLOCK_BYTES

Bytes per interleaved block_q6_Kx8 (8×f16 d + 128 scales + 1024 ql + 512 qh).

Q6_KX8_GEMM_NC

Q6_KX8_NROWS

Number of Q6_K rows packed into one interleaved block.

Q8K_ACTS_X4_NC

A quad of up to Q8K_ACTS_X4_NC Q8_K activations, pre-interleaved into the layout llama.cpp’s ggml_quantize_mat_q8_K_4x8 writes into block_q8_Kx4 (ggml-cpu/repack.cpp): every super-block’s qs, the folded bsums pairs, and the per-block per-row scales.

Q8_0X4_BLOCK_BYTES

Bytes per interleaved block_q8_0x4 (4 × f16 d + 128 qs).

Q8_0X4_GEMM_NC

How many activations one gemm_q8_0x4_group pass keeps in flight. Four f32x4 accumulators plus the eight loaded weight vectors fit comfortably in NEON’s register file, so each weight load is amortized over four activations instead of being repeated per activation.

Q8_0X4_INTERLEAVE

qs interleave width for ggml_gemv_q8_0_4x4_q8_0 (NEON SDOT). The DotProd-only default; q8_0x4_interleave picks 8 on i8mm hosts.

Q8_0X4_NROWS

Number of Q8_0 rows packed into one interleaved block.

Functions

gemm_q4_0x4_group

GEMM counterpart of gemv_q4_0x4_group: one row-group (4 rows) against acts.len() activations at once. out is [row][act]: out[r * acts.len() + j].

gemm_q4_0x4_group_x4

gemm_q4_0x4_group against a pre-interleaved activation quad; interleave-8 packing only, quad prepared once per matmul by prepare_q8_acts_x4. out is [row][act]: out[r * tile.na + a].

gemm_q4_kx8_group

GEMM counterpart of gemv_q4_kx8_group: one row-group (8 rows) against acts.len() activations at once.

gemm_q4_kx8_group_x4

gemm_q4_kx8_group against a pre-interleaved activation quad.

gemm_q5_kx8_group

GEMM counterpart of gemv_q5_kx8_group: one row-group (8 rows) against acts.len() activations at once. out is [row][act]: out[r * acts.len() + j].

gemm_q5_kx8_group_x4

gemm_q5_kx8_group against a pre-interleaved activation quad; the Q5_K counterpart of gemm_q4_kx8_group_x4, with the same contract: interleave-8 packing only, quad prepared once per matmul by prepare_q8_k_acts_x4, out[r * tile.na + a].

gemm_q6_kx8_group

Multi-act GEMM for one Q6_Kx8 row-group; weight decode amortized across acts.

gemm_q6_kx8_group_x4

gemm_q6_kx8_group against a pre-interleaved activation quad; the Q6_K counterpart of gemm_q4_kx8_group_x4, with the same contract: interleave-8 packing only, quad prepared once per matmul by prepare_q8_k_acts_x4 (up to 4 activations, not Q6_KX8_GEMM_NC), out[r * tile.na + a].

gemm_q8_0x4_group

GEMM counterpart of gemv_q8_0x4_group: one row-group (4 rows) against acts.len() activations at once.

gemm_q8_0x4_group_x4

gemm_q8_0x4_group against a pre-interleaved activation quad; interleave-8 packing only, quad prepared once per matmul by prepare_q8_acts_x4. out is [row][act]: out[r * tile.na + a].

gemv_q4_0x4_group

One row-group (4 outputs) starting at group within a packed Q4_0x4 matrix.

gemv_q4_0x4_q8_0

GEMV: interleaved Q4_0 weights × Q8 activation → n_row_groups * 4 f32s. interleave must match the packing (4: NEON SDOT 4x4; 8: NEON 4x8).

gemv_q4_kx8_group

One row-group (8 outputs) starting at group within a packed matrix.

gemv_q4_kx8_q8_k

GEMV: interleaved Q4_K weights × Q8_K activation → n_row_groups * 8 f32s. Dispatches to NEON (both interleaves) / AVX2 (interleave 8) when available.

gemv_q5_kx8_group

One row-group (8 outputs) starting at group within a packed Q5_K matrix.

gemv_q5_kx8_q8_k

GEMV: interleaved Q5_K weights × Q8_K activation → n_row_groups * 8 f32s.

gemv_q6_kx8_group

gemv_q6_kx8_q8_k

gemv_q8_0x4_group

One row-group (4 outputs) starting at group within a packed Q8_0x4 matrix.

gemv_q8_0x4_q8_0

GEMV: interleaved Q8_0 weights × Q8 activation → n_row_groups * 4 f32s. interleave must match the packing (4: NEON SDOT 4x4; 8: NEON 4x8).

make_block_q4_0x4

Pack four canonical Q4_0 blocks (same column-block) into one block_q4_0x4. Nibble bytes are XOR-masked during interleave so NEON can unpack without explicit - 8 bias subtraction.

make_block_q4_kx8

Pack eight canonical Q4_K super-blocks (same column-block index) into one block_q4_Kx8. interleave is 4 (ARM DotProd) or 8 (x86 / ARM i8mm).

make_block_q5_kx8

Pack eight canonical Q5_K super-blocks (same column-block index) into one block_q5_Kx8. interleave is 4 (ARM DotProd) or 8 (x86 / ARM i8mm).

make_block_q6_kx8

Pack eight canonical Q6_K super-blocks into one block_q6_Kx8.

make_block_q8_0x4

Pack four canonical Q8_0 blocks (same column-block) into one block_q8_0x4. interleave is 4 (ARM 4x4) or 8 (4x8).

pack_q4_0_matrix_x4

Repack a Q4_0 matrix into interleaved block_q4_0x4 groups. Tail rows (not divisible by 4) are omitted; caller dots them with crate::dot_q4_0_q8.

pack_q4_k_matrix_x8

Repack a full Q4_K matrix (row-major canonical blocks) into interleaved block_q4_Kx8 groups. Rows not divisible by 8 are left out (caller handles the tail with per-row dots). interleave defaults via q4_kx8_interleave.

pack_q5_k_matrix_x8

Repack a full Q5_K matrix (row-major canonical blocks) into interleaved block_q5_Kx8 groups. Tail rows (not divisible by 8) are omitted.

pack_q6_k_matrix_x8

pack_q8_0_matrix_x4

Repack a Q8_0 matrix into interleaved block_q8_0x4 groups. Tail rows (not divisible by 4) are omitted; caller dots them with crate::dot_q8_0_q8.

prepare_q8_acts_x4

Interleave a quad of Q8_0 activations for the 4x8 i8mm GEMMs (llama.cpp ggml_quantize_mat_q8_0_4x8, minus the quantization we already did). Zero-pads when acts.len() < 4. Available on every target so the portable GEMMs — and the tests pinning the NEON kernels to them — run anywhere.

prepare_q8_k_acts_x4

Interleave a quad of activations for gemm_q4_kx8_group_x4 (llama.cpp ggml_quantize_mat_q8_K_4x8, minus the quantization we already did). Zero-pads when acts.len() < 4, matching what the kernel’s in-loop repack used to emit. Available on every target so the portable GEMM below — and the tests pinning the NEON kernel to it — run anywhere.

q4_0x4_gemm_uses_acts_x4

Whether gemm_q4_0x4_group_x4 is the fast Q4_0 batch path on this CPU: ARM i8mm with the interleave-8 layout (ggml_gemm_q4_0_4x8_q8_0).

q4_0x4_interleave

Preferred qs interleave width: 8 on ARM i8mm (ggml_gemm_q4_0_4x8_q8_0 via ggml_repack_get_optimal_repack_type), 4 on DotProd-only NEON and everywhere else (the scalar fallback handles either).

q4_kx8_gemm_uses_acts_x4

Whether gemm_q4_kx8_group_x4 is the fast Q4_K batch path on this CPU: ARM i8mm with the interleave-8 layout. Everywhere else preparing the quad buys nothing — x86 dispatches to AVX2 inside the per-activation GEMV — so callers should keep using gemm_q4_kx8_group and skip the tiles.

q4_kx8_interleave

Preferred qs interleave width for this CPU: 8 on x86 AVX2 and ARM i8mm (ggml_gemm_q4_K_8x8_q8_K), 4 on DotProd-only NEON (ggml_gemm_q4_K_8x4_q8_K).

q5_kx8_gemm_uses_acts_x4

Whether gemm_q5_kx8_group_x4 is the fast Q5_K batch path on this CPU: ARM i8mm with the interleave-8 layout (ggml_gemm_q5_K_8x8_q8_K).

q5_kx8_interleave

Preferred qs/qh interleave width: 8 on x86 AVX2 and ARM i8mm (ggml_gemm_q5_K_8x8_q8_K), 4 on DotProd-only NEON (8x4).

q6_kx8_gemm_uses_acts_x4

Whether gemm_q6_kx8_group_x4 is the fast Q6_K batch path on this CPU: ARM i8mm with the interleave-8 layout (ggml_gemm_q6_K_8x8_q8_K). The scalar Kx8 GEMM measured slower than the per-row NEON dot on ARM, so batch callers should use the Kx8 layout only when this returns true.

q6_kx8_interleave

Preferred ql/qh interleave width: 8 on x86 AVX2 and ARM i8mm (ggml_gemm_q6_K_8x8_q8_K), 4 on DotProd-only NEON.

q8_0x4_gemm_uses_acts_x4

Whether gemm_q8_0x4_group_x4 is the fast Q8_0 batch path on this CPU: ARM i8mm with the interleave-8 layout (ggml_gemm_q8_0_4x8_q8_0).

q8_0x4_interleave

Preferred qs interleave width: 8 on ARM i8mm (ggml_gemm_q8_0_4x8_q8_0 via ggml_repack_get_optimal_repack_type), 4 on DotProd-only NEON and everywhere else (the scalar fallback handles either).