Module transform 

H.264 4×4 residual transform and quantization (spec §8.5 / §8.6).

H.264 uses a small integer approximation of the DCT — a 4×4 “core” transform whose scaling is folded into the quantizer, so it is exactly invertible in integer arithmetic. This module implements the forward path (encoder: residual → coefficients → quantized levels) and the inverse path (decoder and encoder-reconstruction: levels → coefficients → residual).

Correctness here is non-negotiable: the levels we emit are dequantized by every conforming decoder using the exact spec tables below, so the forward quantizer must be the faithful inverse of that process.

Constants

QUANT_FF_OH

QUANT_MF_OH

Functions

dequantize

Dequantizes levels to scaled coefficients (spec §8.5.12.1, flat scaling list so LevelScale = 16 · normAdjust).

dequantize_8x8

Dequantizes an 8×8 block (spec §8.5.13.1) with a per-position weight scale (raster order, 16 = flat). LevelScale8x8 = weight · normAdjust8x8.

dequantize_weighted

Dequantizes with a per-position weight scale (weightScale4x4 in raster order, 16 = flat) — High-profile scaling matrices (spec §8.5.12.1, LevelScale = weightScale · normAdjust).

forward_core

Forward core transform W = Cf · X · Cfᵀ over a row-major 4×4 block. The output coefficients are pre-quantization (scaling lives in the quantizer).

forward_core_8x8

Forward 8×8 core transform (rows then columns) — the encoder counterpart of inverse_core_8x8. Output are un-normalized transform coefficients for quantize_8x8; the normalization lives in the quant/dequant scale.

forward_dct_blocks

Forward core transform over a batch of 4×4 residual blocks — the encoder’s whole-macroblock DCT, mirroring x264’s sub16x16_dct. SIMD eight at a time (i32x8, AVX2 width), then four (i32x4), then a scalar tail.

forward_quant

Convenience: full forward path, residual → quantized levels (default dead-zones: 3 for intra, 6 for inter).

forward_quant_chroma_dc

Forward transform + quantization of a chroma DC block (4 coeffs, spec §8.5.11).

forward_quant_luma_dc

Forward transform + quantization of the 16 luma DC coefficients of an I_16x16 macroblock (spec §8.5.10). Input/output are row-major 4×4.

hadamard_2x2

2×2 Hadamard for a chroma DC block (its own inverse up to scale).

hadamard_4x4

4×4 Hadamard transform (rows then columns), used for the I_16x16 luma DC block. Symmetric, so the same routine serves forward and inverse.

inverse_core

Inverse core transform + final normalization, turning dequantized coefficients back into a residual block (spec §8.5.12.2: (f + 32) >> 6).

inverse_core_8x8

Inverse 8×8 core transform + normalization ((x + 32) >> 6), rows then columns (non-separable, like the 4×4 — the order is fixed by the spec).

inverse_dct_blocks

Inverse core transform + normalization over a batch of dequantized 4×4 blocks — the whole-macroblock IDCT, mirroring x264’s add16x16_idct. SIMD eight at a time (i32x8), then four (i32x4), then a scalar tail. Bit-identical to inverse_core per block. (Add-prediction + clip stays per-block at the call site, where the prediction layout lives.)

inverse_quant

Convenience: full inverse path, quantized levels → reconstructed residual.

inverse_quant_8x8

Convenience: full inverse 8×8 path, levels → reconstructed residual.

inverse_quant_chroma_dc

Inverse quantization + transform of a chroma DC block (spec §8.5.11.2).

inverse_quant_chroma_dc_weighted

inverse_quant_chroma_dc with the scaling matrix’s DC weight.

inverse_quant_luma_dc

Inverse quantization + transform of the I_16x16 luma DC block, returning the reconstructed DC values to scatter into each 4×4 luma block (spec §8.5.10).

inverse_quant_luma_dc_weighted

inverse_quant_luma_dc with the scaling matrix’s DC weight (w00, the raster (0,0) entry; 16 = flat).

quant_dz_ff

The 8-entry deadzone offset table FF[pos] = round(F / MF) reproducing our (|c|·MF + F) >> qbits deadzone (F = 2^qbits / dz_div) inside openh264’s ((|c| + FF)·MF_oh) >> 16 quantizer — so the asm WelsQuant*4x4 kernels quantize bit-identically to our scalar quantize. Pair with QUANT_MF_OH[qp] as MF.

quantize

Quantizes forward-transform coefficients to levels. intra selects the rounding dead-zone offset (1/3 for intra, 1/6 for inter). Scalar quantization. dz_div sets the rounding dead-zone: the offset added before the right shift is 2^qbits / dz_div, so a smaller dz_div rounds up more (higher quality, more bits). Typical values: 6 for inter, 3 for an I-frame that serves as a reference, 2 for all-intra (where the larger offset is a net rate-distortion win — better-quantized blocks predict their neighbors better, shrinking downstream residuals).

quantize_8x8

Forward-quantizes an 8×8 coefficient block (encoder). level = (|c|·MF + F) >> qbits, qbits = 16+QP/6, deadzone F = 2^qbits / dz_div (like the 4×4 path). weight is the per-position scaling-list value (raster; 16 = flat) — the matched inverse of dequantize_8x8, so dequantize_8x8(quantize_8x8(c)) ≈ c.

satd_4x4_sum

SATD over a slice of 4×4 residual blocks (SIMD four at a time, scalar tail). This is the cost kernel for motion estimation and RD mode decision.

trellis_quant

Rate-distortion–optimized (“trellis”) quantization of a 4×4 residual’s transform coefficients. For each coefficient it chooses between the scalar level and one lower (down to zero) to minimize J = distortion + λ·rate, trading a few bits of coefficient coding for small reconstruction error. lambda is the mode-decision Lagrangian (pixel-SSD domain). Encoder-only — the output is still a valid set of levels any decoder reconstructs.