Module fp8_ffi 

FP8-ACT PREFILL (MEMRA_PP_FP8=1): cuBLASLt FP8-E4M3 TN GEMM for the F8-E4M3-origin projections.

Probe verdict 2026-07-08 (probe/fp8_lt_prefill.cu, JSONL row in research/tune-data): cuBLASLt FP8 GEMM runs 620-795 TF at the 27B prefill shapes vs 47-72 TF for the qmatvec_gemm_q8_0 class those weights ride today (46.5% of pp GPU time) — projected ~1.85x pp from the F8-native layers alone. The weight side is EXACT: the checkpoint’s raw e4m3 bytes + per-tensor f32 weight_scale are stashed at load next to the Q8_0 re-encode (GpuTensor::Quant { fp8 }, following the cutlass optional-operand precedent). The only new rounding vs today is the ACTIVATION: f32 -> e4m3 with ONE per-batch scalar scale (amax/448) instead of q8_1’s per-32 int8 — finer mantissa lost, coarser scale granularity; the run-gen argmax gate arbitrates.

Dispatch: matmul/matmul_pre m>=16 arms ONLY (prefill). Decode (m<16) keeps the Q8_0 dp4a/MMVQ chain bit-for-bit — the spec-exactness law is untouched, and the m=K+1 verify tier (m<=9) never reaches this path.

All device work (amax reduce, scale finalize, e4m3 quantize, cublasLtMatmul) runs on the one gpu.stream inside a single C-ABI call (cu/fp8_prefill.cu) — no host sync anywhere: the act scale is folded with weight_scale into a device scalar fed to the GEMM’s B_SCALE_POINTER (per-token OUTER_VEC B-scales are NOT supported on sm_120 — probed; scalar scales verified exact there).

Structs

Fp8Scratch

Resident scratch for the FP8 prefill GEMM (mirrors CutlassScratch): the quantized activation (grown to the largest m*k seen), the 4-float scale block ([0]=amax, [1]=quant mul, [2]=folded B_SCALE — the GEMM desc holds a POINTER to slot 2, so the buffer must be resident/stable), and the cuBLASLt workspace (64MB, the probe’s size). Single GPU worker => no concurrent use; the Mutex guards lazy build/grow only (matches moe_cache / cutlass_scratch).

Functions

blk_native_nan_refused

Tensors declined by the native block-128 arm’s NaN precondition this process.

fp8_blk_gpu_enabled

MEMRA_FP8_BLK_GPU=1 gate (default OFF; ARM B’, lane fp8-gemm-arm 2026-08-03): block-128 FP8 safetensors weights dequant to Q8_0 ON THE GPU at load instead of host-dequant + host-re-encode. Bit-parity with the CPU path is a kernel-check gate (fp8-blk-gpu arm) and a real-checkpoint argmax gate — the flag exists because the CPU path stays default until both are green on the 5090.

fp8_blk_mmq_native_enabled

Same tile, NATIVE-RESIDENT operand source: a QT_F8_E4M3_BLK tensor’s own blk grid, the checkpoint’s single copy with no slab and no stash. DEFAULT ON since lane/fp8-blk128-decode (2026-08-05); MEMRA_FP8_MMQ=0 is the narrow seam back to dequant-per-call.

fp8_mmq_enabled

MEMRA_FP8_MMQ=1 gate for the per-block MMQ tile’s STASH operand source (default OFF; lane/fp8-mmq 2026-08-04): a SECOND e4m3 copy uploaded next to an already-resident Q8_0 slab, spending from MEMRA_PP_FP8_BUDGET_MB.

fp8_mmq_hits

Number of prefill GEMMs that went through the per-block FP8 MMQ tile so far this process.

fp8_mmq_ledger

entries, gate_off, hits, no_operand, bad_shape, bad_scale, nan_refused — the full ledger. entries is incremented before any guard, so entries == 0 means no prefill GEMM reached the hook at all (a dispatch-wiring fact), while gate_off == entries means the flag was not seen.

note_blk_native_nan_refused

pp_fp8_enabled

MEMRA_PP_FP8=1 gate (default OFF), read once. Gates BOTH the loader stash (model.rs) and the prefill dispatch — unset means zero VRAM / zero dispatch change.

st_e4m3_blk_enabled

Native residency for the BLOCK-128 e4m3 scale class (QT_F8_E4M3_BLK, lane/fp8-blk128-decode 2026-08-05) — the Qwen-official FP8 class that st_e4m3_enabled’s arm deliberately excludes.

st_e4m3_enabled

F8-E4M3-origin safetensors projections load as RAW e4m3 (QT_F8_E4M3) instead of the Q8_0 re-encode. NEW NUMERIC CONFIG: decode reads the checkpoint’s own e4m3 precision (the Q8_0 re-encode was a lossy extra hop) via qmatvec_e4m3_mmvq; prefill (m>=16) rides the cuBLASLt FP8 GEMM on the SAME resident bytes — one weight copy total (frees the ~GBs the MEMRA_PP_FP8 stash duplicated, no budget cap needed). Superset relationship: with this on, MEMRA_PP_FP8 and its budget are irrelevant for F8-origin tensors (they never surface as Q8_0, so the stash arm never fires).