Enum GpuTensor 

pub enum GpuTensor {
    Quant {
        bytes: CudaSlice<u8>,
        qtype: i32,
        row_bytes: usize,
        ne: Vec<u64>,
        scale: f32,
        rp: bool,
        fp8: Option<Fp8Weight>,
        rp4: Option<CudaSlice<u8>>,
        f16: Option<CudaSlice<u8>>,
    },
    Float {
        data: CudaSlice<f32>,
        ne: Vec<u64>,
    },
    FloatBf16 {
        data: CudaSlice<u8>,
        ne: Vec<u64>,
    },
}

A weight tensor resident on GPU. Quantized weights stay in GGUF block bytes (Quant); small non-quant tensors (norms, sometimes embed/lm_head) are kept dequantized as f32 (Float). This keeps VRAM ~= on-disk quant size (fixes the f32-on-load OOM).

Variants

Quant

Fields

bytes: CudaSlice<u8>

qtype: i32

row_bytes: usize

ne: Vec<u64>

scale: f32

rp: bool

SPLIT-PLANE walk-order repack (A6, 2026-07-04): NVFP4 matmul weights are repacked at load into [quant plane out_f x in_f/64 x 32B][scale plane out_f x in_f/64 x 4B] — same bytes, same total size, but a lane’s per-group weight read becomes ONE 16B-aligned LDG.128 + a dense 4B scale word instead of 5 scattered 4B LDGs at 36B stride (the “18B straggle”). Every consumer kernel has an _rp twin (bit-identical: pure byte permutation, same dot order). rp=false = original GGUF block layout (all other dtypes, MoE-staged expert bytes, MEMRA_RP=0 escape).

fp8: Option<Fp8Weight>

FP8-ACT PREFILL operand (MEMRA_PP_FP8=1, probe verdict 2026-07-08): the checkpoint’s RAW e4m3 bytes + per-tensor f32 weight_scale, stashed ALONGSIDE the Q8_0 re-encode for the F8-E4M3-origin 2D projections (~1 B/w extra on those layers). bytes stays Q8_0 so decode (dp4a/MMVQ) is untouched; only the m>=16 prefill dispatch (cuBLASLt FP8 TN, fp8_ffi.rs) reads this. None unless the env is set at load (zero VRAM cost by default).

rp4: Option<CudaSlice<u8>>

Q4_0 SPLIT-PLANE MIRROR (2026-07-10, the 18B-straggle cure for decode): qs plane [out_f x nblk x 16B] + d plane [out_f x nblk x 2B] built device-side at model load (q4_0_split_rp_build) for decode-hot trunk weights. Raw bytes stay resident — prefill (gemm/MMQ) and Stage-A read those; the m<=8 mmvq/batched/fused dispatch reads this when present (_rp twins; microprobe m=1 1.34x, m=3 1.17x, bitwise). None everywhere except where the arch-load hook opted in (VRAM cost = weight size).

f16: Option<CudaSlice<u8>>

FP16 DEQUANT MIRROR (MEMRA_PP_F16=1, probe 2026-07-26): row-major fp16 of a 2D Q8_0 projection, built device-side at load (f16_ffi::build_q8_f16). bytes stay Q8_0 so decode is untouched; the m>=16 prefill dispatch (cuBLASLt FP16 TN, 611-687 TF vs MMQ’s ~200 TF class) reads this. None unless the env is set (VRAM = 2 B/w extra).

Float

Fields

data: CudaSlice<f32>

ne: Vec<u64>

FloatBf16

BF16-RESIDENT full-precision matmul weight (MEMRA_FULL_PREC only). Holds the checkpoint’s raw bf16 bytes (u8, little-endian u16 pairs) — 2 B/w vs the 4 B/w a Float f32 materialization would cost, so the 9B trunk stays ~18GB in VRAM instead of ~36GB. Consumed via dequant-on-use: each matmul expands this to a transient f32 scratch and rides the SAME cuBLASLt f32 GEMV the Float arm uses (bit-identical to a load-time bf16->f32 dequant, just deferred). Never a norm (norms stay Float f32); never on a fast/GEMM/MMQ path (uses_q8_1_fast/gemm_supports = false).

Fields

data: CudaSlice<u8>

ne: Vec<u64>

Implementations

impl GpuTensor

pub fn ne(&self) -> &[u64]

pub fn in_features(&self) -> usize

pub fn out_features(&self) -> usize

pub fn scale(&self) -> f32

Per-tensor post-matmul macro-scale (NVFP4 carries scale != 1.0; all others -> 1.0, a no-op). Used by the fused SwiGLU epilogue to fold the gate/up scale into one kernel.

pub fn load( e: &Engine, g: &GgufFile, name: &str, ) -> Result<Self, Box<dyn Error>>

Load a tensor, keeping quant types packed and float types as f32. (GGUF entry point — thin wrapper over the source-agnostic load_from_source; behavior is unchanged.)

pub fn load_from_source( e: &Engine, src: &dyn TensorSource, name: &str, ) -> Result<Self, Box<dyn Error>>

Source-agnostic load: works from any TensorSource (GGUF or safetensors). The engine’s forward graph only ever asks for ggml-style names; the source maps them to its own layout.

RESIDENCY CENSUS (lane/fp8-decode-v1, 2026-08-05): the wrapper tallies what each 2D matmul weight ACTUALLY became — resident qtype + resident bytes — so the FP8-ST decode arm’s claim (“e4m3 stays native instead of paying the Q8_0-slab tax”) is a measured per-checkpoint fact rather than an assumption about the checkpoint’s dtype mix. Read it with residency_census_report(); zero cost when never read.

pub fn from_quant_bytes( e: &Engine, bytes: &[u8], ty: GgmlType, ne0: u64, ne1: u64, scale: f32, ) -> Result<Self, Box<dyn Error>>

Build a Quant tensor directly from raw ggml block bytes (FR-Spec self-trim: byte-level row gather from an already-loaded weight — rows in every ggml quant are independent, so a contiguous per-row byte copy is a lossless “trim”). ne0 = in_features, ne1 = rows.

pub fn load_opt( e: &Engine, g: &GgufFile, name: &str, ) -> Result<Option<Self>, Box<dyn Error>>

pub fn load_opt_from_source( e: &Engine, src: &dyn TensorSource, name: &str, ) -> Result<Option<Self>, Box<dyn Error>>

pub fn float_data(&self) -> &CudaSlice<f32>

Accessor for tensors that MUST be f32 (norm weights). Panics if quantized.