Struct CudaNf4TransformerBlock 

pub struct CudaNf4TransformerBlock { /* private fields */ }

CUDA-accelerated transformer block with NF4-quantized frozen weights.

Stores the 7 projection weights as packed NF4 (4-bit) + per-block scales instead of fp32, achieving ~8x compression. Norm weights remain fp32 (negligible size).

VRAM Savings (Qwen3-4B example)

Component	fp32	NF4
Frozen weights (36L × 7 projections)	16.0 GB	2.1 GB

Forward Only

NF4 blocks are frozen — no backward pass needed. LoRA adapters (fp32) handle the trainable parameters separately. The forward pass uses fused dequant+GEMM kernels that read NF4 directly without materializing fp32 weights.

Implementations

impl CudaNf4TransformerBlock

pub fn new( config: &TransformerConfig, layer_idx: usize, ctx: Arc<CudaContext>, input_norm_weight: &[f32], post_attn_norm_weight: &[f32], w_q: &[f32], w_k: &[f32], w_v: &[f32], w_o: &[f32], w_gate: &[f32], w_up: &[f32], w_down: &[f32], _max_seq_len: usize, q_lora: Option<(&[f32], &[f32])>, v_lora: Option<(&[f32], &[f32])>, lora_scale: f32, lora_rank: usize, q_norm: Option<&[f32]>, k_norm: Option<&[f32]>, ) -> Result<Self>

Create a new NF4 transformer block from fp32 CPU tensors.

Quantizes all 7 projection weights to NF4 on CPU, then uploads the packed data and scales to GPU. Norm weights are uploaded as fp32.

pub fn set_fp16_weights(&mut self, stream: &CudaStream) -> Result<()>

Cast fp32→fp16 weights + drop fp32 (PMAT-470/472). Frees ~2.6 GB VRAM.

pub fn layer_idx(&self) -> usize

Layer index accessor.

impl CudaNf4TransformerBlock

pub fn download_lora_weights( &self, ) -> Result<(Vec<f32>, Vec<f32>, Vec<f32>, Vec<f32>)>

Download LoRA weights from GPU to CPU for checkpoint saving.

Returns (A_q, B_q, A_v, B_v) as flat f32 vectors. B matrices are returned WITH the baked-in scale (caller can divide by lora_scale if they need the unscaled version).

pub fn upload_lora_weights( &mut self, a_q: &[f32], b_q: &[f32], a_v: &[f32], b_v: &[f32], ) -> Result<()>

Upload LoRA weights from CPU to GPU for checkpoint resume (ENT-276).

Overwrites the current LoRA adapter buffers with trained weights restored from a checkpoint. Call after new() to replace the fresh random init with previously trained adapters.