Skip to main content

parse_bnb_quant_state

fuel_tensor::nf4

Function parse_bnb_quant_state 

Source 
pub fn parse_bnb_quant_state(json_bytes: &[u8]) -> Result<BnbQuantState>
Expand description

Parse the JSON quant_state blob that bitsandbytes 0.43+ stores alongside an NF4-quantized weight. Format documented at https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/bitsandbytes/functional.py (search for quant_state serialization).

Validated fields:

  • quant_type == "nf4" — refuses other formats (fp4, etc.) with a clear error: FP4 needs a different 16-entry codebook than the fused NF4 op carries (tracked as a follow-up).
  • shape is [N, K] two integers.
  • blocksize is a positive usize that divides K.

Double quantization: when nested_blocksize is present the returned state carries a BnbNestedState (nested_blocksize, nested_offset; nested_dtype must be "float32"). Supplying only some of the three is an error.

Rejected with a clear error:

  • nested_absmax / nested_quant_map inside the JSON. bitsandbytes stores those as tensors, so seeing them here means the blob is not in the format this parser understands.

Ignored fields: dtype (original pre-quantization dtype — the loader uses the activations’ dtype at matmul time, not the original weight dtype), device, and any future bnb additions.