Struct Quantized 

pub struct Quantized {
    pub weight: Array,
    pub scales: Array,
    pub biases: Option<Array>,
    pub quant: Quant,
}

Quantized weights: packed codes (uint32), one scale per group (affine: the weights’ dtype; floating-point modes: uint8 codes), affine biases, and how they were quantized.

use nuvec_core::{DType, QuantMode, ops::ones};

let w = ones([2, 64], DType::F32);
let q = w.quantize(QuantMode::Mxfp4);
assert_eq!((q.weight.shape(), q.scales.shape(), q.biases.is_none()), (&[2, 8][..], &[2, 2][..], true));
assert_eq!(q.dequantize(DType::F32).to_vec::<f32>().unwrap(), vec![1.0; 128]);

Fields

weight: Array

The packed codes (uint32, last dimension values · bits / 32).

scales: Array

One scale per group (last dimension values / group_size).

biases: Option<Array>

One bias per group: required for affine, absent for the floating-point modes.

quant: Quant

How the weights were quantized.

Implementations

impl Quantized

pub fn affine( weight: Array, scales: Array, biases: Array, group_size: i32, bits: i32, ) -> Self

Affine weights quantized elsewhere: the biases are required, as MLX requires them (a checkpoint without biases quantized with zero offsets: pass ops::zeros_like(&scales)).

pub fn fp(weight: Array, scales: Array, mode: QuantMode) -> Self

Floating-point-mode weights quantized elsewhere (MXFP4, MXFP8, NVFP4), which have no biases.

pub fn new( weight: Array, scales: Array, biases: Option<Array>, quant: impl Into<Quant>, ) -> Self

Weights quantized elsewhere (a checkpoint’s tensors). biases must be Some for QuantMode::Affine and None for the floating-point modes, as in MLX; otherwise every operation on them fails with an error saying so. Quantized::affine and Quantized::fp make the rule part of the signature.

pub fn dequantize(&self, dtype: impl Into<Option<DType>>) -> Array

The weights restored (MLX’s dequantize), in dtype (default: the scales’ dtype for affine, bfloat16 for the floating-point modes).

pub fn check(&self) -> Result<()>

The first error any of the arrays carries.

Trait Implementations

impl Clone for Quantized

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Quantized

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Leaves for Quantized

A tree of buffers (never trainable, as quantized layers’ tensors) with MLX’s names weight, scales and biases, and quant as a static field: eval(&q), the transforms and save_safetensors take quantized weights directly.

fn visit(&self, v: &mut Visit<'_>)

Reports every leaf (and child tree, enum variant, mode flag and static field) to v, in order.

impl Tree for Quantized

fn visit_mut(&mut self, v: &mut VisitMut<'_>)

Reports every leaf (and child tree and mode flag) to v mutably, in the order of Leaves::visit.

fn for_each(&self, f: impl FnMut(&str, &Array))

Calls f(path, leaf) for every leaf, in order.

fn map(&self, f: impl FnMut(&str, &Array) -> Array) -> Self

The tree with every leaf replaced by f(path, leaf) (same structure; parameters keep their frozen flag).

fn leaves(&self) -> Vec<Array>

The leaves, in order.

fn paths(&self) -> Vec<String>

The leaves’ paths, in order.

fn flatten(&self) -> Vec<(String, Array)>

The leaves with their paths, in order (MLX’s tree_flatten).

fn unflatten(&self, leaves: impl IntoIterator<Item = Array>) -> Result<Self>

A tree of this structure (self is the template) with leaves in order: exactly Tree::len of them, else an error naming both counts.

fn load_flat( &mut self, named: impl IntoIterator<Item = (String, Array)>, mode: LoadMode, ) -> Result<()>

Replaces leaves by path. LoadMode::Strict requires exactly the tree’s paths with matching shapes; LoadMode::Partial assigns the matching names and ignores unknown ones (shapes must still match). Loaded dtypes are kept. On any problem the error lists every one (missing, unknown and mismatched names, sorted) and the tree is unchanged. Duplicate paths (in the tree or in named) are an error.

fn len(&self) -> usize

The number of leaves.

fn is_empty(&self) -> bool

Whether the tree has no leaves.

fn def(&self) -> TreeDef

The tree’s structure.