Struct QuantizationConfig 

pub struct QuantizationConfig {
    pub bits: u8,
    pub group_size: usize,
    pub per_tensor: HashMap<String, TensorQuantConfig>,
}

Per-tensor quantization configuration from quantization_config.json.

This mirrors the JSON structure produced by hf2q’s --quant auto mode, where each tensor may have a different bit-width and group size.

Fields

bits: u8

Default bit-width applied when a tensor has no per-tensor override.

group_size: usize

Default group size applied when a tensor has no per-tensor override.

per_tensor: HashMap<String, TensorQuantConfig>

Per-tensor overrides keyed by tensor name pattern. Each entry maps a tensor name (or glob pattern) to its quant config.

Implementations

impl QuantizationConfig

pub fn from_file(path: &Path) -> Result<Self>

Load and parse a quantization_config.json file from disk.

Errors

Returns MlxError::IoError if the file cannot be read, or MlxError::QuantConfigError if the JSON is malformed.

pub fn from_json(json: &str) -> Result<Self>

Parse a QuantizationConfig from a JSON string.

Errors

Returns MlxError::QuantConfigError if the JSON is malformed.

pub fn from_model_config_json(json: &str) -> Result<Self>

Parse per-tensor quantization overrides from the "quantization" section of an MLX model’s config.json.

In this format, the quantization section contains flat keys for tensor names alongside the default bits and group_size:

{
  "quantization": {
    "bits": 4,
    "group_size": 64,
    "model.layers.0.mlp.down_proj": {"bits": 8, "group_size": 64}
  }
}

This parses the entire "quantization" object, extracting bits and group_size as defaults, and any nested objects as per-tensor overrides.

pub fn from_model_config_file(path: &Path) -> Result<Self>

Parse per-tensor overrides from a config.json file on disk.

pub fn config_for_tensor(&self, tensor_name: &str) -> (u8, usize)

Look up the quantization parameters for a specific tensor name.

Matching strategy (in order):

1. Exact match in per_tensor.
2. Strip .weight / .scales / .biases suffix, then exact match.
3. Strip language_model. prefix (with or without suffix), then match.
4. Add language_model. prefix (with or without suffix), then match.

If no override matches, returns the default bits and group_size.

Trait Implementations

impl Clone for QuantizationConfig

fn clone(&self) -> QuantizationConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for QuantizationConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for QuantizationConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.