Enum Quantization 

pub enum Quantization {
    Q4kM,
    Q2k,
    Q3kS,
    Q3kM,
    Q3kL,
    Q5kM,
    Q6k,
    Q8_0,
    F8,
    F16,
    BF16,
    IQ4_XS,
    Other(String),
}

Quantization level for a model weight file.

Suffixes follow common GGUF naming:

• K marks the newer k-quants family
• L means large, higher-quality size within that family
• M means medium quality within that family
• S means small, lower-quality size within that family

Q4_K_M is a GGUF quantization format for LLMs used in llama.cpp. It is often considered the sweet spot between model size and quality:

• Q4 means 4-bit precision per weight
• K means k-quant, a group-wise quantization scheme
• M means medium, where sensitive tensors are selectively raised to 5-6 bits

IQ is a separate I-quant family that uses an importance matrix for reconstruction.

Variants

Q4kM

Good default balance of quality and size (Q4_K_M), often the sweet spot for GGUF LLMs

Q2k

Lower quality, smallest file size (Q2_K)

Q3kS

Smaller, lower quality (Q3_K_S)

Q3kM

Smaller, lower quality (Q3_K_M)

Q3kL

Smaller, lower quality, larger than Q3_K_M (Q3_K_L)

Q5kM

Better quality, larger file (Q5_K_M)

Q6k

Higher quality, much larger file (Q6_K)

Q8_0

Near full quality, very large file (Q8_0)

F8

8-bit floating-point weights

F16

Full-ish precision, huge file (F16)

BF16

Full-ish precision, huge file (BF16)

IQ4_XS

Importance-aware 4-bit quantization with extra-small size (IQ4_XS)

Other(String)

Catch-all for quantization tags added by upstream catalogs

Implementations

impl Quantization

pub fn from_gguf_filename(filename: &str) -> Option<Self>

Detect a quantization tag in a GGUF filename.

Trait Implementations

impl Clone for Quantization

fn clone(&self) -> Quantization

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Quantization

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Quantization

fn default() -> Quantization

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Quantization

fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>

where D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Display for Quantization

fn fmt(&self, __derive_more_f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<&str> for Quantization

fn from(value: &str) -> Self

Converts to this type from the input type.

impl From<String> for Quantization

fn from(value: String) -> Self

Converts to this type from the input type.

impl FromStr for Quantization

type Err = Infallible

The associated error which can be returned from parsing.

fn from_str(value: &str) -> Result<Self, Self::Err>

Parses a string s to return a value of this type.

impl IntoEnumIterator for Quantization

type Iterator = QuantizationIter

fn iter() -> QuantizationIter

impl JsonSchema for Quantization

fn schema_name() -> Cow<'static, str>

The name of the generated JSON Schema.

fn schema_id() -> Cow<'static, str>

Returns a string that uniquely identifies the schema produced by this type.

fn json_schema(generator: &mut SchemaGenerator) -> Schema

Generates a JSON Schema for this type.

fn inline_schema() -> bool

Whether JSON Schemas generated for this type should be included directly in parent schemas, rather than being re-used where possible using the $ref keyword.

impl PartialEq for Quantization

fn eq(&self, other: &Quantization) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for Quantization

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for Quantization