Enum Quantization 

pub enum Quantization {
    Q4_K_S,
    Q4_K_M,
    Q5_K_S,
    Q5_K_M,
    Q6_K,
    Q8_0,
    F16,
    F32,
}

Quantization levels for GGUF models.

Quantization reduces model size and memory usage at the cost of some quality. Lower quantization (Q4) = smaller, faster, less accurate. Higher quantization (F16) = larger, slower, more accurate.

Variants

Q4_K_S

4-bit quantization, small variant (smallest, fastest).

Q4_K_M

4-bit quantization, medium variant (recommended for most use cases).

Q5_K_S

5-bit quantization, small variant.

Q5_K_M

5-bit quantization, medium variant (balanced quality/size).

Q6_K

6-bit quantization.

Q8_0

8-bit quantization (high quality).

F16

16-bit floating point (full precision).

F32

32-bit floating point (maximum precision, rarely used).

Implementations

impl Quantization

pub const fn name(&self) -> &'static str

Human-readable name for this quantization level.

pub const fn short_name(&self) -> &'static str

Short name without description.

pub const fn memory_multiplier(&self) -> f32

Approximate memory multiplier (bytes per parameter).

Use this to estimate model memory requirements: memory_gb = param_billions * multiplier

pub const fn all() -> &'static [Quantization]

Returns all quantization levels in order from smallest to largest.

Trait Implementations

impl Clone for Quantization

fn clone(&self) -> Quantization

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Quantization

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for Quantization

fn default() -> Quantization

Default to Q4_K_M as it provides the best balance of size and quality.

impl Display for Quantization

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Hash for Quantization

fn hash<__H>(&self, state: &mut __H)

where __H: Hasher,

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for Quantization

fn eq(&self, other: &Quantization) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for Quantization

impl Eq for Quantization

impl StructuralPartialEq for Quantization