Enum QuantFormat 

pub enum QuantFormat {
    F32,
    F16,
    BF16,
    Q4_0,
    Q4_K,
    Q5_K,
    Q6_K,
    Q8_0,
    Gptq {
        bits: u8,
        group_size: u16,
    },
    Awq {
        bits: u8,
    },
}

Supported quantization formats for ComputeBricks.

Based on GGML/llama.cpp quantization types.

Variants

F32

Full precision (baseline)

F16

Half precision

BF16

Brain float 16

Q4_0

4-bit quantization, no scales per block (32 values/block)

Q4_K

4-bit quantization with K-quants (256 values/super-block, 6-bit scales)

Q5_K

5-bit quantization with K-quants

Q6_K

6-bit quantization with K-quants

Q8_0

8-bit quantization, simple (32 values/block)

Gptq

GPTQ format (ExLlama compatible)

Fields

bits: u8

group_size: u16

Awq

AWQ format (activation-aware)

Fields

bits: u8

Implementations

impl QuantFormat

pub fn bits_per_weight(&self) -> f64

Effective bits per weight (including scales and metadata).

pub fn memory_ratio(&self) -> f64

Memory ratio compared to F16 (lower is better).

pub fn expected_ppl_delta(&self) -> f64

Expected perplexity delta compared to F16 (lower is better).

Based on llama.cpp benchmarks.

pub fn block_size(&self) -> usize

Block size (number of weights per quantization block).

pub fn bytes_per_block(&self) -> usize

Bytes per block.

Trait Implementations

impl Clone for QuantFormat

fn clone(&self) -> QuantFormat

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for QuantFormat

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for QuantFormat

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for QuantFormat

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for QuantFormat

fn eq(&self, other: &QuantFormat) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for QuantFormat

impl Eq for QuantFormat

impl StructuralPartialEq for QuantFormat