Enum IsqBits 

pub enum IsqBits {
    Two,
    Three,
    Four,
    Five,
    Six,
    Eight,
}

Target bit width for automatic ISQ quantization.

On Metal, these select AFQ variants; on CUDA/CPU, they select Q*K variants.

Variants

Two

2-bit quantization (AFQ2 on Metal, Q2K otherwise).

Three

3-bit quantization (AFQ3 on Metal, Q3K otherwise).

Four

4-bit quantization (AFQ4 on Metal, Q4K otherwise).

Five

5-bit quantization (Q5K on all platforms).

Six

6-bit quantization (AFQ6 on Metal, Q6K otherwise).

Eight

8-bit quantization (AFQ8 on Metal, Q8_0 otherwise).

Implementations

impl IsqBits

pub fn resolve(self, device: &Device) -> IsqType

Resolve to the platform-appropriate IsqType for the given device.

pub fn expand(self) -> Vec<IsqType>

Return all platform variants, with the current platform’s preferred variant first. On Metal, AFQ variants come first; on other platforms, GGUF/Q variants come first.

Trait Implementations

impl Clone for IsqBits

fn clone(&self) -> IsqBits

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for IsqBits

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for IsqBits

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for IsqBits

fn eq(&self, other: &IsqBits) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl TryFrom<&str> for IsqBits

type Error = ()

The type returned in the event of a conversion error.

fn try_from(s: &str) -> Result<Self, ()>

Performs the conversion.

impl Copy for IsqBits

impl Eq for IsqBits

impl StructuralPartialEq for IsqBits