Skip to main content

QuantType

Enum QuantType

pub enum QuantType {
Show 30 variants    F32,
    F16,
    BF16,
    Q8_0,
    Q8_K,
    Q6_K,
    Q5_K_S,
    Q5_K_M,
    Q5_0,
    Q5_1,
    Q4_K_S,
    Q4_K_M,
    Q4_0,
    Q4_1,
    Q3_K_S,
    Q3_K_M,
    Q3_K_L,
    Q2_K_S,
    Q2_K,
    IQ4_NL,
    IQ4_XS,
    IQ3_S,
    IQ3_M,
    IQ3_XS,
    IQ3_XXS,
    IQ2_S,
    IQ2_XS,
    IQ2_XXS,
    IQ1_S,
    IQ1_M,
}

Expand description

Common quantization types (GGUF spec naming convention)

Variants§

F32

Full precision (FP32)

F16

Half precision (FP16)

BF16

Brain floating point (BF16)

Q8_0

8-bit integer

Q8_K

8-bit with K-quants

Q6_K

6-bit K-quants

Q5_K_S

5-bit K-quants (small)

Q5_K_M

5-bit K-quants (medium)

Q5_0

5-bit (legacy)

Q5_1

5-bit with 1 (legacy)

Q4_K_S

4-bit K-quants (small)

Q4_K_M

4-bit K-quants (medium)

Q4_0

4-bit (legacy)

Q4_1

4-bit with 1 (legacy)

Q3_K_S

3-bit K-quants (small)

Q3_K_M

3-bit K-quants (medium)

Q3_K_L

3-bit K-quants (large)

Q2_K_S

2-bit K-quants (small)

Q2_K

2-bit K-quants

IQ4_NL

Importance-weighted 4-bit (non-linear)

IQ4_XS

Importance-weighted 4-bit (extra small)

IQ3_S

Importance-weighted 3-bit (small)

IQ3_M

Importance-weighted 3-bit (medium)

IQ3_XS

Importance-weighted 3-bit (extra small)

IQ3_XXS

Importance-weighted 3-bit (extra extra small)

IQ2_S

Importance-weighted 2-bit (small)

IQ2_XS

Importance-weighted 2-bit (extra small)

IQ2_XXS

Importance-weighted 2-bit (extra extra small)

IQ1_S

Importance-weighted 1-bit (small)

IQ1_M

Importance-weighted 1-bit (medium)

Implementations§

impl QuantType

pub fn from_str(s: &str) -> Option<Self>

Parse quantization type from string

pub const fn bits_per_weight(&self) -> f32

Get bits per weight

pub fn estimate_size(&self, parameters: u64) -> u64

Estimate file size for given parameter count

pub const fn quality_tier(&self) -> u8

Get quality tier (1-5, higher is better quality)

pub fn vram_requirement(&self, parameters: u64) -> f64

Get recommended VRAM in GB for given parameter count

Trait Implementations§

impl Clone for QuantType

fn clone(&self) -> QuantType

Returns a duplicate of the value. Read more

1.0.0 · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more

impl Debug for QuantType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

impl<'de> Deserialize<'de> for QuantType

fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer. Read more

impl Display for QuantType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

impl Hash for QuantType

fn hash<H: Hasher>(&self, state: &mut H)

Feeds this value into the given Hasher. Read more

1.3.0 · Source§

fn hash_slice<H>(data: &[Self], state: &mut H)
where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher. Read more

impl PartialEq for QuantType

fn eq(&self, other: &QuantType) -> bool

Tests for self and other values to be equal, and is used by ==.

1.0.0 · Source§

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for QuantType

fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where S: Serializer,

Serialize this value into the given Serde serializer. Read more

impl Copy for QuantType

impl Eq for QuantType

impl StructuralPartialEq for QuantType

Auto Trait Implementations§

impl Freeze for QuantType

impl RefUnwindSafe for QuantType

impl Send for QuantType

impl Sync for QuantType

impl Unpin for QuantType

impl UnsafeUnpin for QuantType

impl UnwindSafe for QuantType

Blanket Implementations§

impl<T> Any for T
where T: 'static + ?Sized,

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more

impl<T> Borrow<T> for T
where T: ?Sized,

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more

impl<T> BorrowMut<T> for T
where T: ?Sized,

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more

impl<T> CloneToUninit for T
where T: Clone,

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)

Performs copy-assignment from self to dest. Read more

impl<Q, K> Equivalent<K> for Q
where Q: Eq + ?Sized, K: Borrow<Q> + ?Sized,

fn equivalent(&self, key: &K) -> bool

Checks if this value is equivalent to the given key. Read more

impl<Q, K> Equivalent<K> for Q
where Q: Eq + ?Sized, K: Borrow<Q> + ?Sized,

fn equivalent(&self, key: &K) -> bool

Checks if this value is equivalent to the given key. Read more

impl<Q, K> Equivalent<K> for Q
where Q: Eq + ?Sized, K: Borrow<Q> + ?Sized,

fn equivalent(&self, key: &K) -> bool

Compare self to key and return true if they are equal.

impl<T> From<T> for T

fn from(t: T) -> T

Returns the argument unchanged.

impl<T> Instrument for T

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more

impl<T, U> Into<U> for T
where U: From<T>,

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

impl<T> PolicyExt for T
where T: ?Sized,

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns Action::Follow only if self and other return Action::Follow. Read more

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns Action::Follow if either self or other returns Action::Follow. Read more

impl<T> Same for T

type Output = T

Should always be Self

impl<T> ToOwned for T
where T: Clone,

type Owned = T

The resulting type after obtaining ownership.

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more

impl<T> ToString for T
where T: Display + ?Sized,

fn to_string(&self) -> String

Converts the given value to a String. Read more

impl<T> ToStringFallible for T
where T: Display,

fn try_to_string(&self) -> Result<String, TryReserveError>

ToString::to_string, but without panic on OOM.

impl<T, U> TryFrom<U> for T
where U: Into<T>,

type Error = Infallible

The type returned in the event of a conversion error.

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

fn vzip(self) -> V

impl<T> WithSubscriber for T

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more

impl<T> DeserializeOwned for T
where T: for<'de> Deserialize<'de>,

impl<A, B, T> HttpServerConnExec<A, B> for T
where B: Body,