Enum KvCacheType 

#[non_exhaustive]
pub enum KvCacheType {
    F32,
    F16,
    BF16,
    F64,
    Q4_0,
    Q4_1,
    Q5_0,
    Q5_1,
    Q8_0,
    Q8_1,
    Q2_K,
    Q3_K,
    Q4_K,
    Q5_K,
    Q6_K,
    Q8_K,
    IQ2_XXS,
    IQ2_XS,
    IQ2_S,
    IQ3_XXS,
    IQ3_S,
    IQ1_S,
    IQ1_M,
    IQ4_XS,
    IQ4_NL,
    I8,
    I16,
    I32,
    I64,
    TQ1_0,
    TQ2_0,
    MXFP4,
}

Data type used for an entry in the attention KV cache.

Mirrors the subset of ggml_type values that llama.cpp accepts as KV cache element types. The F16 default preserves full attention quality; quantizing (e.g. Q8_0 ≈ ½ size, Q4_0 ≈ ¼ size) trades a small amount of accuracy for a large VRAM reduction at long n_ctx.

This is a local shim around llama_cpp_2::context::params::KvCacheType so a future llama-cpp-2 update doesn’t force a breaking release of rig-llama-cpp. Marked #[non_exhaustive]: when llama.cpp adds a new ggml_type, we add a corresponding variant in a minor (0.1.x) release.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

F32

IEEE 754 single precision.

F16

IEEE 754 half precision (llama.cpp’s default for both K and V).

BF16

Brain floating-point 16, common on newer NVIDIA / AMD GPUs.

F64

IEEE 754 double precision.

Q4_0

4-bit block quantization, type 0.

Q4_1

4-bit block quantization, type 1.

Q5_0

5-bit block quantization, type 0.

Q5_1

5-bit block quantization, type 1.

Q8_0

8-bit block quantization, type 0.

Q8_1

8-bit block quantization, type 1.

Q2_K

2-bit K-quant.

Q3_K

3-bit K-quant.

Q4_K

4-bit K-quant.

Q5_K

5-bit K-quant.

Q6_K

6-bit K-quant.

Q8_K

8-bit K-quant.

IQ2_XXS

Importance-weighted 2-bit, extra-extra-small.

IQ2_XS

Importance-weighted 2-bit, extra-small.

IQ2_S

Importance-weighted 2-bit, small.

IQ3_XXS

Importance-weighted 3-bit, extra-extra-small.

IQ3_S

Importance-weighted 3-bit, small.

IQ1_S

Importance-weighted 1-bit, small.

IQ1_M

Importance-weighted 1-bit, medium.

IQ4_XS

Importance-weighted 4-bit, extra-small.

IQ4_NL

Importance-weighted 4-bit, non-linear.

I8

Signed 8-bit integer.

I16

Signed 16-bit integer.

I32

Signed 32-bit integer.

I64

Signed 64-bit integer.

TQ1_0

Ternary 1-bit, type 0.

TQ2_0

Ternary 2-bit, type 0.

MXFP4

Microscaling FP4.

Trait Implementations

impl Clone for KvCacheType

fn clone(&self) -> KvCacheType

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for KvCacheType

impl Debug for KvCacheType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for KvCacheType

impl From<KvCacheType> for KvCacheType

fn from(value: KvCacheType) -> Self

Converts to this type from the input type.

impl Hash for KvCacheType

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for KvCacheType

fn eq(&self, other: &KvCacheType) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for KvCacheType