Enum KvCacheLevel 

#[non_exhaustive]
pub enum KvCacheLevel {
    Fp16,
    Q8,
    Fp8,
    Q4,
}

KV cache precision tier.

Lower variants are higher precision, larger memory footprint; higher variants are lower precision, smaller memory footprint.

Compactness ordering (ordinal): Fp16 (0) < Q8 (1) < Fp8 (2) < Q4 (3).

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

Fp16

FP16 — full quality, baseline memory.

Q8

INT8 quantized — half the memory of FP16.

Fp8

FP8 quantized — half of FP32, same byte width as INT8 but floating-point distribution preserves more dynamic range for attention activations.

Q4

INT4 quantized — quarter the memory of FP16.

Implementations

impl KvCacheLevel

pub const fn memory_factor(self) -> f32

Memory factor relative to FP16.

Level	Factor
Fp16	1.0
Q8	0.5
Fp8	0.5
Q4	0.25

pub const fn ordinal(self) -> u8

Compactness order: higher = more compact (more aggressive).

Ordering: Fp16=0 < Q8=1 < Fp8=2 < Q4=3. Fp8 sits between Q8 and Q4 because both use 1 byte per value but FP8’s floating-point distribution makes it preferable to INT8 for KV cache activations while still being intermediate before INT4.

pub const fn tag(self) -> &'static str

Human-readable tag.

Trait Implementations

impl Clone for KvCacheLevel

fn clone(&self) -> KvCacheLevel

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for KvCacheLevel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for KvCacheLevel

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for KvCacheLevel

fn eq(&self, other: &KvCacheLevel) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for KvCacheLevel

impl Eq for KvCacheLevel

impl StructuralPartialEq for KvCacheLevel