Enum KvPrecision 

pub enum KvPrecision {
    Bf16,
    Fp8,
    Int4,
}

The element type the cache is stored in.

Int4 is half a byte per element, which is why element size is a f64 rather than an integer.

Variants

Bf16

bf16 or fp16: two bytes per element. The default for most runtimes.

Fp8

fp8 KV cache: one byte per element.

Int4

int4 KV cache: half a byte per element.

Implementations

impl KvPrecision

pub fn bytes_per_element(self) -> f64

Bytes occupied by one cached element.

Trait Implementations

impl Clone for KvPrecision

fn clone(&self) -> KvPrecision

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for KvPrecision

impl Debug for KvPrecision

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for KvPrecision

impl PartialEq for KvPrecision

fn eq(&self, other: &KvPrecision) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for KvPrecision