Trait BackendKvDtype

Source

pub trait BackendKvDtype<K: KvDtypeKind>: BackendPagedKv {
    type KvBuffer: Send + Sync;
    type KvScales: Send + Sync + Default;
}

Expand description

Capability-trait for backends that can store + read a KV cache of type K.

The two associated types carry the K-specific storage shape:

KvBuffer: per-layer K/V element storage. For K = KvFp16 it is the backend’s normal Self::Buffer (FP16). For K = KvInt8 it is the backend’s INT8 buffer (e.g. CudaSlice<i8> on CUDA).
KvScales: per-token-per-kv-head scales. For K = KvFp16 this is the unit type () (no scales). For K = KvInt8 / KvFp8 it is a backend-specific FP16 buffer.

Models that want INT8 KV use: where B: BackendKvDtype<KvInt8> — the buffers in KvCache<B, KvInt8> are then CudaSlice<i8> and CudaSlice<f16>, distinct from the FP16 path’s Self::Buffer.