Struct KVCacheCompressor 

pub struct KVCacheCompressor<B: Backend> {
    pub method: CompressionMethod,
    pub quant_bits: u8,
    /* private fields */
}

KV cache compressor for low-rank and quantized representations.

Fields

method: CompressionMethod

Compression method selection.

quant_bits: u8

Default quantization bits (INT4/INT8).

Implementations

impl<B: Backend> KVCacheCompressor<B>

pub fn new(method: CompressionMethod, quant_bits: u8) -> Self

Create a new KV cache compressor.

pub fn method(&self) -> &CompressionMethod

Access the compression method.

pub fn quant_bits(&self) -> u8

Default quantization bits.

pub fn compress_kv( &self, k: Tensor<B, 4>, v: Tensor<B, 4>, ) -> Result<CompressedKV<B>, &'static str>

Compress batched KV tensors.

Shapes

• k, v: [batch, num_heads, seq_len, head_dim]

pub fn compress_kv_3d( &self, k: Tensor<B, 3>, v: Tensor<B, 3>, ) -> Result<CompressedKV<B>, &'static str>

Compress KV tensors without batch dimension.

Shapes

• k, v: [num_heads, seq_len, head_dim]

pub fn decompress_kv( &self, compressed: CompressedKV<B>, ) -> Result<(Tensor<B, 4>, Tensor<B, 4>), &'static str>

Decompress batched KV tensors.

Shapes

• returns: [batch, num_heads, seq_len, head_dim]

pub fn decompress_kv_3d( &self, compressed: CompressedKV<B>, ) -> Result<(Tensor<B, 3>, Tensor<B, 3>), &'static str>

Decompress KV tensors without batch dimension.

Shapes

• returns: [num_heads, seq_len, head_dim]

pub fn compress_paged_cache( &self, cache: &PagedKVCache<B>, layer: usize, seq_id: usize, ) -> Result<CompressedKV<B>, &'static str>

Compress a sequence from a paged KV cache.

pub fn decompress_to_paged_cache( &self, compressed: CompressedKV<B>, cache: &mut PagedKVCache<B>, layer: usize, seq_id: usize, ) -> Result<(), &'static str>

Decompress into a paged KV cache by appending tokens.

pub fn compress_mla_cache( &self, cache: &MlaCompressedKVCache<B>, layer: usize, seq_id: usize, ) -> Result<CompressedKV<B>, &'static str>

Compress a sequence from an MLA compressed cache.

pub fn decompress_to_mla_cache( &self, compressed: CompressedKV<B>, cache: &mut MlaCompressedKVCache<B>, layer: usize, seq_id: usize, ) -> Result<(), &'static str>

Decompress into an MLA compressed cache by appending tokens.

Trait Implementations

impl<B: Clone + Backend> Clone for KVCacheCompressor<B>

fn clone(&self) -> KVCacheCompressor<B>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<B: Debug + Backend> Debug for KVCacheCompressor<B>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.