Struct CacheConfig 

pub struct CacheConfig {
    pub max_seq_len: usize,
    pub page_size: usize,
    pub kind: CacheKind,
    pub quantize_kv: bool,
}

Configuration for a KV cache instance.

Fields

max_seq_len: usize

Maximum number of cached positions (arena capacity).

page_size: usize

Tokens per page (paged cache only).

kind: CacheKind

Implementation to use.

quantize_kv: bool

Store global-layer KV pages int8-quantized (group-32 along head_dim, packed 4 bytes per int element) — ~3.5× less KV memory on the f32 build at near-lossless int8 fidelity. Sliding-window layers stay in float (they hold at most w-1 tokens). Set from COMBS_KV_QUANT=1 by the engine.

Implementations

impl CacheConfig

pub const DEFAULT_PAGE_SIZE: usize = 16

Default page size (MLC uses 16 as well).

pub fn paged(max_seq_len: usize) -> Self

Paged cache with the default page size.

pub fn contiguous(max_seq_len: usize) -> Self

Contiguous (baseline) cache.

pub fn num_pages(&self) -> usize

Number of pages in the arena.

Trait Implementations

impl Clone for CacheConfig

fn clone(&self) -> CacheConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for CacheConfig

impl Debug for CacheConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.