Struct ResidentCacheConfig 

pub struct ResidentCacheConfig {
    pub max_entries: usize,
    pub max_bytes: u64,
    pub min_tokens: u64,
    pub reserved_seq_count: i32,
    pub max_resident_tokens: u64,
}

Fields

max_entries: usize

max_bytes: u64

min_tokens: u64

reserved_seq_count: i32

max_resident_tokens: u64

Maximum number of native KV cell positions the cache may hold at one time, in tokens. Under kv_unified = true (skippy patch 0034) the resident prefix cache shares one n_ctx cell pool with the active execution lanes. Without this cap the cache budget is bounded only by max_entries and max_bytes, both of which can easily allow more pinned tokens than the cell pool has cells — the lanes then can’t find a free slot and the embedded runtime surfaces HTTP 502 RuntimeError: llama_decode failed (decode: failed to find a memory slot).

Set this to a fraction of the model’s n_ctx (typically n_ctx / 2 or similar). A value of 0 disables the cap and behaves like the legacy unbounded-by-tokens cache. The cap is only useful when n_ctx is comfortably larger than min_tokens; see [derive_max_resident_tokens] for the floor.

Implementations

impl ResidentCacheConfig

pub fn from_stage(config: &StageConfig, cache: &StageKvCacheConfig) -> Self

Trait Implementations

impl Clone for ResidentCacheConfig

fn clone(&self) -> ResidentCacheConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for ResidentCacheConfig

impl Debug for ResidentCacheConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for ResidentCacheConfig

impl PartialEq for ResidentCacheConfig

fn eq(&self, other: &ResidentCacheConfig) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for ResidentCacheConfig