Struct PrefixKvCache 

pub struct PrefixKvCache { /* private fields */ }

A radix-tree based prefix KV cache.

Stores KV cache states indexed by token prefix sequences. When a new prompt shares a prefix with a previously-cached sequence, the matching KV state is reused and only the remaining tokens need prefill.

Implementations

impl PrefixKvCache

pub fn new(config: PrefixCacheConfig) -> Self

Create a new prefix KV cache with the given configuration.

pub fn lookup(&mut self, tokens: &[u32]) -> Option<(usize, &CachedKvState)>

Look up the longest matching prefix for the given tokens.

Returns (matching_prefix_length, cached_kv_state_ref). Returns None if no prefix matches or the match is shorter than min_prefix_len.

pub fn store( &mut self, tokens: &[u32], kv_cache: &dyn KvCacheAccess, seq_len: usize, kv_dim: usize, num_layers: usize, )

Store KV cache state for a token prefix.

Extracts the relevant KV data from the live cache via the KvCacheAccess trait. If the prefix is shorter than min_prefix_len, the store is silently skipped.

pub fn store_snapshot(&mut self, tokens: &[u32], snapshot: CachedKvState)

Store a pre-built CachedKvState directly for a token prefix.

This is useful when the caller has already constructed the snapshot.

pub fn restore(cached: &CachedKvState, target: &mut KvCache)

Restore a cached prefix into a live KV cache.

Copies the cached KV data into the target cache’s buffers and resets the target’s sequence position to match the snapshot.

pub fn len(&self) -> usize

Current number of cached prefixes (nodes with KV data).

pub fn is_empty(&self) -> bool

Whether the cache is empty (no cached KV data).

pub fn clear(&mut self)

Clear all cached entries.

pub fn memory_usage(&self) -> usize

Current estimated memory usage in bytes.

pub fn hits(&self) -> u64

Number of cache hits since creation.

pub fn misses(&self) -> u64

Number of cache misses since creation.