Struct PrefixCache 

pub struct PrefixCache {
    pub hits: u64,
    pub misses: u64,
    pub evictions: u64,
    /* private fields */
}

Prefix KV-cache with trie-based lookup and LRU eviction.

The trie is keyed by complete blocks of block_size tokens. Each internal node in the trie corresponds to one block boundary; leaf nodes that carry a block_idx have a fully populated CacheBlock.

Fields

hits: u64

Total cache hits since creation.

misses: u64

Total cache misses since creation.

evictions: u64

Total blocks evicted since creation.

Implementations

impl PrefixCache

pub fn new( max_blocks: usize, block_size: usize, num_layers: usize, num_kv_heads: usize, head_dim: usize, ) -> Self

Create a new, empty prefix cache.

pub fn lookup(&mut self, token_ids: &[u32]) -> (usize, Vec<&CacheBlock>)

Look up the longest cached prefix of token_ids.

Walks the trie block-by-block. For every complete block whose tokens match and whose trie node carries a cached block, the block’s last_used stamp is refreshed and the block is returned.

Returns (matched_len, Vec<&CacheBlock>).

pub fn insert( &mut self, token_ids: &[u32], block_start: usize, keys: Vec<Vec<f32>>, values: Vec<Vec<f32>>, ) -> usize

Insert a new block for token_ids[block_start .. block_start + block_size].

Evicts the LRU block if the cache is at capacity. Returns the index of the inserted block in self.blocks.

pub fn release(&mut self, block_idx: usize)

Decrement the reference count of a block, making it eligible for eviction.

pub fn len(&self) -> usize

Number of currently live (occupied) blocks.

pub fn is_empty(&self) -> bool

Returns true if the cache contains no live blocks.

pub fn capacity(&self) -> usize

Maximum number of blocks this cache can hold.

pub fn memory_bytes(&self) -> usize

Total memory consumed by all live blocks’ KV tensors.

pub fn hit_rate(&self) -> f32

Cache hit rate in [0, 1]. Returns 0.0 if no lookups have been made.

pub fn block_size(&self) -> usize

Tokens per block.

pub fn get_block(&self, idx: usize) -> Option<&CacheBlock>

Borrow a cached block by its index in the underlying arena.

Block indices are returned by CacheSession::block_indices when a session is prepared via PrefixAwarePrefill::prepare. None means the index is out of bounds (e.g. the sentinel usize::MAX placed for trie path failures).

pub fn clear(&mut self)

Remove all cached blocks, resetting the trie to an empty root.