Struct PrefixCache 

pub struct PrefixCache { /* private fields */ }

LRU-bounded store of StoredPrefix snapshots, searched for the longest common prefix with an incoming token sequence.

Implementations

impl PrefixCache

pub fn new(max_entries: usize) -> Self

pub fn clear(&mut self)

Drops every stored prefix, keeping the capacity and the lifetime hit/miss counters.

For a KV-side cache rebuild. A stored prefix names positions in an allocation that is about to stop existing, so handing one back afterwards would restore another request’s state into this one – silently, since a KV cache carries no identity of its own. The counters survive because they describe what this process has served, which a re-split does not undo.

pub fn find_longest_prefix(&mut self, tokens: &[usize]) -> PrefixMatch

Finds the stored prefix with the longest common leading subsequence with tokens, and returns a ready-to-use clone of its KV state truncated to exactly that common length (a stored prefix may itself be longer than the common part, if a later, different continuation was stored under it – the KV cache is truncated to the matching length before being handed back, so the caller never sees state from a divergent continuation).

pub fn store( &mut self, tokens: Vec<usize>, kv_caches: Vec<KvCache>, pending_logits: Vec<f32>, )

Stores a snapshot for tokens (all tokens processed so far, prompt plus any generated continuation) with the given KV cache state and next-token logits, evicting the least recently USED entry if already at capacity.

Used, not stored. This used to drop entries[0] – the oldest arrival – while the type documented itself as LRU-bounded. The difference is the whole value of the cache: a system prompt that every request shares is the oldest entry precisely BECAUSE it is the most reused, so FIFO evicted the one entry worth keeping as soon as max_entries newer prompts arrived, and the next request off that system prompt recomputed all of it.

pub fn stats(&self) -> PrefixCacheStats