Struct PrefixCache 

pub struct PrefixCache { /* private fields */ }

LRU-bounded store of StoredPrefix snapshots, searched for the longest common prefix with an incoming token sequence.

Implementations

impl PrefixCache

pub fn new(max_entries: usize) -> Self

pub fn find_longest_prefix(&mut self, tokens: &[usize]) -> PrefixMatch

Finds the stored prefix with the longest common leading subsequence with tokens, and returns a ready-to-use clone of its KV state truncated to exactly that common length (a stored prefix may itself be longer than the common part, if a later, different continuation was stored under it – the KV cache is truncated to the matching length before being handed back, so the caller never sees state from a divergent continuation).

pub fn store( &mut self, tokens: Vec<usize>, kv_caches: Vec<KvCache>, pending_logits: Vec<f32>, )

Stores a snapshot for tokens (all tokens processed so far, prompt plus any generated continuation) with the given KV cache state and next-token logits, evicting the least-recently-stored entry if already at capacity.

pub fn stats(&self) -> PrefixCacheStats