Module prefix_cache 

KV-prefix caching: when a new request’s tokens share a leading subsequence with a previously processed request, skip recomputing the KV state for that shared prefix entirely, restoring it from a stored snapshot instead of running forward_batch over tokens that were already processed.

This is the harder sibling of ferrox-server::cache::ResponseCache (which only helps exact-repeat requests): prefix caching helps any request that starts with something seen before, which is the common case for multi-turn chat (each turn’s full prompt is the previous turn’s prompt plus a little more) even when no single request repeats exactly.

Deliberately scoped: this does a linear scan over a small, LRU-bounded set of stored prefixes to find the longest common prefix, not a trie/radix-tree structure (vLLM’s and SGLang’s RadixAttention do this properly at production scale). For the small number of concurrent conversations a demo server actually handles, a linear scan is simpler and correctness is easier to verify.

“LRU-bounded” is now true. It was not: eviction dropped the oldest ARRIVAL while nothing on the hit path recorded that an entry had been used, so the policy was first-in-first-out under an LRU name. That inverted the cache’s purpose, because the entry a prefix cache exists for – the system prompt every request shares – is the oldest one precisely because it is the most reused.

What is still true of the scope: each entry CLONES its Vec<KvCache>, so N conversations off one system prompt hold N copies of its KV rather than sharing the pages. Fixing that is the radix cache’s job (crate::policy::radix), which shares nodes and reference-counts pages, and which needs a serving path that reads paged KV before it can be wired in.

Structs

PrefixCache

LRU-bounded store of StoredPrefix snapshots, searched for the longest common prefix with an incoming token sequence.

PrefixCacheStats

PrefixMatch

What was found (or not) for an incoming token sequence.