Module prefix_cache 

KV-prefix caching: when a new request’s tokens share a leading subsequence with a previously processed request, skip recomputing the KV state for that shared prefix entirely, restoring it from a stored snapshot instead of running forward_batch over tokens that were already processed.

This is the harder sibling of ferrox-server::cache::ResponseCache (which only helps exact-repeat requests): prefix caching helps any request that starts with something seen before, which is the common case for multi-turn chat (each turn’s full prompt is the previous turn’s prompt plus a little more) even when no single request repeats exactly.

Deliberately scoped: this does a linear scan over a small, LRU-bounded set of stored prefixes to find the longest common prefix, not a trie/radix-tree structure (vLLM’s and SGLang’s RadixAttention do this properly at production scale). For the small number of concurrent conversations a demo server actually handles, a linear scan is simpler and correctness is easier to verify.

Structs

PrefixCache

LRU-bounded store of StoredPrefix snapshots, searched for the longest common prefix with an incoming token sequence.

PrefixCacheStats

PrefixMatch

What was found (or not) for an incoming token sequence.