Struct PrefixCachedEngine 

pub struct PrefixCachedEngine<'a> {
    pub inner: InferenceEngine<'a>,
    pub prefix_cache: PrefixAwarePrefill,
}

An InferenceEngine augmented with prefix KV-cache reuse.

On each generate call the engine:

1. Resets the model’s KV cache (single-engine, sequential request model).
2. Looks up the longest cached prefix in the trie.
3. Injects the matched KV blocks back into the model’s CPU cache.
4. Runs prefill only on the uncached suffix at the correct pos_start.
5. Extracts any newly produced full blocks of KV state and stores them in the trie for subsequent requests (skipped on GPU tiers where the CPU cache stays empty).
6. Sample-decodes new tokens up to params.max_tokens or EOS.
7. Releases the session (decrements ref counts) when done.

Fields

inner: InferenceEngine<'a>

The underlying inference engine.

prefix_cache: PrefixAwarePrefill

Prefix-cache-aware prefill helper with the block trie.

Implementations

impl<'a> PrefixCachedEngine<'a>

pub fn new(engine: InferenceEngine<'a>, max_cache_blocks: usize) -> Self

Wrap an existing InferenceEngine with a prefix cache.

Derives num_layers, num_kv_heads, and head_dim directly from the engine’s model configuration, so no manual wiring is required.

Parameters

• engine — the inference engine to wrap.
• max_cache_blocks — maximum number of simultaneously live cache blocks. Each block holds BLOCK_SIZE (16) tokens of KV data for every layer; memory per block is approximately 2 × num_layers × num_kv_heads × head_dim × 16 × 4 bytes.

pub fn generate( &mut self, prompt_tokens: &[u32], params: &SamplingParams, ) -> Vec<u32>

Generate tokens from prompt_tokens, reusing any cached prefix.

Returns the generated token IDs (not including the prompt). On any internal error the method logs via tracing::warn and returns an empty vector — generate itself is infallible from the caller’s perspective so it can be dropped into batch pipelines.

pub fn cache_stats(&self) -> PrefixCacheStats

Return a snapshot of the current prefix-cache statistics.

pub fn clear_cache(&mut self)

Clear all entries from the prefix cache.

Does not reset the inner engine’s KV cache.