Struct PrefixAwarePrefill 

pub struct PrefixAwarePrefill {
    pub cache: PrefixCache,
}

Wraps a PrefixCache and exposes a higher-level prefill API.

The typical call pattern for one request is:

let (session, uncached_start) = prefill.prepare(&token_ids);
// run your model prefill on token_ids[uncached_start..]
prefill.store_blocks(&token_ids, uncached_start, new_kv_blocks);
prefill.release_session(session);

Fields

cache: PrefixCache

The underlying prefix cache.

Implementations

impl PrefixAwarePrefill

pub fn new(cache: PrefixCache) -> Self

Wrap an existing PrefixCache.

pub fn prepare(&mut self, token_ids: &[u32]) -> (CacheSession, usize)

Determine how much of token_ids is already cached.

Returns (session, uncached_start) where uncached_start is the index of the first token that must be processed by the model.

pub fn store_blocks( &mut self, token_ids: &[u32], uncached_start: usize, keys_by_block: Vec<KvBlockPair>, )

After prefill, store the newly computed KV blocks back into the cache.

keys_by_block is a list of (keys, values) for each newly computed block, in order, starting from the block at uncached_start.

pub fn release_session(&mut self, session: CacheSession)

Release all blocks held by a session (decrement their ref counts).

pub fn stats(&self) -> PrefixCacheStats

Snapshot of current cache statistics.