Struct PagedKvCache 

pub struct PagedKvCache { /* private fields */ }

Paged KV cache for efficient memory management.

Based on vLLM’s PagedAttention algorithm. Manages KV cache memory using fixed-size blocks to prevent fragmentation and enable efficient memory sharing.

Implementations

impl PagedKvCache

pub fn new( num_blocks: usize, block_size: usize, num_heads: usize, head_dim: usize, ) -> Self

Create a new PagedKvCache.

Arguments

• num_blocks: Total number of physical blocks
• block_size: Tokens per block
• num_heads: Number of attention heads
• head_dim: Dimension of each head

pub fn with_eviction_strategy(self, strategy: EvictionStrategy) -> Self

Set eviction strategy.

pub fn with_eviction_threshold(self, threshold: f64) -> Self

Set eviction threshold (0.0-1.0).

pub fn block_size(&self) -> usize

Get block size.

pub fn total_blocks(&self) -> usize

Get total number of blocks.

pub fn free_block_count(&self) -> usize

Get number of free blocks.

pub fn used_block_count(&self) -> usize

Get number of used blocks.

pub fn utilization(&self) -> f64

Memory utilization percentage (0.0-1.0).

pub fn block_memory_bytes(&self) -> usize

Calculate memory for a block in bytes.

pub fn total_memory_bytes(&self) -> usize

Total memory capacity in bytes.

pub fn used_memory_bytes(&self) -> usize

Used memory in bytes.

pub fn needs_eviction(&self) -> bool

Check if eviction is needed.

pub fn num_sequences(&self) -> usize

Get number of active sequences.

pub fn get_sequence(&self, seq_id: SeqId) -> Option<&SequenceInfo>

Get sequence info.

pub fn stats(&self) -> &CacheStats

Get cache statistics.

pub fn eviction_strategy(&self) -> &EvictionStrategy

Get eviction strategy.

pub fn allocate( &mut self, seq_id: SeqId, num_tokens: usize, ) -> PagedKvResult<()>

Allocate blocks for a new sequence.

pub fn append( &mut self, seq_id: SeqId, num_new_tokens: usize, ) -> PagedKvResult<()>

Append tokens to an existing sequence.

pub fn free(&mut self, seq_id: SeqId) -> PagedKvResult<()>

Free all blocks for a sequence.

pub fn fork(&mut self, src_seq: SeqId, dst_seq: SeqId) -> PagedKvResult<()>

Copy-on-write fork for beam search.

Creates a new sequence that shares blocks with the source sequence. Blocks are only copied when modified (copy-on-write).

pub fn select_eviction_target(&self) -> Option<SeqId>

Select sequence to evict based on strategy.

pub fn evict(&mut self) -> PagedKvResult<SeqId>

Evict a sequence to free memory.

pub fn evict_to_threshold( &mut self, target_util: f64, ) -> PagedKvResult<Vec<SeqId>>

Evict until memory utilization is below threshold.

pub fn apply_streaming_llm( &mut self, seq_id: SeqId, sink_tokens: usize, window_tokens: usize, ) -> PagedKvResult<usize>

Apply StreamingLLM eviction to a sequence.

Keeps sink tokens at the beginning and a recent window at the end, evicting middle tokens.

pub fn sequence_ids(&self) -> Vec<SeqId>

Get all sequence IDs.

Trait Implementations

impl Debug for PagedKvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for PagedKvCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.