Skip to main content

Crate spec_stream_cache

Crate spec_stream_cache 

Source
Expand description

§spec-stream-cache

Paged KV-cache memory manager for Large Language Models. Implements virtual-to-physical block tables, prefix caching (prompt sharing), and O(1) rollbacks for speculative decoding branch pruning.

Re-exports§

pub use allocator::BlockAllocator;
pub use block::PhysicalBlock;
pub use block::PhysicalBlockId;
pub use block::DEFAULT_BLOCK_SIZE;
pub use error::CacheError;
pub use eviction::EvictionPolicy;
pub use eviction::LruEvictionPolicy;
pub use manager::PagedKvCacheManager;
pub use snapshot::PrefixCacheSnapshot;
pub use snapshot::PrefixSnapshotEntry;
pub use table::BlockTable;

Modules§

allocator
block
error
eviction
Cache Eviction Policies & Memory Pressure Management
manager
snapshot
table