Expand description
§spec-stream-cache
Paged KV-cache memory manager for Large Language Models. Implements virtual-to-physical block tables, prefix caching (prompt sharing), and O(1) rollbacks for speculative decoding branch pruning.
Re-exports§
pub use allocator::BlockAllocator;pub use block::PhysicalBlock;pub use block::PhysicalBlockId;pub use block::DEFAULT_BLOCK_SIZE;pub use error::CacheError;pub use eviction::EvictionPolicy;pub use eviction::LruEvictionPolicy;pub use manager::PagedKvCacheManager;pub use snapshot::PrefixCacheSnapshot;pub use snapshot::PrefixSnapshotEntry;pub use table::BlockTable;