aprender-cbtop 0.69.1

Compute Block Top - Real-time load testing and hardware monitoring TUI
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

//! PagedKvCache Implementation (PMAT-014)
//!
//! Implements PagedAttention-style KV cache management per cbtop spec S18.
//!
//! # Overview
//!
//! PagedAttention manages KV cache memory using fixed-size blocks, enabling:
//! - Dynamic memory allocation without fragmentation
//! - Copy-on-write for beam search
//! - Efficient eviction under memory pressure
//!
//! # Citations
//!
//! - [Kwon et al. 2023] "Efficient Memory Management for LLM Serving with PagedAttention" SOSP
//! - [Xiao et al. 2023] "StreamingLLM: Efficient Streaming with Attention Sinks" arXiv
//! - [Yu et al. 2022] "ORCA: A Distributed Serving System for Transformer-Based Models" OSDI

mod cache;
mod types;

pub use cache::PagedKvCache;
pub use types::{
    BlockId, CacheStats, EvictionStrategy, KvBlock, PagedKvError, PagedKvResult, SeqId,
    SequenceInfo,
};

#[cfg(test)]
mod tests;