Skip to main content

PagedKvCacheManager

Struct PagedKvCacheManager 

Source
pub struct PagedKvCacheManager { /* private fields */ }
Expand description

Thread-safe high-level manager orchestrating KV-caches across multiple concurrent inference sequences.

Implementations§

Source§

impl PagedKvCacheManager

Source

pub fn new(total_blocks: usize, block_size: usize) -> PagedKvCacheManager

Source

pub fn with_defaults(total_blocks: usize) -> PagedKvCacheManager

Source

pub fn create_sequence(&self, seq_id: u64)

Registers a new sequence/request with the cache manager.

Source

pub fn append_tokens( &self, seq_id: u64, tokens: &[u32], ) -> Result<(), CacheError>

Appends tokens to an active sequence.

Source

pub fn rollback_sequence( &self, seq_id: u64, valid_token_count: usize, ) -> Result<usize, CacheError>

Executes an O(1) rollback on the sequence when speculative tokens are rejected.

Source

pub fn remove_sequence(&self, seq_id: u64) -> Result<(), CacheError>

Releases all KV-cache memory held by a sequence when the request finishes.

Source

pub fn evict_lru_sequence(&self) -> Result<Option<u64>, CacheError>

Evicts the least recently used sequence to reclaim blocks under memory pressure. Returns the sequence ID of the evicted sequence, if any.

Source

pub fn defragment(&self)

Defragments the free physical block list to optimize spatial memory cache locality.

Source

pub fn utilization_ratio(&self) -> f32

Current physical block pool utilization ratio [0.0 - 1.0].

Source

pub fn fragmentation_ratio(&self) -> f32

Current free memory fragmentation ratio [0.0 - 1.0].

Source

pub fn fork_sequence(&self, src_id: u64, dest_id: u64) -> Result<(), CacheError>

Clones and forks a sequence’s KV-cache table (e.g. for speculative branch evaluation).

Source

pub fn free_blocks(&self) -> usize

Returns the number of free physical blocks in the pool.

Source

pub fn allocated_blocks(&self) -> usize

Returns the number of allocated physical blocks.

Source

pub fn prefix_entries(&self) -> usize

Returns the number of active prefix cache entries.

Source§

impl PagedKvCacheManager

Source

pub fn export_snapshot(&self) -> PrefixCacheSnapshot

Exports current warm prefix cache to a snapshot.

Source

pub fn import_snapshot( &self, snapshot: &PrefixCacheSnapshot, ) -> Result<usize, CacheError>

Imports a prefix cache snapshot into the active pool.

Source

pub fn save_snapshot_json(&self) -> Result<String, CacheError>

Saves the warm prefix cache to a JSON string.

Source

pub fn load_snapshot_json(&self, json_str: &str) -> Result<usize, CacheError>

Loads and warms the cache from a JSON snapshot string.

Trait Implementations§

Source§

impl Debug for PagedKvCacheManager

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T> Instrument for T

Source§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
Source§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V

Source§

impl<T> WithSubscriber for T

Source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more