Struct TieredCache 

#[non_exhaustive]
pub struct TieredCache<P: TierPolicy = LruPolicy> { /* private fields */ }

Generic tiered cache for GPU buffers.

Tracks hot/cold buffers. TierPolicy decides promotion and eviction. This is the vyre primitive that helix builds inference intelligence on top of.

Implementations

impl TieredCache<LruPolicy>

pub fn new(tiers: Vec<CacheTier>) -> Self

Create a new cache with the given tiers and a default LruPolicy.

impl<P: TierPolicy> TieredCache<P>

pub fn with_policy(tiers: Vec<CacheTier>, policy: P) -> Self

Create a new cache with a custom policy implementation.

pub fn get(&self, key: u64) -> Option<&CacheEntry>

Return a reference to the entry with the given key, if it exists.

pub fn insert(&mut self, key: u64, size: u64) -> Result<(), CacheError>

Insert a new entry into the lowest tier that can fit it.

Errors

Returns CacheError::EntryTooLarge when no tier can hold the entry.

pub fn record_access(&mut self, key: u64)

Record an access for the given key.

pub fn promote(&mut self, key: u64) -> Result<(), CacheError>

Promote the entry to the next faster tier if the policy allows it.

Errors

Returns CacheError::KeyNotFound when the key does not exist.

pub fn demote(&mut self, key: u64) -> Result<(), CacheError>

Demote the entry to the next slower tier.

Errors

Returns CacheError::KeyNotFound when the key does not exist.