Struct TieredCache 

#[non_exhaustive]
pub struct TieredCache { /* private fields */ }

Generic tiered cache for GPU buffers.

Tracks hot/cold buffers using the built-in LruPolicy. This is the vyre primitive that helix builds inference intelligence on top of.

Implementations

impl TieredCache

pub fn new(tiers: Vec<CacheTier>) -> Self

Create a new cache with the given tiers and a default LruPolicy.

pub fn try_new(tiers: Vec<CacheTier>) -> Result<Self, BackendError>

Fallible version of Self::new.

Errors

Returns vyre_driver::BackendError if cache access metadata cannot be reserved.

impl TieredCache

pub fn with_policy(tiers: Vec<CacheTier>, policy: LruPolicy) -> Self

Create a new cache with a custom LRU policy.

pub fn try_with_policy( tiers: Vec<CacheTier>, policy: LruPolicy, ) -> Result<Self, BackendError>

Fallible version of Self::with_policy.

Errors

Returns vyre_driver::BackendError if cache access metadata cannot be reserved.

pub fn get(&self, key: u64) -> Option<&CacheEntry>

Return a reference to the entry with the given key, if it exists.

pub fn insert(&mut self, key: u64, size: u64) -> Result<(), CacheError>

Insert a new entry into the lowest tier that can fit it.

Errors

Returns CacheError::EntryTooLarge when no tier can hold the entry.

pub fn record_access(&mut self, key: u64)

Record an access for the given key.

pub fn promote(&mut self, key: u64) -> Result<(), CacheError>

Promote the entry to the next faster tier if the policy allows it.

Errors

Returns CacheError::KeyNotFound when the key does not exist.

pub fn demote(&mut self, key: u64) -> Result<(), CacheError>

Demote the entry to the next slower tier.

Errors

Returns CacheError::KeyNotFound when the key does not exist.

pub fn evict_coldest(&mut self) -> Option<u64>

Find and remove the coldest entry from the cache.

This follows the LRU policy across all tiers, starting from the lowest (coldest) tier. Returns the key of the evicted entry.