Struct ModelCache 

pub struct ModelCache {
    pub hits: AtomicU64,
    pub misses: AtomicU64,
    /* private fields */
}

Thread-safe in-process model cache.

Uses a Mutex-guarded HashMap internally. Eviction is based on idle time (longest-idle entry is removed first) when the slot or memory budget is exceeded.

Fields

hits: AtomicU64

Cumulative number of cache hits.

misses: AtomicU64

Cumulative number of cache misses.

Implementations

impl ModelCache

pub fn new(config: ModelCacheConfig) -> Self

Create a new, empty cache with the given configuration.

pub fn get_or_insert<F>(&self, key: &str, loader: F) -> Arc<ModelEntry>

where F: FnOnce() -> ModelEntry,

Return a shared reference to the cached entry for key, or insert a new one produced by loader if none exists (or if the existing entry is stale).

The returned Arc allows callers to hold a reference to the entry while the cache mutex is not held.

pub fn contains(&self, key: &str) -> bool

Return true if a non-stale entry exists for key.

pub fn evict(&self, key: &str) -> bool

Remove the entry for key. Returns true if an entry was removed.

pub fn evict_stale(&self) -> usize

Remove all entries that have been idle longer than the configured TTL. Returns the number of entries removed.

pub fn len(&self) -> usize

Number of entries currently in the cache.

pub fn is_empty(&self) -> bool

true when the cache holds no entries.

pub fn hit_rate(&self) -> f32

Cache hit rate as a fraction in [0.0, 1.0].

Returns 0.0 when no lookups have been performed yet.

pub fn total_memory_bytes(&self) -> usize

Sum of memory_bytes across all cached entries.

pub fn stats(&self) -> ModelCacheStats

Take a statistics snapshot of the current cache state.