Struct ModelCache 

pub struct ModelCache { /* private fields */ }

Multi-model cache with LRU eviction under VRAM pressure.

Invariants:

• At most one engine has residency == Gpu at a time (single-GPU inference).
• lru_order tracks all entries from least-recently-used (front) to most-recently-used (back).
• max_cached limits total entries across both residency states (Gpu and Parked).
• in_flight holds names temporarily checked out via take() until they are returned via restore(). Such names are physically absent from entries/lru_order but logically still part of the cache for the purposes of contains() and snapshot() — without this, a parallel reader during a take window would mistakenly conclude the model is uncached.

Implementations

impl ModelCache

pub fn new(max_cached: usize) -> Self

pub fn insert( &mut self, engine: Box<dyn InferenceEngine>, vram_bytes: u64, ) -> Option<Box<dyn InferenceEngine>>

Insert an engine into the cache. If the cache is full, the LRU entry is dropped entirely. Returns the evicted engine (if any) for cleanup.

pub fn get(&self, model_name: &str) -> Option<&CachedEngine>

Get a reference to a cached engine entry (does not update LRU order).

pub fn get_mut(&mut self, model_name: &str) -> Option<&mut CachedEngine>

Get a mutable reference to the engine for a model, if cached.

pub fn take(&mut self, model_name: &str) -> Option<CachedEngine>

Remove an engine from the cache, returning the full entry. Used by the take-and-restore pattern: remove before inference, re-insert after. While the engine is checked out, the name remains in in_flight so parallel readers (contains, snapshot, cached_model_names) still see the model as logically cached.

pub fn restore(&mut self, cached: CachedEngine)

Re-insert a taken engine after inference completes.

pub fn clear_in_flight(&mut self, model_name: &str)

Clear an in-flight marker without restoring an engine. Called when the take-and-restore window terminates abnormally (JoinError, panic that escapes catch_unwind, etc.) and we have no engine to put back — otherwise the name leaks into in_flight forever, making contains() permanently lie to ensure_model_ready while take()/get() keep returning None.

pub fn insert_loaded( &mut self, model_name: String, engine: Box<dyn InferenceEngine>, vram_bytes: u64, ) -> Option<Box<dyn InferenceEngine>>

Insert a loaded engine with a known VRAM footprint. Unlike insert(), this takes a name separately from the engine.

pub fn contains(&self, model_name: &str) -> bool

Check if a model is in the cache. Treats names taken-but-not-restored as still cached so concurrent readers don’t see a transient hole during a take/restore window.

pub fn remove(&mut self, model_name: &str) -> Option<Box<dyn InferenceEngine>>

Remove a model from the cache entirely, returning its engine. Also clears the name from in_flight so we never claim a model is cached after explicit removal.

pub fn unload_all(&mut self) -> Vec<String>

Unload all models from GPU. Returns names of models that were unloaded. Each is transitioned to Parked (retain tokenizers/caches for faster reload).

pub fn unload_active(&mut self) -> Option<String>

Unload the current GPU-resident model (if any) to make room for a new one. The engine is parked (retains tokenizers/caches) for faster reload. Returns the name of the unloaded model.

pub fn clear(&mut self) -> Vec<Box<dyn InferenceEngine>>

Drop all entries, returning all engines for cleanup. Also clears in_flight — any caller still holding a checked-out engine must drop it on their own (they own that CachedEngine); we just stop claiming it’s logically present.

pub fn active_vram_bytes(&self) -> u64

VRAM footprint of the currently GPU-resident model (0 if none loaded).

pub fn active_model(&self) -> Option<&str>

The currently GPU-loaded model name.

pub fn cached_model_names(&self) -> Vec<String>

All cached model names (any residency, including names temporarily taken-out for in-flight inference).

pub fn snapshot(&self) -> EngineSnapshot

Snapshot of the cache’s current state — what /api/models and /api/status report. Derived directly from the cache so there’s no parallel field that can drift. During a take/restore window the engine that was GPU-resident is reflected via in_flight_active so readers don’t see a transient “no model loaded” hole.

pub fn len(&self) -> usize

Number of cached entries.

pub fn is_empty(&self) -> bool

pub fn evict_idle( &mut self, ttl: Duration, ) -> Vec<(String, Box<dyn InferenceEngine>)>

Reclaim cache entries whose last_used is older than ttl. Only entries that are not GPU-resident are eligible — the active model is always preserved. Skipped entirely when the cache holds at most one entry (so we never tear down the only warm engine after a quiet period).

Returns evicted (name, engine) pairs so callers can drop the engines outside any cache mutex — cuMemFree and safetensor unmap on drop can block other cache users for non-trivial time.

pub fn evict_lru_parked_except( &mut self, skip: Option<&str>, ) -> Option<(String, Box<dyn InferenceEngine>)>

Evict the LRU entry that is not GPU-resident and (optionally) not the named model. Returns (name, engine) so the caller can drop the engine outside the cache lock and then issue any GPU reclamation.

Used by the load-time evict-to-fit recovery path: when a fresh load’s preflight fails, we shrink the parked working set one entry at a time and retry. The skip parameter exists because the parked-reload branch must not evict the very entry it’s about to reload.