Struct ModelCache 

pub struct ModelCache { /* private fields */ }

Multi-model cache with LRU eviction under VRAM pressure.

Invariants:

• At most one engine has residency == Gpu at a time (single-GPU inference).
• lru_order tracks all entries from least-recently-used (front) to most-recently-used (back).
• max_cached limits total entries (Gpu, Unloaded, and Parked).

Implementations

impl ModelCache

pub fn new(max_cached: usize) -> Self

pub fn insert( &mut self, engine: Box<dyn InferenceEngine>, vram_bytes: u64, ) -> Option<Box<dyn InferenceEngine>>

Insert an engine into the cache. If the cache is full, the LRU entry is dropped entirely. Returns the evicted engine (if any) for cleanup.

pub fn get_mut(&mut self, model_name: &str) -> Option<&mut CachedEngine>

Get a mutable reference to the engine for a model, if cached.

pub fn contains(&self, model_name: &str) -> bool

Check if a model is in the cache.

pub fn remove(&mut self, model_name: &str) -> Option<Box<dyn InferenceEngine>>

Remove a model from the cache entirely, returning its engine.

pub fn unload_all(&mut self) -> Vec<String>

Unload all models from GPU. Returns names of models that were unloaded. Unloaded models are parked (retain tokenizers/caches for faster reload).

pub fn unload_active(&mut self) -> Option<String>

Unload the current GPU-resident model (if any) to make room for a new one. The engine is parked (retains tokenizers/caches) for faster reload. Returns the name of the unloaded model.

pub fn clear(&mut self) -> Vec<Box<dyn InferenceEngine>>

Drop all entries, returning all engines for cleanup.

pub fn active_vram_bytes(&self) -> u64

VRAM footprint of the currently GPU-resident model (0 if none loaded).

pub fn active_model(&self) -> Option<&str>

The currently GPU-loaded model name.

pub fn cached_model_names(&self) -> Vec<String>

All cached model names (any residency).

pub fn len(&self) -> usize

Number of cached entries.

pub fn is_empty(&self) -> bool