Struct ModelPool 

pub struct ModelPool { /* private fields */ }

The multi-model LRU warm-pool.

Owned by the inference worker thread; no Send + Sync requirement for the HashMap internals because all accesses happen on one thread.

Implementations

impl ModelPool

pub fn new(capacity: usize, mem_budget_mb: usize) -> Self

Create a new empty pool.

• capacity: maximum number of models that may be resident at once.
• mem_budget_mb: memory budget in MiB (0 = unlimited).

pub fn loader_register(&mut self, id: impl Into<String>, spec: ModelSpec)

Register a model spec so it can be loaded on demand via acquire.

Called by the admin POST /admin/models/load route before initiating a background load, and at startup for models listed in [router] preload.

pub fn loader(&self) -> &ModelLoader

Access the embedded loader (read-only).

pub fn acquire( &mut self, model_id: &str, ext_loader: Option<&ModelLoader>, ) -> ServerResult<Arc<RwLock<LoadedModel>>>

Acquire an engine for model_id.

If the model is already loaded it is promoted to the MRU position and its Arc<RwLock<LoadedModel>> is returned immediately.

Otherwise the model is loaded synchronously (blocking the calling thread) after evicting LRU entries as needed.

The optional loader parameter allows callers (like tests) to supply an external loader; None uses the pool’s embedded loader.

pub fn release(&self, model_id: &str)

Decrement inflight count for a model when the caller is done with it.

pub fn unload(&mut self, model_id: &str) -> ServerResult<()>

Explicitly unload a model, freeing its memory.

Returns an error if the model ID is not currently loaded.

pub fn list(&self) -> Vec<ModelStatus>

List the status of all known models (loaded + pending).

pub fn mark_loading(&mut self, model_id: impl Into<String>)

Mark a model as being loaded in a background task.

pub fn mark_ready( &mut self, model_id: &str, engine: InferenceEngine, mem_bytes: usize, ) -> ServerResult<()>

Mark a pending model as ready (called after a background load succeeds).

Moves the engine from the temporary pending state into the loaded map.

pub fn mark_failed(&mut self, model_id: &str, reason: String)

Mark a pending model as failed to load.

pub fn current_mem_bytes(&self) -> usize

Total estimated bytes currently consumed by loaded models.