Struct ModelHost 

pub struct ModelHost { /* private fields */ }

Cheap to clone; every clone is the same host.

Implementations

impl ModelHost

pub fn new( catalog: Arc<Mutex<Catalog>>, runtime: Arc<dyn ModelRuntime>, probe: Arc<dyn MemoryProbe + Send + Sync>, residency: Residency, ) -> Self

pub fn status(&self, id: &str) -> Result<ModelStatus, HostError>

Status of one catalogue model.

pub fn statuses(&self) -> Vec<ModelStatus>

Status of every catalogue model, in catalogue order.

pub fn subscribe(&self) -> Receiver<ModelStatus>

Receive every state transition from now on.

pub fn loaded_gib(&self) -> f32

Sum of the estimates of models holding (or about to hold) memory.

pub fn load(&self, id: &str) -> Result<ModelStatus, HostError>

Load id and mark it resident. Answers the state after the request: loading, or loaded when it already was.

pub fn unload(&self, id: &str) -> Result<ModelStatus, HostError>

Unload id and clear its residency.

pub fn restore_residents(&self)

Load every resident model, in catalogue order. Failures are logged; a refused or failed model stays resident for next time.

pub fn with_lane<R>( &self, id: &str, f: impl FnOnce(&dyn LoadedModel, &Lane) -> R, ) -> Result<R, HostError>

Serve one request on id’s lane. Blocks while the lane is busy.

pub fn try_with_lane<R>( &self, id: &str, f: impl FnOnce(&dyn LoadedModel, &Lane) -> R, ) -> Result<R, HostError>

Like Self::with_lane but refuses (LaneBusy) instead of waiting, for long requests such as a stream that would otherwise queue.

pub fn can_load(&self, model: &CatalogModel) -> bool

Whether model can be loaded (its engine has an in-process loader).

pub fn wait_for( &self, id: &str, pred: impl Fn(&ModelState) -> bool, timeout: Duration, ) -> Option<ModelStatus>

Block until id’s state satisfies pred, or timeout passes.

Trait Implementations

impl Clone for ModelHost

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.