Struct InferencePools 

pub struct InferencePools { /* private fields */ }

The world’s live per-model inference pools. Cheap to clone-share behind an Arc; semaphores are created lazily the first time a model is seen.

Implementations

impl InferencePools

pub fn new(config: InferencePoolConfig) -> Self

Build the pools from a configuration.

pub fn with_wake(self, wake: Arc<Notify>) -> Self

Attach the tick-loop wake handle, so that releasing a permit wakes the driver.

This is load-bearing, not a nicety. dispatch_inference leaves a slot-starved agent ReadyToInfer to be retried “on a later tick”, and the loop is event-driven: a later tick only happens when something wakes it. So every path that frees a permit owes the loop a wake, or the freed slot is invisible and everything queued behind it stays parked (issue #189).

Hanging the wake off the permit’s Drop rather than off each release site is the point: the obligation can’t be forgotten by a new call site, and it covers the paths that don’t report an outcome at all - notably a cancelled job, which frees its permit and returns with nothing to send.

pub async fn acquire(&self, model: &str) -> InferencePermit

Acquire a permit for model, waiting for a free slot if the pool is full. The returned InferencePermit releases the slot when dropped - so the caller holds it for exactly the duration of the inference request.

pub fn try_acquire(&self, model: &str) -> Option<InferencePermit>

Try to take a permit for model without waiting. Returns None if the pool is currently full.

This is what the synchronous ECS inference-dispatch system calls: a system can’t .await, so instead of blocking on a full pool it leaves the agent ReadyToInfer and retries on a later tick.

pub fn occupancy(&self) -> Vec<PoolOccupancy>

How many slots are in use per model, against each model’s cap. None as the cap means the model is unbounded. Only models that have actually been used appear - the semaphores are created lazily.

This is what makes “the pool is full and has been for hours” observable rather than inferred.

Trait Implementations

impl Debug for InferencePools

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.