Struct InferencePools 

pub struct InferencePools { /* private fields */ }

The world’s live per-model inference pools. Cheap to clone-share behind an Arc; semaphores are created lazily the first time a model is seen.

Implementations

impl InferencePools

pub fn new(config: InferencePoolConfig) -> Self

Build the pools from a configuration.

pub async fn acquire(&self, model: &str) -> InferencePermit

Acquire a permit for model, waiting for a free slot if the pool is full. The returned InferencePermit releases the slot when dropped - so the caller holds it for exactly the duration of the inference request.

pub fn try_acquire(&self, model: &str) -> Option<InferencePermit>

Try to take a permit for model without waiting. Returns None if the pool is currently full.

This is what the synchronous ECS inference-dispatch system calls: a system can’t .await, so instead of blocking on a full pool it leaves the agent ReadyToInfer and retries on a later tick.

Trait Implementations

impl Debug for InferencePools

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.