Struct EnginePool 

pub struct EnginePool { /* private fields */ }

A pool of InferenceEngine replicas guarded by a semaphore.

Construct with EnginePool::new; acquire an engine with EnginePool::acquire. The returned EngineLease derefs to the engine and returns it to the pool on drop.

Implementations

impl EnginePool

pub fn new(engines: Vec<InferenceEngine<'static>>) -> Arc<Self>

Build a pool from a vector of engine replicas.

The pool size is engines.len(), guaranteed to be at least 1 (an empty input yields a 1-permit pool with no engines, which would only ever return PoolError::Empty; callers must pass at least one engine). The semaphore is seeded with size permits.

pub fn size(&self) -> usize

Number of replicas in the pool.

pub fn set_metrics_all( &self, metrics: &Arc<InferenceMetrics>, ) -> Result<(), PoolError>

Attach a shared crate::metrics::InferenceMetrics to every replica in the pool.

This wires the per-engine telemetry (prefill / decode-token / tokens-per-second histograms recorded inside generate*) onto each replica, mirroring the single-engine engine.set_metrics(..) call the CLI serve handler performed before this pool existed. It must be called while the pool is idle (right after construction, before any lease is handed out), so all replicas are present in idle.

Returns PoolError::Poisoned if the idle mutex was poisoned. The metrics Arc is cloned once per replica so they all share one instance.

pub async fn acquire(self: &Arc<Self>) -> Result<EngineLease, PoolError>

Acquire an engine from the pool, waiting asynchronously if all replicas are currently in use.

Returns an EngineLease that derefs to the engine and returns it to the pool on drop. The acquired permit is held for the lease’s lifetime.

Trait Implementations

impl Debug for EnginePool

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.