Struct AppState 

pub struct AppState { /* private fields */ }

Server state.

Holds a pool of inference-engine replicas behind a semaphore rather than a single mutex, so up to pool.size() requests can generate concurrently. The default path (a 1-element pool) is byte-identical to the previous single-mutex design.

Implementations

impl AppState

pub async fn acquire_engine(&self) -> Result<EngineLease, PoolError>

Acquire an exclusive lease on one engine replica from the pool, waiting asynchronously if every replica is currently busy.

The returned EngineLease derefs to the engine (so callers invoke the usual generate* methods) and returns it to the pool on drop.

pub fn engines(&self) -> &Arc<EnginePool>

Access the underlying engine pool.

pub fn tokenizer(&self) -> Option<&TokenizerBridge>

Access the optional tokenizer.

pub fn metrics(&self) -> &Arc<InferenceMetrics>

Access the shared metrics instance.