Struct AsyncInferenceEngine 

pub struct AsyncInferenceEngine { /* private fields */ }

Async inference engine with bounded concurrency.

Wraps a synchronous InferenceEngine and provides async methods that use spawn_blocking under the hood. The semaphore limits how many concurrent inference requests can be in flight, protecting both memory and CPU utilization.

Implementations

impl AsyncInferenceEngine

pub fn new(engine: InferenceEngine<'static>, max_concurrent: usize) -> Self

Create a new async inference engine wrapping the given engine.

max_concurrent controls how many inference requests may execute concurrently. A value of 1 serializes all requests.

pub fn with_metrics(self, metrics: Arc<InferenceMetrics>) -> Self

Attach shared metrics for recording inference telemetry.

pub async fn generate( &self, prompt_tokens: Vec<u32>, max_tokens: usize, ) -> RuntimeResult<Vec<u32>>

Generate tokens asynchronously.

Blocks the caller until a semaphore permit is acquired, then dispatches the CPU-bound generation to a blocking thread.

pub async fn generate_streaming( &self, prompt_tokens: Vec<u32>, max_tokens: usize, ) -> RuntimeResult<UnboundedReceiver<u32>>

Generate tokens with streaming via an unbounded channel.

Returns a receiver that yields tokens as they are generated. The generation happens on a blocking thread; the receiver can be consumed asynchronously.

pub fn active_requests(&self) -> usize

Current number of active (in-flight) requests.

Computed as max_concurrent - available_permits.

pub fn max_concurrent(&self) -> usize

Maximum concurrent requests this engine allows.

pub fn has_capacity(&self) -> bool

Check if the engine has capacity for at least one more request.

pub fn engine(&self) -> &Arc<Mutex<InferenceEngine<'static>>>

Get a reference to the underlying engine (behind a mutex).