Struct AsyncBatchProcessor 

pub struct AsyncBatchProcessor { /* private fields */ }

Batch processor for async inference.

Collects multiple requests and processes them in batches for improved throughput.

Implementations

impl AsyncBatchProcessor

pub fn new( session: AsyncSession, max_batch_size: usize, timeout_ms: u64, ) -> Self

Create a new batch processor.

Arguments

• session - The async session to use for inference
• max_batch_size - Maximum number of requests to batch together
• timeout_ms - Maximum time to wait for batch to fill (milliseconds)

Example

let processor = AsyncBatchProcessor::new(
    session,
    32,    // batch up to 32 requests
    10,    // wait max 10ms for batch to fill
);

pub async fn infer( &self, inputs: HashMap<String, Tensor>, ) -> Result<HashMap<String, Tensor>>

Submit a request for batched inference.

The request will be batched with other concurrent requests up to max_batch_size or until timeout_ms elapses.

Arguments

• inputs - Input tensors for this request

Returns

Output tensors for this specific request