Struct BatchProcessor 

pub struct BatchProcessor { /* private fields */ }

Batch processor for high-throughput inference

Automatically batches incoming requests according to the configured strategy, executes them in a single forward pass, and returns individual results.

Performance

• Static batching: 3-5x throughput for stable workloads
• Dynamic batching: 5-10x throughput with variable request rates
• Optimal for GPU inference where batch processing is highly efficient

Example

use ronn_api::{Model, SessionOptions, BatchProcessor, BatchConfig, BatchStrategy};
use std::collections::HashMap;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
    let model = Model::load("model.onnx")?;
    let session = model.create_session(SessionOptions::default())?;

    let config = BatchConfig {
        strategy: BatchStrategy::Dynamic {
            max_batch_size: 32,
            timeout_ms: 10,
        },
        ..Default::default()
    };

    let processor = BatchProcessor::new(session, config);

    // Submit requests - they will be automatically batched
    let inputs = HashMap::new(); // Add your inputs here
    let output = processor.process(inputs).await?;
    Ok(())
}

Implementations

impl BatchProcessor

pub fn new(session: InferenceSession, config: BatchConfig) -> Self

Create a new batch processor

pub async fn process( &self, inputs: HashMap<String, Tensor>, ) -> Result<HashMap<String, Tensor>>

Submit a request for batch processing

Arguments

• inputs - Input tensors for inference

Returns

Future that resolves to the inference outputs

pub fn config(&self) -> &BatchConfig

Get the current configuration