Struct AdaptiveBatcher 

pub struct AdaptiveBatcher { /* private fields */ }

Adaptive batch size controller

Tracks recent inference latencies and adjusts the recommended batch size to keep per-batch latency near the configured target.

Implementations

impl AdaptiveBatcher

pub fn new(config: AdaptiveBatchConfig) -> Self

Create a new AdaptiveBatcher starting at min_batch_size.

pub fn recommended_batch_size(&self) -> usize

Return the current recommended batch size.

pub fn update_latency(&mut self, latency_ms: f64, batch_size: usize)

Update the batch-size estimate based on the observed latency for a completed batch.

Parameters

• latency_ms: wall-clock time the batch took in milliseconds
• batch_size: number of items that were in the completed batch

pub fn create_batches( &self, requests: Vec<PredictionRequest>, ) -> Vec<Vec<PredictionRequest>>

Group a flat list of requests into batches, each of at most recommended_batch_size() items.

pub fn average_latency_ms(&self) -> f64

Rolling average latency over the recent observation window.

Returns 0.0 when no observations have been recorded yet.

pub fn total_batches(&self) -> u64

Total number of completed batches.

pub fn total_items(&self) -> u64

Total number of individual items processed across all batches.

pub fn config(&self) -> &AdaptiveBatchConfig

Return a reference to the current configuration.