Struct ParallelBatchProcessor 

pub struct ParallelBatchProcessor { /* private fields */ }

Advanced parallel batch processor using SciRS2 and Rayon

This processor leverages parallel operations for:

• Optimal work distribution across cores
• Adaptive load balancing
• Memory-efficient chunking
• NUMA-aware processing

Implementations

impl ParallelBatchProcessor

pub fn new(config: ParallelBatchConfig) -> Result<Self>

Create new parallel batch processor

pub fn num_workers(&self) -> usize

Get number of worker threads

pub fn process_parallel<T, F, R>( &self, items: Vec<T>, process_fn: F, ) -> Result<Vec<R>>

where T: Send + Sync, F: Fn(&T) -> R + Send + Sync, R: Send,

Process batch in parallel with automatic load balancing

pub fn process_with_load_balancing<T, F, R>( &self, items: Vec<T>, process_fn: F, ) -> Result<Vec<R>>

where T: Send + Sync, F: Fn(&T) -> R + Send + Sync, R: Send,

Process batch with dynamic load balancing (using rayon’s work stealing)

Uses Rayon’s work stealing for:

• Automatic work stealing between threads
• Dynamic work distribution
• Optimal CPU utilization

pub fn process_memory_efficient<T, F, R>( &self, items: Vec<T>, process_fn: F, ) -> Result<Vec<R>>

where T: Send + Sync, F: Fn(&T) -> R + Send + Sync, R: Send,

Process very large batches with memory-efficient chunking

Uses chunked parallel processing to:

• Respect memory limits
• Optimize cache locality
• Minimize memory allocations

pub fn process_nested_parallel<T, F, R>( &self, items: Vec<Vec<T>>, process_fn: F, ) -> Result<Vec<Vec<R>>>

where T: Send + Sync, F: Fn(&T) -> R + Send + Sync, R: Send,

Process with nested parallelism using rayon

Enables safe nested parallel processing:

• Automatic thread lifetime management
• Rayon’s work stealing
• Cache-friendly execution

pub fn get_stats(&self) -> ParallelProcessingStats

Get processing statistics