Struct ParallelBatchProcessor 

pub struct ParallelBatchProcessor<Item> { /* private fields */ }

Processes batch items with sequential fallback and scoped standard threads.

The processor stores the supplied consumer as an ArcConsumer so every worker can share it safely. By default, small batches run sequentially to avoid thread setup overhead. Larger batches use scoped worker threads for each BatchProcessor::process call, therefore input items may borrow data from the caller as long as they are Send. Running progress is reported between items on the sequential path and from a scoped reporter thread on the parallel path.

Type Parameters

• Item - Item type consumed by the stored consumer.

use std::{
    sync::{
        Arc,
        atomic::{
            AtomicUsize,
            Ordering,
        },
    },
};

use qubit_batch::{
    BatchProcessor,
    ParallelBatchProcessor,
};

let total = Arc::new(AtomicUsize::new(0));
let total_for_consumer = Arc::clone(&total);
let mut processor = ParallelBatchProcessor::builder(move |item: &usize| {
    total_for_consumer.fetch_add(*item, Ordering::Relaxed);
})
.thread_count(2)
.sequential_threshold(0)
.build()
.expect("parallel processor configuration should be valid");

let result = processor
    .process([1, 2, 3])
    .expect("array length should be exact");

assert!(result.is_success());
assert_eq!(total.load(Ordering::Relaxed), 6);

Implementations

impl<Item> ParallelBatchProcessor<Item>

pub const DEFAULT_REPORT_INTERVAL: Duration

Default interval between progress callbacks.

pub const DEFAULT_SEQUENTIAL_THRESHOLD: usize = 100

Default maximum batch size that still uses sequential processing.

pub fn new<C>(consumer: C) -> Self

where C: Consumer<Item> + Send + Sync + 'static,

Creates a parallel consumer-backed batch processor.

Parameters

• consumer - Thread-safe consumer invoked once for each accepted item.

Returns

A processor storing consumer as an ArcConsumer and using Self::default_thread_count workers.

pub fn builder<C>(consumer: C) -> ParallelBatchProcessorBuilder<Item>

where C: Consumer<Item> + Send + Sync + 'static,

Creates a builder for configuring a parallel consumer-backed processor.

Parameters

• consumer - Thread-safe consumer invoked once for each accepted item.

Returns

A builder initialized with default settings.

pub fn default_thread_count() -> usize

Returns the default worker-thread count.

Returns

The available CPU parallelism, or 1 if it cannot be detected.

pub const fn thread_count(&self) -> usize

Returns the configured worker-thread count.

Returns

The maximum number of scoped worker threads used for one batch.

pub const fn sequential_threshold(&self) -> usize

Returns the configured sequential fallback threshold.

Returns

The maximum item count that still runs sequentially.

pub const fn report_interval(&self) -> Duration

Returns the configured progress-report interval.

Returns

The minimum time between due-based running progress callbacks.

pub fn reporter(&self) -> &Arc<dyn ProgressReporter>

Returns the configured progress reporter.

Returns

A shared reference to the configured progress reporter.

pub const fn consumer(&self) -> &ArcConsumer<Item>

Returns the stored consumer.

Returns

A shared reference to the arc-backed consumer.

pub fn into_consumer(self) -> ArcConsumer<Item>

Consumes this processor and returns the stored consumer.

Returns

The arc-backed consumer used by this processor.

Trait Implementations

impl<Item> BatchProcessor<Item> for ParallelBatchProcessor<Item>

where Item: Send,

fn process_with_count<I>( &mut self, items: I, count: usize, ) -> Result<BatchProcessResult, Self::Error>

where I: IntoIterator<Item = Item>,

Processes items sequentially for small batches or on scoped workers.

Parameters

• items - Item source for the batch.
• count - Declared number of items expected from items.

Returns

A result with completed and processed counts equal to the number of consumer calls when the input source yields exactly count items.

Errors

Returns BatchProcessError::CountShortfall when the source ends before count, or BatchProcessError::CountExceeded when the source yields an extra item. Extra items are observed but not passed to the consumer.

Panics

Propagates any panic raised by the stored consumer from the caller thread or a worker thread, or by the configured progress reporter.

type Error = BatchProcessError

Error returned by this processor.

fn process<I>(&mut self, items: I) -> Result<BatchProcessResult, Self::Error>

where I: IntoIterator<Item = Item>, I::IntoIter: ExactSizeIterator,

Processes items as one batch using its exact iterator length.