Module parallel 

Parallel data loading with multi-worker support.

Provides a parallel data loader that uses multiple threads to load data in parallel, similar to PyTorch’s DataLoader with num_workers > 0.

Example

use alimentar::{parallel::ParallelDataLoader, ArrowDataset, Dataset};

let dataset = ArrowDataset::from_parquet("data.parquet").unwrap();
let loader = ParallelDataLoader::new(dataset)
    .batch_size(32)
    .num_workers(4)
    .prefetch(2);

for batch in loader {
    println!("Batch has {} rows", batch.num_rows());
}

Structs

ParallelDataLoader

Parallel data loader with multi-worker support.

ParallelDataLoaderBuilder

Builder for parallel data loader configuration.

Enums

ParallelDataLoaderIterator

Iterator for parallel data loader.