Struct DataLoaderBuilder 

pub struct DataLoaderBuilder<Src, S, C, T = Identity> { /* private fields */ }

Builder for DataLoader.

Implementations

impl<Src, S, C, T> DataLoaderBuilder<Src, S, C, T>

pub fn batch_size(self, n: usize) -> Self

Number of items per batch. Default: 1.

pub fn prefetch_depth(self, n: usize) -> Self

Number of batches workers may run ahead of the consumer. At most num_workers + prefetch_depth batches are in flight or buffered. Default: 2.

pub fn drop_last(self, v: bool) -> Self

Whether to drop a non-full final batch. Default: false.

pub fn num_workers(self, n: usize) -> Self

Number of batches processed in parallel — PyTorch’s num_workers.

Batches run on the loader’s thread pool, built once and reused by every epoch. 0 (default) processes batches directly in Iterator::next.

pub fn item_concurrency(self, n: usize) -> Self

Number of items of one batch fetched in parallel. Default: 1.

With 1, a batch is fetched with one Dataset::get_batch call. With n > 1, each batch is fetched by n concurrent lanes, each calling Dataset::get for the batch’s next item, so at most n items of a batch are in flight. Useful for I/O-bound datasets (network, object storage) and for few, large CPU-bound batches.

The loader’s pool has max(num_workers, 1) × item_concurrency threads; sizing it for the machine is the caller’s choice.

pub fn items_in_parallel(self, n: usize) -> Self

👎Deprecated since 0.3.0:

renamed to item_concurrency

Former name of item_concurrency.

pub fn sampler<S2: Sampler>( self, sampler: S2, ) -> DataLoaderBuilder<Src, S2, C, T>

Replace the sampler and preserve all other settings.

pub fn transform<T2>(self, transform: T2) -> DataLoaderBuilder<Src, S, C, T2>

where Src: LoaderSource, T2: Transform<Src::Item>,

Apply transform to every item, with its index, before collation; it replaces any previous one (chain them with Transform::then). Default: Identity, at no cost.

It runs where the item is loaded: right after get / get_batch on the worker for a Dataset (in each item lane when item_concurrency is above 1, so items are transformed in parallel), or on the loader’s pool once the item is fetched for an AsyncDataset, so the async runtime never does CPU work. The collator receives the transformed items. An error fails the item’s batch; a panic is re-raised by the iterator.

use dataloader_rs::{DataLoader, Dataset, error::Result};

struct Numbers;

impl Dataset for Numbers {
    type Item = u64;
    fn get(&self, index: usize) -> Result<u64> { Ok(index as u64) }
    fn len(&self) -> usize { 4 }
}

let mut loader = DataLoader::builder(Numbers)
    .batch_size(2)
    .transform(|index: usize, x: u64| -> Result<String> { Ok(format!("{index}:{}", x * 10)) })
    .build();
let batches: Vec<Vec<String>> = loader.iter().map(|b| b.unwrap()).collect();
assert_eq!(batches, [["0:0", "1:10"], ["2:20", "3:30"]]);

pub fn collator<C2>(self, collator: C2) -> DataLoaderBuilder<Src, S, C2, T>

Replace the collator and preserve all other settings.

pub fn build(self) -> DataLoader<Src, S, C, T>

where Src: LoaderSource, S: Sampler, T: Transform<Src::Item>, C: Collator<T::Output>,

Finalize configuration and construct a DataLoader.