Struct DataLoader 

pub struct DataLoader<Src, S: Sampler, C, T = Identity> { /* private fields */ }

High-performance DataLoader with a PyTorch-like interface.

The dataset fetches items; the loader applies its Transform to each one (T, Identity unless set with transform), then collates them with C. Src holds the dataset: SyncSource for a Dataset, AsyncSource for an AsyncDataset (feature async).

Implementations

impl<D: Dataset> DataLoader<SyncSource<D>, SequentialSampler, VecCollator>

pub fn builder( dataset: D, ) -> DataLoaderBuilder<SyncSource<D>, SequentialSampler, VecCollator>

Create a DataLoaderBuilder starting with SequentialSampler and VecCollator as defaults.

impl<Src, S, C, T> DataLoader<Src, S, C, T>

where Src: LoaderSource, S: Sampler, T: Transform<Src::Item>, C: Collator<T::Output>,

pub fn iter(&mut self) -> DataLoaderIter<Src, C, T>

Start one epoch of iteration.

Advances the sampler (hence &mut self); the returned iterator does not borrow the loader.

pub fn iter_with_indices(&mut self) -> IndexedIter<Src, C, T>

Start one epoch, like iter, whose batches come with the dataset indices of their items: (indices, batch), indices[k] being the index of the batch’s k-th item as the sampler drew it. A failed batch is still one step of the epoch: the next batch gets the next indices.

use dataloader_rs::{DataLoader, Dataset, RandomSampler, error::Result};

struct Squares;

impl Dataset for Squares {
    type Item = usize;
    fn get(&self, index: usize) -> Result<usize> { Ok(index * index) }
    fn len(&self) -> usize { 10 }
}

let mut loader = DataLoader::builder(Squares)
    .batch_size(4)
    .sampler(RandomSampler::new(7))
    .build();
for batch in loader.iter_with_indices() {
    let (indices, squares) = batch.unwrap();
    assert!(indices.iter().zip(squares).all(|(i, s)| i * i == s));
}

pub fn dataset(&self) -> &Src::Dataset

Reference to the underlying dataset.

pub fn transform(&self) -> &T

Reference to the transform.

pub fn collator(&self) -> &C

Reference to the collator.

pub fn len(&self) -> usize

Total number of samples in the dataset.

pub fn batch_len(&self) -> usize

Total number of batches for one epoch with current batch settings.

Counts what the sampler yields, so a sharding or subsampling sampler (e.g. DistributedSampler) is accounted for.

pub fn is_empty(&self) -> bool

Returns true when the dataset is empty.

Trait Implementations

impl<Src, S, C, T> IntoIterator for &mut DataLoader<Src, S, C, T>

where Src: LoaderSource, S: Sampler, T: Transform<Src::Item>, C: Collator<T::Output>,

type Item = Result<<C as Collator<<T as Transform<<Src as Sealed>::Item>>::Output>>::Batch, Box<dyn Error + Sync + Send>>

The type of the elements being iterated over.

type IntoIter = DataLoaderIter<Src, C, T>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.