pub struct DataLoader<'a, D = BuilderDatasetMarker, S = Empty<usize>, C = (), B = (), E = Infallible>where
D: Dataset,{ /* private fields */ }Expand description
Load indexed samples through a borrowed iterator or a reusable owned loader.
Choose the constructor according to who should own the dataset:
| Constructor | Ownership and use | Default batch |
|---|---|---|
Self::new | Borrows &dataset; iterate the returned value directly | Vec<Sample> |
Self::with_collate | Same borrowed lifetime, with a collation closure | The closure’s result |
Self::builder | Owns dataset; call build(), then iter() each epoch | DefaultCollator output |
The borrowed iterator fetches on the calling thread and yields errors once
before ending. Samples are moved, so they need not implement Clone.
For shuffling across epochs, worker prefetch, or checkpoint recovery, start
with Self::builder and follow the DataLoaderBuilder examples.
§Examples
A seeded sampler and a collation closure can produce tensor batches:
use std::convert::Infallible;
use rusttorch_core::{Result, Tensor};
use rusttorch_data::{DataLoader, Dataset, RandomSampler};
struct Rows(usize);
impl Dataset for Rows {
type Sample = Tensor;
type Error = Infallible;
fn len(&self) -> usize {
self.0
}
fn get(&self, index: usize) -> std::result::Result<Tensor, Infallible> {
Ok(Tensor::from_slice(&[index as f32]))
}
}
fn main() -> Result<()> {
let dataset = Rows(8);
let sampler = RandomSampler::new(dataset.len(), 42)?;
let loader = DataLoader::with_collate(
&dataset,
sampler,
4,
false,
|samples| Ok::<_, Infallible>(Tensor::stack(&samples, 0)),
)?;
for batch in loader {
let tensor = batch.expect("dataset and collation are infallible");
assert_eq!(tensor.size(), [4, 1]);
}
Ok(())
}Implementations§
Source§impl DataLoader<'static, BuilderDatasetMarker, Empty<usize>, (), (), Infallible>
impl DataLoader<'static, BuilderDatasetMarker, Empty<usize>, (), (), Infallible>
Sourcepub fn builder<D>(
dataset: D,
) -> DataLoaderBuilder<D, AutoBatch<SequentialSampler>, DefaultCollator>where
D: Dataset,
pub fn builder<D>(
dataset: D,
) -> DataLoaderBuilder<D, AutoBatch<SequentialSampler>, DefaultCollator>where
D: Dataset,
Starts an owned, re-iterable loader builder for dataset.
Defaults are batch size one, sequential sampling, recursive collation,
zero workers, and keeping the final short batch. The builder’s generic
parameters are inferred from the dataset and options; callers normally
do not write an explicit DataLoaderBuilder type.
use std::convert::Infallible;
use rusttorch_core::Result;
use rusttorch_data::{DataLoader, Dataset, VecCollate};
struct Rows(Vec<i64>);
impl Dataset for Rows {
type Sample = i64;
type Error = Infallible;
fn len(&self) -> usize { self.0.len() }
fn get(&self, index: usize) -> std::result::Result<i64, Infallible> {
Ok(self.0[index])
}
}
fn main() -> Result<()> {
let mut loader = DataLoader::builder(Rows(vec![1, 2, 3]))
.batch_size(2)
.collate(VecCollate)
.build()?;
let batches = loader.iter().collect::<std::result::Result<Vec<_>, _>>()
.expect("the dataset and VecCollate are infallible");
assert_eq!(batches, [vec![1, 2], vec![3]]);
Ok(())
}Source§impl<'a, D, S> DataLoader<'a, D, S, fn(Vec<D::Sample>) -> Result<Vec<D::Sample>, D::Error>, Vec<D::Sample>, D::Error>
impl<'a, D, S> DataLoader<'a, D, S, fn(Vec<D::Sample>) -> Result<Vec<D::Sample>, D::Error>, Vec<D::Sample>, D::Error>
Sourcepub fn new(
dataset: &'a D,
sampler: S,
batch_size: usize,
drop_last: bool,
) -> Result<Self>
pub fn new( dataset: &'a D, sampler: S, batch_size: usize, drop_last: bool, ) -> Result<Self>
Creates a loader whose batches are vectors of owned samples.
dataset is borrowed, while sampler is consumed by the loader.
A short final batch is omitted when drop_last is true.
This constructor keeps each sample unchanged in a Vec; it does not
stack tensor rows. See Self::with_collate for tensor stacking or
Self::builder for default recursive collation.
use std::convert::Infallible;
use rusttorch_data::{DataLoader, Dataset};
struct Rows([i64; 3]);
impl Dataset for Rows {
type Sample = i64;
type Error = Infallible;
fn len(&self) -> usize { self.0.len() }
fn get(&self, i: usize) -> Result<i64, Infallible> { Ok(self.0[i]) }
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
let dataset = Rows([5, 8, 13]);
let loader = DataLoader::new(&dataset, 0..dataset.len(), 2, false)?;
assert_eq!(loader.collect::<Result<Vec<_>, _>>()?, [vec![5, 8], vec![13]]);
assert_eq!(dataset.get(0)?, 5); // The dataset is still available.
Ok(())
}§Errors
Returns RustTorchError::InvalidConfiguration when batch_size is
zero.
Source§impl<'a, D, S, C, B, E> DataLoader<'a, D, S, C, B, E>
impl<'a, D, S, C, B, E> DataLoader<'a, D, S, C, B, E>
Sourcepub fn with_collate(
dataset: &'a D,
sampler: S,
batch_size: usize,
drop_last: bool,
collate: C,
) -> Result<Self>
pub fn with_collate( dataset: &'a D, sampler: S, batch_size: usize, drop_last: bool, collate: C, ) -> Result<Self>
Creates a loader with fallible custom collation.
The closure receives ownership of exactly one vector of samples and
may return any batch type. Dataset failures are converted into the
closure’s error type through From. A short final batch is omitted
when drop_last is true.
§Errors
Returns RustTorchError::InvalidConfiguration when batch_size is
zero. Dataset and collation failures are returned by iteration.
Trait Implementations§
Source§impl<D, S, C, B, E> Iterator for DataLoader<'_, D, S, C, B, E>
impl<D, S, C, B, E> Iterator for DataLoader<'_, D, S, C, B, E>
Source§fn next(&mut self) -> Option<Self::Item>
fn next(&mut self) -> Option<Self::Item>
Source§fn next_chunk<const N: usize>(
&mut self,
) -> Result<[Self::Item; N], IntoIter<Self::Item, N>>where
Self: Sized,
fn next_chunk<const N: usize>(
&mut self,
) -> Result<[Self::Item; N], IntoIter<Self::Item, N>>where
Self: Sized,
iter_next_chunk)N values. Read more1.0.0 (const: unstable) · Source§fn size_hint(&self) -> (usize, Option<usize>)
fn size_hint(&self) -> (usize, Option<usize>)
1.0.0 (const: unstable) · Source§fn count(self) -> usizewhere
Self: Sized,
fn count(self) -> usizewhere
Self: Sized,
1.0.0 (const: unstable) · Source§fn last(self) -> Option<Self::Item>where
Self: Sized,
fn last(self) -> Option<Self::Item>where
Self: Sized,
Source§fn advance_by(&mut self, n: usize) -> Result<(), NonZero<usize>>
fn advance_by(&mut self, n: usize) -> Result<(), NonZero<usize>>
iter_advance_by)n elements. Read more1.0.0 (const: unstable) · Source§fn nth(&mut self, n: usize) -> Option<Self::Item>
fn nth(&mut self, n: usize) -> Option<Self::Item>
nth element of the iterator. Read more1.28.0 (const: unstable) · Source§fn step_by(self, step: usize) -> StepBy<Self> ⓘwhere
Self: Sized,
fn step_by(self, step: usize) -> StepBy<Self> ⓘwhere
Self: Sized,
1.0.0 (const: unstable) · Source§fn chain<U>(self, other: U) -> Chain<Self, <U as IntoIterator>::IntoIter> ⓘ
fn chain<U>(self, other: U) -> Chain<Self, <U as IntoIterator>::IntoIter> ⓘ
1.0.0 (const: unstable) · Source§fn zip<U>(self, other: U) -> Zip<Self, <U as IntoIterator>::IntoIter> ⓘwhere
Self: Sized,
U: IntoIterator,
fn zip<U>(self, other: U) -> Zip<Self, <U as IntoIterator>::IntoIter> ⓘwhere
Self: Sized,
U: IntoIterator,
Source§fn intersperse(self, separator: Self::Item) -> Intersperse<Self> ⓘ
fn intersperse(self, separator: Self::Item) -> Intersperse<Self> ⓘ
iter_intersperse)separator between items
of the original iterator. Read moreSource§fn intersperse_with<G>(self, separator: G) -> IntersperseWith<Self, G> ⓘ
fn intersperse_with<G>(self, separator: G) -> IntersperseWith<Self, G> ⓘ
iter_intersperse)separator
between items of the original iterator. Read more1.0.0 (const: unstable) · Source§fn map<B, F>(self, f: F) -> Map<Self, F> ⓘ
fn map<B, F>(self, f: F) -> Map<Self, F> ⓘ
1.21.0 (const: unstable) · Source§fn for_each<F>(self, f: F)
fn for_each<F>(self, f: F)
1.0.0 (const: unstable) · Source§fn filter<P>(self, predicate: P) -> Filter<Self, P> ⓘ
fn filter<P>(self, predicate: P) -> Filter<Self, P> ⓘ
1.0.0 (const: unstable) · Source§fn filter_map<B, F>(self, f: F) -> FilterMap<Self, F> ⓘ
fn filter_map<B, F>(self, f: F) -> FilterMap<Self, F> ⓘ
1.0.0 (const: unstable) · Source§fn enumerate(self) -> Enumerate<Self> ⓘwhere
Self: Sized,
fn enumerate(self) -> Enumerate<Self> ⓘwhere
Self: Sized,
1.0.0 (const: unstable) · Source§fn take_while<P>(self, predicate: P) -> TakeWhile<Self, P> ⓘ
fn take_while<P>(self, predicate: P) -> TakeWhile<Self, P> ⓘ
1.57.0 (const: unstable) · Source§fn map_while<B, P>(self, predicate: P) -> MapWhile<Self, P> ⓘ
fn map_while<B, P>(self, predicate: P) -> MapWhile<Self, P> ⓘ
1.0.0 (const: unstable) · Source§fn skip(self, n: usize) -> Skip<Self> ⓘwhere
Self: Sized,
fn skip(self, n: usize) -> Skip<Self> ⓘwhere
Self: Sized,
n elements. Read more1.0.0 (const: unstable) · Source§fn take(self, n: usize) -> Take<Self> ⓘwhere
Self: Sized,
fn take(self, n: usize) -> Take<Self> ⓘwhere
Self: Sized,
n elements, or fewer
if the underlying iterator ends sooner. Read more1.0.0 (const: unstable) · Source§fn scan<St, B, F>(self, initial_state: St, f: F) -> Scan<Self, St, F> ⓘ
fn scan<St, B, F>(self, initial_state: St, f: F) -> Scan<Self, St, F> ⓘ
1.0.0 (const: unstable) · Source§fn flat_map<U, F>(self, f: F) -> FlatMap<Self, U, F> ⓘ
fn flat_map<U, F>(self, f: F) -> FlatMap<Self, U, F> ⓘ
1.29.0 (const: unstable) · Source§fn flatten(self) -> Flatten<Self> ⓘ
fn flatten(self) -> Flatten<Self> ⓘ
Source§fn map_windows<F, R, const N: usize>(self, f: F) -> MapWindows<Self, F, N> ⓘ
fn map_windows<F, R, const N: usize>(self, f: F) -> MapWindows<Self, F, N> ⓘ
iter_map_windows)f for each contiguous window of size N over
self and returns an iterator over the outputs of f. Like slice::windows(),
the windows during mapping overlap as well. Read more1.0.0 (const: unstable) · Source§fn inspect<F>(self, f: F) -> Inspect<Self, F> ⓘ
fn inspect<F>(self, f: F) -> Inspect<Self, F> ⓘ
1.0.0 (const: unstable) · Source§fn by_ref(&mut self) -> &mut Selfwhere
Self: Sized,
fn by_ref(&mut self) -> &mut Selfwhere
Self: Sized,
Iterator. Read more1.0.0 (const: unstable) · Source§fn collect<B>(self) -> B
fn collect<B>(self) -> B
Source§fn try_collect<B>(
&mut self,
) -> <<Self::Item as Try>::Residual as Residual<B>>::TryType
fn try_collect<B>( &mut self, ) -> <<Self::Item as Try>::Residual as Residual<B>>::TryType
iterator_try_collect)Source§fn collect_into<E>(self, collection: &mut E) -> &mut E
fn collect_into<E>(self, collection: &mut E) -> &mut E
iter_collect_into)1.0.0 (const: unstable) · Source§fn partition<B, F>(self, f: F) -> (B, B)
fn partition<B, F>(self, f: F) -> (B, B)
Source§fn is_partitioned<P>(self, predicate: P) -> bool
fn is_partitioned<P>(self, predicate: P) -> bool
iter_is_partitioned)true precede all those that return false. Read more1.27.0 (const: unstable) · Source§fn try_fold<B, F, R>(&mut self, init: B, f: F) -> R
fn try_fold<B, F, R>(&mut self, init: B, f: F) -> R
1.27.0 (const: unstable) · Source§fn try_for_each<F, R>(&mut self, f: F) -> R
fn try_for_each<F, R>(&mut self, f: F) -> R
1.0.0 (const: unstable) · Source§fn fold<B, F>(self, init: B, f: F) -> B
fn fold<B, F>(self, init: B, f: F) -> B
1.51.0 (const: unstable) · Source§fn reduce<F>(self, f: F) -> Option<Self::Item>
fn reduce<F>(self, f: F) -> Option<Self::Item>
Source§fn try_reduce<R>(
&mut self,
f: impl FnMut(Self::Item, Self::Item) -> R,
) -> <<R as Try>::Residual as Residual<Option<<R as Try>::Output>>>::TryType
fn try_reduce<R>( &mut self, f: impl FnMut(Self::Item, Self::Item) -> R, ) -> <<R as Try>::Residual as Residual<Option<<R as Try>::Output>>>::TryType
iterator_try_reduce)1.0.0 (const: unstable) · Source§fn all<F>(&mut self, f: F) -> bool
fn all<F>(&mut self, f: F) -> bool
1.0.0 (const: unstable) · Source§fn any<F>(&mut self, f: F) -> bool
fn any<F>(&mut self, f: F) -> bool
1.0.0 (const: unstable) · Source§fn find<P>(&mut self, predicate: P) -> Option<Self::Item>
fn find<P>(&mut self, predicate: P) -> Option<Self::Item>
1.30.0 (const: unstable) · Source§fn find_map<B, F>(&mut self, f: F) -> Option<B>
fn find_map<B, F>(&mut self, f: F) -> Option<B>
Source§fn try_find<R>(
&mut self,
f: impl FnMut(&Self::Item) -> R,
) -> <<R as Try>::Residual as Residual<Option<Self::Item>>>::TryType
fn try_find<R>( &mut self, f: impl FnMut(&Self::Item) -> R, ) -> <<R as Try>::Residual as Residual<Option<Self::Item>>>::TryType
try_find)1.0.0 (const: unstable) · Source§fn position<P>(&mut self, predicate: P) -> Option<usize>
fn position<P>(&mut self, predicate: P) -> Option<usize>
1.0.0 (const: unstable) · Source§fn max(self) -> Option<Self::Item>
fn max(self) -> Option<Self::Item>
1.0.0 (const: unstable) · Source§fn min(self) -> Option<Self::Item>
fn min(self) -> Option<Self::Item>
1.6.0 (const: unstable) · Source§fn max_by_key<B, F>(self, f: F) -> Option<Self::Item>
fn max_by_key<B, F>(self, f: F) -> Option<Self::Item>
1.15.0 (const: unstable) · Source§fn max_by<F>(self, compare: F) -> Option<Self::Item>
fn max_by<F>(self, compare: F) -> Option<Self::Item>
1.6.0 (const: unstable) · Source§fn min_by_key<B, F>(self, f: F) -> Option<Self::Item>
fn min_by_key<B, F>(self, f: F) -> Option<Self::Item>
1.15.0 (const: unstable) · Source§fn min_by<F>(self, compare: F) -> Option<Self::Item>
fn min_by<F>(self, compare: F) -> Option<Self::Item>
1.0.0 (const: unstable) · Source§fn unzip<A, B, FromA, FromB>(self) -> (FromA, FromB)
fn unzip<A, B, FromA, FromB>(self) -> (FromA, FromB)
1.36.0 (const: unstable) · Source§fn copied<'a, T>(self) -> Copied<Self> ⓘ
fn copied<'a, T>(self) -> Copied<Self> ⓘ
Source§fn array_chunks<const N: usize>(self) -> ArrayChunks<Self, N> ⓘwhere
Self: Sized,
fn array_chunks<const N: usize>(self) -> ArrayChunks<Self, N> ⓘwhere
Self: Sized,
iter_array_chunks)N elements of the iterator at a time. Read more1.11.0 (const: unstable) · Source§fn product<P>(self) -> P
fn product<P>(self) -> P
Source§fn cmp_by<I, F>(self, other: I, cmp: F) -> Ordering
fn cmp_by<I, F>(self, other: I, cmp: F) -> Ordering
iter_order_by)Iterator with those
of another with respect to the specified comparison function. Read more1.5.0 (const: unstable) · Source§fn partial_cmp<I>(self, other: I) -> Option<Ordering>
fn partial_cmp<I>(self, other: I) -> Option<Ordering>
PartialOrd elements of
this Iterator with those of another. The comparison works like short-circuit
evaluation, returning a result without comparing the remaining elements.
As soon as an order can be determined, the evaluation stops and a result is returned. Read moreSource§fn partial_cmp_by<I, F>(self, other: I, partial_cmp: F) -> Option<Ordering>where
Self: Sized,
I: IntoIterator,
F: FnMut(Self::Item, <I as IntoIterator>::Item) -> Option<Ordering>,
fn partial_cmp_by<I, F>(self, other: I, partial_cmp: F) -> Option<Ordering>where
Self: Sized,
I: IntoIterator,
F: FnMut(Self::Item, <I as IntoIterator>::Item) -> Option<Ordering>,
iter_order_by)Iterator with those
of another with respect to the specified comparison function. Read moreSource§fn eq_by<I, F>(self, other: I, eq: F) -> bool
fn eq_by<I, F>(self, other: I, eq: F) -> bool
iter_order_by)1.5.0 (const: unstable) · Source§fn lt<I>(self, other: I) -> bool
fn lt<I>(self, other: I) -> bool
Iterator are lexicographically
less than those of another. Read more1.5.0 (const: unstable) · Source§fn le<I>(self, other: I) -> bool
fn le<I>(self, other: I) -> bool
Iterator are lexicographically
less or equal to those of another. Read more1.5.0 (const: unstable) · Source§fn gt<I>(self, other: I) -> bool
fn gt<I>(self, other: I) -> bool
Iterator are lexicographically
greater than those of another. Read more1.5.0 (const: unstable) · Source§fn ge<I>(self, other: I) -> bool
fn ge<I>(self, other: I) -> bool
Iterator are lexicographically
greater than or equal to those of another. Read more