Struct DataLoader 

pub struct DataLoader<D: Dataset> { /* private fields */ }

A data loader that provides batched iteration over a dataset.

The DataLoader wraps a dataset and provides:

• Configurable batch sizes
• Optional shuffling with reproducible seeds
• Option to drop incomplete final batches

Example

use alimentar::{ArrowDataset, DataLoader};

let dataset = ArrowDataset::from_parquet("data.parquet").unwrap();
let loader = DataLoader::new(dataset)
    .batch_size(32)
    .shuffle(true)
    .seed(42);

for batch in loader {
    println!("Processing batch with {} rows", batch.num_rows());
}

Implementations

impl<D: Dataset> DataLoader<D>

pub fn new(dataset: D) -> Self

Creates a new DataLoader wrapping the given dataset.

Default configuration:

• batch_size: 1
• shuffle: false
• drop_last: false
• seed: None (random)

pub fn batch_size(self, size: usize) -> Self

Sets the batch size.

Each iteration will yield a RecordBatch with at most this many rows. The final batch may have fewer rows unless drop_last is set.

#[requires(true)] #[ensures(result.batch_size >= 1)] #[ensures(size >= 1 ==> result.batch_size == size)] #[ensures(size == 0 ==> result.batch_size == 1)]

pub fn shuffle(self, shuffle: bool) -> Self

Enables or disables shuffling.

When enabled, the row order is randomized before each epoch. Requires the shuffle feature.

pub fn drop_last(self, drop_last: bool) -> Self

Sets whether to drop the last incomplete batch.

When true, if the dataset size is not evenly divisible by the batch size, the final partial batch is skipped.

pub fn seed(self, seed: u64) -> Self

Sets the random seed for shuffling.

Setting a seed makes shuffling deterministic and reproducible. Requires the shuffle feature.

pub fn get_batch_size(&self) -> usize

Returns the configured batch size.

pub fn is_shuffle(&self) -> bool

Returns whether shuffling is enabled.

pub fn is_drop_last(&self) -> bool

Returns whether drop_last is enabled.

pub fn num_batches(&self) -> usize

Returns the number of batches that will be yielded.

pub fn len(&self) -> usize

Returns the total number of rows in the underlying dataset.

pub fn is_empty(&self) -> bool

Returns true if the dataset is empty.

Trait Implementations

impl<D: Debug + Dataset> Debug for DataLoader<D>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<D: Dataset> IntoIterator for DataLoader<D>

type Item = RecordBatch

The type of the elements being iterated over.

type IntoIter = DataLoaderIterator<D>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.