Struct WeightedDataLoader 

pub struct WeightedDataLoader<D: Dataset> { /* private fields */ }

A data loader that samples with per-sample weights.

Unlike DataLoader which samples uniformly, WeightedDataLoader samples proportional to the provided weights. This is useful for:

• Importance sampling in imbalanced datasets
• CITL reweighting (--reweight 1.5 for compiler-verified labels)
• Curriculum learning with difficulty-based sampling

Example

use alimentar::{ArrowDataset, Dataset, WeightedDataLoader};

let dataset = ArrowDataset::from_parquet("data.parquet").unwrap();
let weights = vec![1.0; dataset.len()]; // Uniform weights

let loader = WeightedDataLoader::new(dataset, weights)
    .unwrap()
    .batch_size(32)
    .seed(42);

for batch in loader {
    println!("Batch with {} rows", batch.num_rows());
}

Implementations

impl<D: Dataset> WeightedDataLoader<D>

pub fn new(dataset: D, weights: Vec<f32>) -> Result<Self>

Creates a new weighted data loader.

Arguments

• dataset - The dataset to sample from
• weights - Per-sample weights (must match dataset length)

Errors

Returns an error if weights length doesn’t match dataset length, or if any weight is negative.

pub fn with_reweight(dataset: D, reweight: f32) -> Result<Self>

Creates a weighted loader with a uniform reweight factor.

Multiplies all weights by the given factor. Useful for CITL’s --reweight 1.5 which boosts compiler-verified samples.

Arguments

• dataset - The dataset to sample from
• reweight - Factor to multiply all weights by

pub fn batch_size(self, size: usize) -> Self

Sets the batch size.

pub fn num_samples(self, n: usize) -> Self

Sets the total number of samples per epoch.

By default, samples len() items per epoch. Set this to oversample or undersample the dataset.

pub fn drop_last(self, drop_last: bool) -> Self

Sets whether to drop the last incomplete batch.

pub fn seed(self, seed: u64) -> Self

Sets the random seed for reproducibility.

pub fn get_batch_size(&self) -> usize

Returns the configured batch size.

pub fn get_num_samples(&self) -> usize

Returns the number of samples per epoch.

pub fn weights(&self) -> &[f32]

Returns the weights.

pub fn num_batches(&self) -> usize

Returns the number of batches that will be yielded.

pub fn len(&self) -> usize

Returns the dataset length.

pub fn is_empty(&self) -> bool

Returns true if the dataset is empty.

Trait Implementations

impl<D: Debug + Dataset> Debug for WeightedDataLoader<D>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<D: Dataset> IntoIterator for WeightedDataLoader<D>

Available on crate feature shuffle only.

type Item = RecordBatch

The type of the elements being iterated over.

type IntoIter = WeightedDataLoaderIterator<D>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.