Struct ShardedLoader 

pub struct ShardedLoader {
    pub total_samples: usize,
    pub n_shards: usize,
    pub seed: u64,
}

A loader that partitions a dataset into consistently shuffled shards for multi-process or multi-node distributed training.

Given the same seed, total_samples, and n_shards, every call to ShardedLoader::get_shard with the same arguments will return the same DatasetShard, making the assignment deterministic and reproducible across independent processes.

Example

use scirs2_datasets::ShardedLoader;

let loader = ShardedLoader::new(100, 4, 42);
assert!(loader.verify_coverage());

let shard0 = loader.get_shard(0);
let shard1 = loader.get_shard(1);
// No overlap between shards.
for &i in &shard0.indices {
    assert!(!shard1.indices.contains(&i));
}

Fields

total_samples: usize

Total number of samples in the dataset.

n_shards: usize

Number of shards to partition into.

seed: u64

Seed used for the deterministic shuffle.

Implementations

impl ShardedLoader

pub fn new(total_samples: usize, n_shards: usize, seed: u64) -> Self

Create a new ShardedLoader.

• total_samples — number of samples in the full dataset.
• n_shards — number of shards to divide the dataset into.
• seed — seed for the deterministic Fisher-Yates shuffle.

pub fn global_permutation(&self) -> Vec<usize>

Compute the global shuffled permutation of all sample indices.

Calling this function with the same seed always returns the same permutation, regardless of which process calls it.

pub fn get_shard(&self, shard_id: usize) -> DatasetShard

Return shard shard_id (0-indexed) from the consistently shuffled partition of the dataset.

If shard_id >= self.n_shards an empty shard is returned.

pub fn verify_coverage(&self) -> bool

Verify that the union of all shard index sets covers every sample index exactly once (no gaps, no duplicates).

Returns true when coverage is complete and disjoint.

Trait Implementations

impl Clone for ShardedLoader

fn clone(&self) -> ShardedLoader

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ShardedLoader

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.