Module distributed_loading 

Distributed dataset loading

Shard-aware loading for distributed training with multi-node coordination and distributed caching. Distributed dataset loading

This module provides shard-aware dataset loading for distributed training, with multi-node coordination, distributed caching, and rank-aware data partitioning to ensure each worker processes a unique subset of the data.

Structs

DistributedCache

Distributed cache for sharing data across nodes

DistributedConfig

Configuration for distributed loading

DistributedLoader

Distributed dataset loader

Shard

Shard information

Functions

create_loader

Create a distributed loader for a dataset

create_loader_with_config

Create a distributed loader with custom configuration