Expand description
Prepare training and inference data with typed datasets and iterable batches.
A Dataset supplies owned samples by index. A sampler chooses
their order, a collator combines them, and DataLoader produces
batches on demand. Samples can be tensors, Rust records, or nested structures;
dataset and pipeline failures are returned to the caller.
The same API is available through rusttorch::data.
That module includes an end-to-end classifier training example; this crate
focuses on preparing its inputs without depending on a model or optimizer.
§Choose a loading pattern
| Input or requirement | API |
|---|---|
| Features and labels already stored in tensors | TensorDataset |
| Files or records addressable by index | Implement Dataset |
| An existing fallible iterator | batches, batches_with_collate |
| Repeated epochs over indexed samples | DataLoader::builder |
| Deterministic randomized sample order | RandomSampler |
| Custom batch padding, packing, or conversion | FnCollate |
| Explicitly partitioned worker streams | StreamDataLoaderBuilder |
| Resume at a recorded iteration boundary | DataLoaderBuilder::resume_from |
§Borrow once or own for repeated epochs
DataLoader::new borrows a dataset and returns an iterator directly. It
keeps samples in a Vec; use DataLoader::with_collate to change that.
DataLoader::builder takes ownership, defaults to recursive tensor
collation, and returns an OwnedDataLoader after build(). Call its
iter() method for each pass and set_epoch(epoch) to advance seeded
shuffling. Neither path collects the entire dataset before yielding a batch.
§Batch paired tensors
Store features and labels with the same first dimension. The default
collator stacks corresponding sample fields: a dataset containing [6, 3]
features and [6] labels yields [2, 3] features and [2] labels when the
batch size is two.
use rusttorch_core::{Device, Kind, Tensor};
use rusttorch_data::{DataLoader, TensorDataset};
let features = Tensor::f_zeros([6, 3], (Kind::Float, Device::Cpu))?;
let labels = Tensor::f_from_slice(&[0_i64, 1, 0, 1, 0, 1])?;
let dataset = TensorDataset::new(vec![features, labels])?;
let mut loader = DataLoader::builder(dataset)
.batch_size(2)
.shuffle(42)?
.build()?;
for epoch in 0..2 {
loader.set_epoch(epoch);
for batch in loader.iter() {
let batch = batch?;
assert_eq!(batch[0].size(), [2, 3]);
assert_eq!(batch[1].size(), [2]);
// batch[0] is the model input; batch[1] contains its target classes.
}
}The loader owns the dataset; call iter() again for another pass. Use
DataLoaderBuilder::drop_last when the model requires fixed-size batches.
Tensor samples share storage with their dataset; avoid modifying views when
the original values are needed for later epochs.
A custom dataset returning (Tensor, i64) instead yields a typed
(Tensor, Tensor) batch, so the training loop can use
let (features, labels) = batch?. See Dataset for a working example.
§Load your own records
Implement Dataset to read a sample from a file, decode an image, or fetch
a record from an indexed source. The associated error type preserves loading
failures. Use VecCollate to keep records as Rust values:
use std::convert::Infallible;
use rusttorch_data::{DataLoader, Dataset, VecCollate};
struct TextRows(Vec<String>);
impl Dataset for TextRows {
type Sample = String;
type Error = Infallible;
fn len(&self) -> usize { self.0.len() }
fn get(&self, index: usize) -> Result<String, Infallible> {
Ok(self.0[index].clone())
}
}
let dataset = TextRows(vec!["first".into(), "second".into(), "third".into()]);
let mut loader = DataLoader::builder(dataset)
.batch_size(2)
.collate(VecCollate)
.build()?;
let batches = loader.iter().collect::<Result<Vec<_>, _>>()?;
assert_eq!(batches, [vec!["first", "second"], vec!["third"]]);§Customize batching and overlap preparation
Start with serial loading, then use DataLoaderBuilder::workers to
overlap sample preparation when your dataset satisfies the worker ownership
requirements. A positive worker count shares the dataset through Arc;
each worker fetches and transforms samples, and the calling thread collates
them. DataLoaderBuilder::prefetch_factor bounds outstanding batches per
worker. A dataset of file paths or ordinary Rust records can create new
tensors during get; TensorDataset itself uses the serial path.
DataLoaderBuilder demonstrates variable-length padding with FnCollate
and inference without an extra batch dimension using without_batching().
StreamDataLoaderBuilder demonstrates lazy, independently owned shards
when records do not have random-access indices. All shards are merged
before batching, so drop_last drops at most one global tail.
§Resume a training job
Exact resumption requires replay-safe data and checkpointable pipeline
components. LoaderState contains a complete map-loader save/restore
example; ExactStreamDataLoader shows resumable stream cursors. Save at a
successful batch boundary together with the corresponding model, optimizer
and training counters. Loader checkpoints store the input position and
component state, not model weights or decoded samples.
Re-exports§
pub use checkpoint::CheckpointBuildError;pub use checkpoint::CheckpointDisabled;pub use checkpoint::CheckpointFresh;pub use checkpoint::CheckpointPinRequest;pub use checkpoint::CheckpointPinStatus;pub use checkpoint::CheckpointResume;pub use checkpoint::CheckpointSourceFactory;pub use checkpoint::Checkpointable;pub use checkpoint::CheckpointableSource;pub use checkpoint::DatasetCheckpoint;pub use checkpoint::LOADER_STATE_SCHEMA_VERSION;pub use checkpoint::LoaderConfiguration;pub use checkpoint::LoaderState;pub use checkpoint::ReplaySafeDataset;pub use checkpoint::Stateless;pub use checkpoint::StatelessWorker;pub use checkpoint::StreamCheckpointBuildError;pub use checkpoint::StreamCheckpointConfiguration;pub use checkpoint::StreamLaneState;pub use checkpoint::StreamLoaderState;pub use checkpoint::TransactionalCheckpoint;pub use checkpoint::TransactionalWorker;pub use checkpoint::WorkerCheckpoint;pub use checkpoint::WorkerLaneState;pub use checkpoint::WorkerTransformLanes;pub use checkpoint::WorkerTransformState;pub use loader::AutoBatch;pub use loader::DataLoaderBuilder;pub use loader::ExplicitBatches;pub use loader::LoaderIter;pub use loader::NoBatch;pub use loader::OwnedDataLoader;pub use loader::SerialExecution;pub use loader::WorkerExecution;pub use loader::WorkerLoaderIter;pub use sampler::BatchSampler;pub use sampler::BatchSamplerState;pub use sampler::BatchSource;pub use sampler::DistributedSampler;pub use sampler::DistributedSamplerState;pub use sampler::FnBatchSource;pub use sampler::FnSampler;pub use sampler::RandomReplacement;pub use sampler::RandomSampler;pub use sampler::RandomSamplerState;pub use sampler::Sampler;pub use sampler::SequentialSampler;pub use sampler::SequentialSamplerState;pub use sampler::SubsetRandomSampler;pub use sampler::SubsetRandomSamplerState;pub use sampler::WeightedRandomSampler;pub use sampler::WeightedRandomSamplerState;pub use worker_context::CancellationToken;pub use worker_context::Deadline;pub use worker_context::FnWorkerInit;pub use worker_context::LoaderCancelled;pub use worker_context::NoWorkerInit;pub use worker_context::WORKER_SEED_DERIVATION_VERSION;pub use worker_context::WaitOutcome;pub use worker_context::WorkerContext;pub use worker_context::WorkerInfo;pub use worker_context::WorkerInit;pub use worker_context::get_worker_info;
Structs§
- Auto
- Automatic pinning mode selecting CUDA device zero when available.
- Auto
Batch - Automatic fixed-size batching over a reusable sampler.
- Batch
Sampler - Groups sampler indices into fixed-size batches.
- Batch
Sampler State - Serialized cursor for a compositional
BatchSampler. - Bytes
- A byte-string record that is not treated as a recursive numeric sequence.
- Cancellation
Token - Cloneable cooperative cancellation notification.
- Checkpoint
Disabled - Default builder/loader state with exact checkpointing disabled.
- Checkpoint
Fresh - Builder state after a dataset identity enables a fresh exact checkpoint.
- Checkpoint
Resume - Builder state carrying a typed checkpoint selected for resume.
- Clone
Transform Factory - Factory used by
crate::DataLoaderBuilder::transformto clone a template. - Concat
Dataset - A dataset that maps one global index across consecutive child datasets.
- Data
Loader - Load indexed samples through a borrowed iterator or a reusable owned loader.
- Data
Loader Builder - Configure repeated passes over an owned indexed dataset.
- Dataset
Samples - A borrowing, sequential iterator over a
Dataset. - Deadline
- Cloneable monotonic deadline visible to cooperative worker code.
- Default
Collator - Applies
DefaultCollateto an owned group of samples. - Default
Converter - Applies
DefaultConvertin no-auto-batching mode. - Distributed
Sampler - Deterministically shards a finite index range across distributed ranks.
- Distributed
Sampler State - Serialized configuration and cursor for
DistributedSampler. - Exact
Stream Data Loader - A reusable sharded stream whose unread position can survive a restart.
- Exact
Stream Loader Iter - A resumable pass over an
ExactStreamDataLoader. - Explicit
- Explicit pinning mode validated against an available CUDA device.
- Explicit
Batches - Explicit reusable batches of indices.
- FnBatch
Source - Adapts an epoch-aware batch-iterator factory into a reusable
BatchSource. - FnCollate
- Adapts an ordinary fallible closure to
Collate. - FnSampler
- Adapts an epoch-aware iterator factory into a reusable
Sampler. - FnTransform
- Adapts a fallible closure into a
Transform. - FnTransform
Factory - Adapts a fallible closure into a
TransformFactory. - FnWorker
Init - Adapts a fallible closure into a
WorkerInit. - Identity
Transform - Identity sample transform used by the default factory.
- Identity
Transform Factory - Default factory that creates infallible identity transforms.
- Loader
Cancelled - Error returned by
WorkerContext::checkafter cancellation or expiry. - Loader
Configuration - Loader settings that must match when a serial checkpoint is resumed.
- Loader
Iter - One pass over an
OwnedDataLoader, yielding a typed result for each batch. - Loader
State - Serializable input position and pipeline state for an exact map-loader restart.
- Logical
Sample Id - Stable sample identity used by task randomness and stream checkpoints.
- Memory
Disabled - Builder state for item-bounded loading without byte accounting.
- Memory
Enabled - Builder state for post-transform byte-bounded prefetch.
- NoBatch
- One-sample loading without an automatic batch dimension.
- NoWorker
Init - Default worker initializer that performs no work.
- Owned
Data Loader - A reusable dataset pipeline, created by
crate::DataLoader::builder. - PinDisabled
- Builder state for loading without recursive batch pinning.
- PinEnabled
- Builder state for enabled recursive batch pinning.
- Random
Sampler - A seeded sampler over a finite index range.
- Random
Sampler State - Serialized configuration and cursor for
RandomSampler. - Replay
Safe Map - Explicit exact-checkpoint adapter for a replay-safe map dataset.
- Replay
Safe Tensor Dataset - Tensor dataset with private backing storage and independently owned rows.
- Resource
Limits - Resource ceilings applied before allocation whenever metadata permits.
- Sequence
Id - A record’s position in one ordered global stream generation.
- Sequential
Sampler - An allocation-free sampler that yields indices in ascending order.
- Sequential
Sampler State - Serialized cursor for
SequentialSampler. - Serial
Execution - Default execution capability preserving fully local serial data types.
- Stack
Dataset - A dataset that returns aligned samples from a tuple of child datasets.
- Stateless
Worker - Explicit unit-state adapter for a transform the caller asserts is stateless.
- Stream
Checkpoint Configuration - Exact streaming configuration, with no sampler state.
- Stream
Data Loader - Reusable owner of a sharded input pipeline.
- Stream
Data Loader Builder - Merge lazy worker-owned streams into batches for training or inference.
- Stream
Lane State - Paired source and transform boundary for one worker shard.
- Stream
Loader Iter - One pass over a
StreamDataLoader, yielding globally collated batches. - Stream
Loader State - Serializable input position and component state for resuming a sharded stream.
- Subset
- A dataset containing selected indices from another dataset.
- Subset
Random Sampler - A seeded sampler that shuffles a caller-provided subset of indices.
- Subset
Random Sampler State - Serialized configuration and cursor for
SubsetRandomSampler. - Task
Context - Logical identity used to seed one transformation occurrence.
- Tensor
Dataset - A map-style dataset backed by tensors with a shared first dimension.
- Transactional
Map - Explicit serial exact-checkpoint adapter for a transactional dataset.
- Transactional
Worker - Explicit transactional adapter for a stateful transform.
- VecCollate
- Moves samples unchanged into their existing vector.
- Weighted
Random Sampler - A seeded sampler that draws indices according to nonnegative weights.
- Weighted
Random Sampler State - Serialized configuration and cursor for
WeightedRandomSampler. - Worker
Context - Cancellation and deadline state passed to worker lifecycle callbacks.
- Worker
Execution - Execution capability selected by
DataLoaderBuilder::workers. - Worker
Info - Stable information visible while a loader worker callback is active.
- Worker
Lane State - One deterministic worker lane’s committed transform state.
- Worker
Loader Iter - A pass over an owned loader configured through
DataLoaderBuilder::workers. - Worker
Record - An owned sample plus its ordering and reproducibility identifiers.
- Worker
Transform State - Versioned worker replay envelope within schema-v1 loader state.
Enums§
- Checkpoint
Build Error - Resume-build error preserving either a RustTorch validation error or the transform factory’s original error value.
- Checkpoint
PinRequest - Pinning request recorded in a loader checkpoint.
- Checkpoint
PinStatus - Effective pinning behavior recorded in a loader checkpoint.
- Collate
Error - An error produced by built-in collation or conversion.
- Loader
Error - A typed failure produced while an owned data loader is active.
- PinMemory
Status - Effective recursive pinning behavior for a built loader.
- Pipeline
Error - A preserved failure from one typed data-pipeline stage.
- Random
Replacement - Replacement mode stored by
RandomSamplerState. - Split
Length - One requested output length for
random_split. - Stream
Checkpoint Build Error - Typed stream construction or transactional resume failure.
- Wait
Outcome - Outcome of waiting for cooperative cancellation or a deadline.
- Worker
Transform Lanes - Explicit zero-worker or positive-worker transform state.
Constants§
- LOADER_
STATE_ SCHEMA_ VERSION - Schema version emitted by serial map-loader checkpoints.
- TASK_
RNG_ DERIVATION_ VERSION - Version of RustTorch’s deterministic task-RNG seed derivation.
- WORKER_
SEED_ DERIVATION_ VERSION - Version of RustTorch’s deterministic worker-seed derivation.
Traits§
- Batch
Source - A reusable, epoch-aware source of finite index batches.
- Checkpoint
Source Factory - Stable factory identity for explicitly checkpointable stream sources.
- Checkpointable
- A component whose state can be validated before it is restored.
- Checkpointable
Source - A stream reader whose next unread position can be saved and restored exactly.
- Collate
- Converts an owned group of samples into one batch.
- Dataset
- A finite, indexable collection of samples.
- Dataset
Checkpoint - Dataset state used by exact serial loader checkpoints.
- Default
Collate - Recursively converts one Rust sample type into its default batch type.
- Default
Convert - Recursively converts one sample without adding a batch dimension.
- Memory
Footprint - Conservative logical payload bytes retained by one prefetched value.
- PinMemory
- Recursively moves a value into host memory pinned for
device. - Replay
Safe Dataset - Promise that fetching an index again from fixed dataset contents is safe.
- Sampler
- A reusable, epoch-aware source of finite sample indices.
- Stack
Tuple - A sealed tuple shape accepted by
StackDataset. - Stateless
- Marker for a component with no mutable checkpoint state.
- Transactional
Checkpoint - Transactional state contract for a serial component.
- Transform
- A fallible sample transformation with task-local deterministic context.
- Transform
Factory - Constructs one transform for a serial iterator or worker.
- Worker
Checkpoint - Checkpoint contract for one transform instance.
- Worker
Init - Initializes one worker after its context becomes active.
- Worker
Source Factory - Open an independent input shard for each worker and each iteration.
Functions§
- batches
- Batches an ordinary fallible sample iterator into vectors.
- batches_
with_ collate - Batches an ordinary fallible sample iterator with custom collation.
- chain_
datasets - Flattens datasets or other iterables using the standard iterator adapter.
- get_
worker_ info - Returns worker information for the active callback, if any.
- random_
split - Randomly partitions a shared dataset into non-overlapping subsets.