Skip to main content

Crate rusttorch_data

Crate rusttorch_data 

Source
Expand description

Prepare training and inference data with typed datasets and iterable batches.

A Dataset supplies owned samples by index. A sampler chooses their order, a collator combines them, and DataLoader produces batches on demand. Samples can be tensors, Rust records, or nested structures; dataset and pipeline failures are returned to the caller.

The same API is available through rusttorch::data. That module includes an end-to-end classifier training example; this crate focuses on preparing its inputs without depending on a model or optimizer.

§Choose a loading pattern

Input or requirementAPI
Features and labels already stored in tensorsTensorDataset
Files or records addressable by indexImplement Dataset
An existing fallible iteratorbatches, batches_with_collate
Repeated epochs over indexed samplesDataLoader::builder
Deterministic randomized sample orderRandomSampler
Custom batch padding, packing, or conversionFnCollate
Explicitly partitioned worker streamsStreamDataLoaderBuilder
Resume at a recorded iteration boundaryDataLoaderBuilder::resume_from

§Borrow once or own for repeated epochs

DataLoader::new borrows a dataset and returns an iterator directly. It keeps samples in a Vec; use DataLoader::with_collate to change that. DataLoader::builder takes ownership, defaults to recursive tensor collation, and returns an OwnedDataLoader after build(). Call its iter() method for each pass and set_epoch(epoch) to advance seeded shuffling. Neither path collects the entire dataset before yielding a batch.

§Batch paired tensors

Store features and labels with the same first dimension. The default collator stacks corresponding sample fields: a dataset containing [6, 3] features and [6] labels yields [2, 3] features and [2] labels when the batch size is two.

use rusttorch_core::{Device, Kind, Tensor};
use rusttorch_data::{DataLoader, TensorDataset};

let features = Tensor::f_zeros([6, 3], (Kind::Float, Device::Cpu))?;
let labels = Tensor::f_from_slice(&[0_i64, 1, 0, 1, 0, 1])?;
let dataset = TensorDataset::new(vec![features, labels])?;
let mut loader = DataLoader::builder(dataset)
    .batch_size(2)
    .shuffle(42)?
    .build()?;

for epoch in 0..2 {
    loader.set_epoch(epoch);
    for batch in loader.iter() {
        let batch = batch?;
        assert_eq!(batch[0].size(), [2, 3]);
        assert_eq!(batch[1].size(), [2]);
        // batch[0] is the model input; batch[1] contains its target classes.
    }
}

The loader owns the dataset; call iter() again for another pass. Use DataLoaderBuilder::drop_last when the model requires fixed-size batches. Tensor samples share storage with their dataset; avoid modifying views when the original values are needed for later epochs.

A custom dataset returning (Tensor, i64) instead yields a typed (Tensor, Tensor) batch, so the training loop can use let (features, labels) = batch?. See Dataset for a working example.

§Load your own records

Implement Dataset to read a sample from a file, decode an image, or fetch a record from an indexed source. The associated error type preserves loading failures. Use VecCollate to keep records as Rust values:

use std::convert::Infallible;
use rusttorch_data::{DataLoader, Dataset, VecCollate};

struct TextRows(Vec<String>);

impl Dataset for TextRows {
    type Sample = String;
    type Error = Infallible;

    fn len(&self) -> usize { self.0.len() }
    fn get(&self, index: usize) -> Result<String, Infallible> {
        Ok(self.0[index].clone())
    }
}

let dataset = TextRows(vec!["first".into(), "second".into(), "third".into()]);
let mut loader = DataLoader::builder(dataset)
    .batch_size(2)
    .collate(VecCollate)
    .build()?;
let batches = loader.iter().collect::<Result<Vec<_>, _>>()?;
assert_eq!(batches, [vec!["first", "second"], vec!["third"]]);

§Customize batching and overlap preparation

Start with serial loading, then use DataLoaderBuilder::workers to overlap sample preparation when your dataset satisfies the worker ownership requirements. A positive worker count shares the dataset through Arc; each worker fetches and transforms samples, and the calling thread collates them. DataLoaderBuilder::prefetch_factor bounds outstanding batches per worker. A dataset of file paths or ordinary Rust records can create new tensors during get; TensorDataset itself uses the serial path.

DataLoaderBuilder demonstrates variable-length padding with FnCollate and inference without an extra batch dimension using without_batching(). StreamDataLoaderBuilder demonstrates lazy, independently owned shards when records do not have random-access indices. All shards are merged before batching, so drop_last drops at most one global tail.

§Resume a training job

Exact resumption requires replay-safe data and checkpointable pipeline components. LoaderState contains a complete map-loader save/restore example; ExactStreamDataLoader shows resumable stream cursors. Save at a successful batch boundary together with the corresponding model, optimizer and training counters. Loader checkpoints store the input position and component state, not model weights or decoded samples.

Structs§

Auto
Automatic pinning mode selecting CUDA device zero when available.
AutoBatch
Automatic fixed-size batching over a reusable sampler.
BatchSampler
Groups sampler indices into fixed-size batches.
BatchSamplerState
Serialized cursor for a compositional BatchSampler.
Bytes
A byte-string record that is not treated as a recursive numeric sequence.
CancellationToken
Cloneable cooperative cancellation notification.
CheckpointDisabled
Default builder/loader state with exact checkpointing disabled.
CheckpointFresh
Builder state after a dataset identity enables a fresh exact checkpoint.
CheckpointResume
Builder state carrying a typed checkpoint selected for resume.
CloneTransformFactory
Factory used by crate::DataLoaderBuilder::transform to clone a template.
ConcatDataset
A dataset that maps one global index across consecutive child datasets.
DataLoader
Load indexed samples through a borrowed iterator or a reusable owned loader.
DataLoaderBuilder
Configure repeated passes over an owned indexed dataset.
DatasetSamples
A borrowing, sequential iterator over a Dataset.
Deadline
Cloneable monotonic deadline visible to cooperative worker code.
DefaultCollator
Applies DefaultCollate to an owned group of samples.
DefaultConverter
Applies DefaultConvert in no-auto-batching mode.
DistributedSampler
Deterministically shards a finite index range across distributed ranks.
DistributedSamplerState
Serialized configuration and cursor for DistributedSampler.
ExactStreamDataLoader
A reusable sharded stream whose unread position can survive a restart.
ExactStreamLoaderIter
A resumable pass over an ExactStreamDataLoader.
Explicit
Explicit pinning mode validated against an available CUDA device.
ExplicitBatches
Explicit reusable batches of indices.
FnBatchSource
Adapts an epoch-aware batch-iterator factory into a reusable BatchSource.
FnCollate
Adapts an ordinary fallible closure to Collate.
FnSampler
Adapts an epoch-aware iterator factory into a reusable Sampler.
FnTransform
Adapts a fallible closure into a Transform.
FnTransformFactory
Adapts a fallible closure into a TransformFactory.
FnWorkerInit
Adapts a fallible closure into a WorkerInit.
IdentityTransform
Identity sample transform used by the default factory.
IdentityTransformFactory
Default factory that creates infallible identity transforms.
LoaderCancelled
Error returned by WorkerContext::check after cancellation or expiry.
LoaderConfiguration
Loader settings that must match when a serial checkpoint is resumed.
LoaderIter
One pass over an OwnedDataLoader, yielding a typed result for each batch.
LoaderState
Serializable input position and pipeline state for an exact map-loader restart.
LogicalSampleId
Stable sample identity used by task randomness and stream checkpoints.
MemoryDisabled
Builder state for item-bounded loading without byte accounting.
MemoryEnabled
Builder state for post-transform byte-bounded prefetch.
NoBatch
One-sample loading without an automatic batch dimension.
NoWorkerInit
Default worker initializer that performs no work.
OwnedDataLoader
A reusable dataset pipeline, created by crate::DataLoader::builder.
PinDisabled
Builder state for loading without recursive batch pinning.
PinEnabled
Builder state for enabled recursive batch pinning.
RandomSampler
A seeded sampler over a finite index range.
RandomSamplerState
Serialized configuration and cursor for RandomSampler.
ReplaySafeMap
Explicit exact-checkpoint adapter for a replay-safe map dataset.
ReplaySafeTensorDataset
Tensor dataset with private backing storage and independently owned rows.
SequenceId
A record’s position in one ordered global stream generation.
SequentialSampler
An allocation-free sampler that yields indices in ascending order.
SequentialSamplerState
Serialized cursor for SequentialSampler.
SerialExecution
Default execution capability preserving fully local serial data types.
StackDataset
A dataset that returns aligned samples from a tuple of child datasets.
StatelessWorker
Explicit unit-state adapter for a transform the caller asserts is stateless.
StreamCheckpointConfiguration
Exact streaming configuration, with no sampler state.
StreamDataLoader
Reusable owner of a sharded input pipeline.
StreamDataLoaderBuilder
Merge lazy worker-owned streams into batches for training or inference.
StreamLaneState
Paired source and transform boundary for one worker shard.
StreamLoaderIter
One pass over a StreamDataLoader, yielding globally collated batches.
StreamLoaderState
Serializable input position and component state for resuming a sharded stream.
Subset
A dataset containing selected indices from another dataset.
SubsetRandomSampler
A seeded sampler that shuffles a caller-provided subset of indices.
SubsetRandomSamplerState
Serialized configuration and cursor for SubsetRandomSampler.
TaskContext
Logical identity used to seed one transformation occurrence.
TensorDataset
A map-style dataset backed by tensors with a shared first dimension.
TransactionalMap
Explicit serial exact-checkpoint adapter for a transactional dataset.
TransactionalWorker
Explicit transactional adapter for a stateful transform.
VecCollate
Moves samples unchanged into their existing vector.
WeightedRandomSampler
A seeded sampler that draws indices according to nonnegative weights.
WeightedRandomSamplerState
Serialized configuration and cursor for WeightedRandomSampler.
WorkerContext
Cancellation and deadline state passed to worker lifecycle callbacks.
WorkerExecution
Execution capability selected by DataLoaderBuilder::workers.
WorkerInfo
Stable information visible while a loader worker callback is active.
WorkerLaneState
One deterministic worker lane’s committed transform state.
WorkerLoaderIter
A pass over an owned loader configured through DataLoaderBuilder::workers.
WorkerRecord
An owned sample plus its ordering and reproducibility identifiers.
WorkerTransformState
Versioned worker replay envelope within schema-v1 loader state.

Enums§

CheckpointBuildError
Resume-build error preserving either a RustTorch validation error or the transform factory’s original error value.
CheckpointPinRequest
Pinning request recorded in a loader checkpoint.
CheckpointPinStatus
Effective pinning behavior recorded in a loader checkpoint.
CollateError
An error produced by built-in collation or conversion.
LoaderError
A typed failure produced while an owned data loader is active.
PinMemoryStatus
Effective recursive pinning behavior for a built loader.
PipelineError
A preserved failure from one typed data-pipeline stage.
RandomReplacement
Replacement mode stored by RandomSamplerState.
SplitLength
One requested output length for random_split.
StreamCheckpointBuildError
Typed stream construction or transactional resume failure.
WaitOutcome
Outcome of waiting for cooperative cancellation or a deadline.
WorkerTransformLanes
Explicit zero-worker or positive-worker transform state.

Constants§

LOADER_STATE_SCHEMA_VERSION
Schema version emitted by serial map-loader checkpoints.
TASK_RNG_DERIVATION_VERSION
Version of RustTorch’s deterministic task-RNG seed derivation.
WORKER_SEED_DERIVATION_VERSION
Version of RustTorch’s deterministic worker-seed derivation.

Traits§

BatchSource
A reusable, epoch-aware source of finite index batches.
CheckpointSourceFactory
Stable factory identity for explicitly checkpointable stream sources.
Checkpointable
A component whose state can be validated before it is restored.
CheckpointableSource
A stream reader whose next unread position can be saved and restored exactly.
Collate
Converts an owned group of samples into one batch.
Dataset
A finite, indexable collection of samples.
DatasetCheckpoint
Dataset state used by exact serial loader checkpoints.
DefaultCollate
Recursively converts one Rust sample type into its default batch type.
DefaultConvert
Recursively converts one sample without adding a batch dimension.
MemoryFootprint
Conservative logical payload bytes retained by one prefetched value.
PinMemory
Recursively moves a value into host memory pinned for device.
ReplaySafeDataset
Promise that fetching an index again from fixed dataset contents is safe.
Sampler
A reusable, epoch-aware source of finite sample indices.
StackTuple
A sealed tuple shape accepted by StackDataset.
Stateless
Marker for a component with no mutable checkpoint state.
TransactionalCheckpoint
Transactional state contract for a serial component.
Transform
A fallible sample transformation with task-local deterministic context.
TransformFactory
Constructs one transform for a serial iterator or worker.
WorkerCheckpoint
Checkpoint contract for one transform instance.
WorkerInit
Initializes one worker after its context becomes active.
WorkerSourceFactory
Open an independent input shard for each worker and each iteration.

Functions§

batches
Batches an ordinary fallible sample iterator into vectors.
batches_with_collate
Batches an ordinary fallible sample iterator with custom collation.
chain_datasets
Flattens datasets or other iterables using the standard iterator adapter.
get_worker_info
Returns worker information for the active callback, if any.
random_split
Randomly partitions a shared dataset into non-overlapping subsets.