Skip to main content

Crate rusttorch_data

Crate rusttorch_data 

Source
Expand description

Typed datasets, samplers, batching, and loading for RustTorch.

§rusttorch-data compatibility

This page is generated from compat/pytorch_api.toml against PyTorch v2.13.0 at commit cf30153, through tch 0.26.0. Each entry is independently scoped. Supported applies only to its written scope; row counts are not a compatibility score.

§Supported

§data.batched_fetch

§data.batches

§data.chain_dataset

§data.concat_dataset

§data.dataset

  • PyTorch: torch.utils.data.Dataset
  • RustTorch: rusttorch_data::Dataset, rusttorch_data::DatasetSamples, rusttorch::data::Dataset, rusttorch::data::DatasetSamples
  • Implementation: Implemented by RustTorch
  • Scope: Finite map-style datasets with owned samples, fallible indexed access, length and empty queries, and a lazy borrowing iterator over samples in index order.
  • Pinned source: torch/utils/data/dataset.py
  • Evidence: tests/data.rs::dataset_default_is_empty_follows_len, tests/data.rs::dataset_samples_borrow_and_fetch_lazily_in_order
  • Notes: rusttorch-data owns the direct surface and rusttorch::data preserves it. DatasetSamples performs no dataset clone or precollection; format-specific decoding belongs in optional packages.

§data.loader

§data.loader.checkpoint_stream

§data.random_split

§data.sampler

§data.sampler.distributed

§data.stack_dataset

§data.subset

§data.tensor_dataset

§Partial

§data.collation

§data.dataset.datapipes

  • PyTorch: torch.utils.data.DataChunk, torch.utils.data.DFIterDataPipe, torch.utils.data.IterDataPipe, torch.utils.data.MapDataPipe
  • RustTorch: std::iter::Iterator, rusttorch_data::Dataset, rusttorch_data::WorkerSourceFactory, rusttorch_data::Transform, rusttorch::data::Dataset, rusttorch::data::WorkerSourceFactory, rusttorch::data::Transform
  • Implementation: Implemented by RustTorch
  • Scope: Ordinary typed Rust iterators replace basic iterable DataPipe traversal, Dataset replaces indexed MapDataPipe access, WorkerSourceFactory creates independent sharded iterators, and Transform composes typed sample operations.
  • Pinned source: torch/utils/data/datapipes/datapipe.py
  • Evidence: crates/rusttorch-data/tests/stream_workers.rs::four_explicit_modulo_shards_produce_each_logical_record_once, tests/data.rs::dataset_samples_borrow_and_fetch_lazily_in_order
  • Notes: RustTorch does not reproduce functional-name registration, reset/single-iterator hooks, Python serialization hooks, dataframe tracing, or DFIterDataPipe. Vec is the closest DataChunk container but does not retain a separate original-items raw iterator, so DataChunk is not claimed as supported.

§data.loader.checkpoint

§data.loader.pinning

§data.loader.prefetch

§data.loader.transforms

§data.loader.workers

§data.stream_workers

§Planned

No entries.

§Python-only

§data.dataset.datapipes.decorators

  • PyTorch: torch.utils.data.argument_validation, torch.utils.data.functional_datapipe, torch.utils.data.guaranteed_datapipes_determinism, torch.utils.data.non_deterministic, torch.utils.data.runtime_validation, torch.utils.data.runtime_validation_disabled
  • RustTorch: —
  • Implementation: Not implemented
  • Scope: Python decorators dynamically register DataPipe functions, label determinism, and toggle runtime validation or construction-time determinism contexts.
  • Pinned source: torch/utils/data/datapipes/_decorator.py
  • Evidence: —
  • Notes: Rust traits and the type system provide compile-time composition and validation but are not semantic implementations of these mutable Python registries and context managers.

§data.internal.exports

  • PyTorch: torch.utils.data._DatasetKind
  • RustTorch: —
  • Implementation: Not implemented
  • Scope: The private loader-kind dispatch enum exported by the pinned module remains a Python implementation detail.
  • Pinned source: torch/utils/data/dataloader.py
  • Evidence: —
  • Notes: RustTorch uses type-state builders instead of exposing Python’s internal enum. argument_validation is separately attributed to the pinned DataPipe decorator module.

§Not supported

No entries.

Structs§

Auto
Automatic pinning mode selecting CUDA device zero when available.
AutoBatch
Automatic fixed-size batching over a reusable sampler.
BatchSampler
Groups sampler indices into fixed-size batches.
BatchSamplerState
Serialized cursor for a compositional BatchSampler.
Bytes
A byte-string record that is not treated as a recursive numeric sequence.
CancellationToken
Cloneable cooperative cancellation notification.
CheckpointDisabled
Default builder/loader state with exact checkpointing disabled.
CheckpointFresh
Builder state after a dataset identity enables a fresh exact checkpoint.
CheckpointResume
Builder state carrying a typed checkpoint selected for resume.
CloneTransformFactory
Factory used by crate::DataLoaderBuilder::transform to clone a template.
ConcatDataset
A dataset that maps one global index across consecutive child datasets.
DataLoader
A single-threaded iterator over batches from a map-style Dataset.
DataLoaderBuilder
Builder for an owned, re-iterable map-style data loader.
DatasetSamples
A borrowing, sequential iterator over a Dataset.
Deadline
Cloneable monotonic deadline visible to cooperative worker code.
DefaultCollator
Applies DefaultCollate to an owned group of samples.
DefaultConverter
Applies DefaultConvert in no-auto-batching mode.
DistributedSampler
Deterministically shards a finite index range across distributed ranks.
DistributedSamplerState
Serialized configuration and cursor for DistributedSampler.
ExactStreamDataLoader
Exact stream owner with suspended, transactionally validated worker sources.
ExactStreamLoaderIter
Iterator whose checkpoint rolls unpublished reads back to the visible boundary.
Explicit
Explicit pinning mode validated against an available CUDA device.
ExplicitBatches
Explicit reusable batches of indices.
FnBatchSource
Adapts an epoch-aware batch-iterator factory into a reusable BatchSource.
FnCollate
Adapts an ordinary fallible closure to Collate.
FnSampler
Adapts an epoch-aware iterator factory into a reusable Sampler.
FnTransform
Adapts a fallible closure into a Transform.
FnTransformFactory
Adapts a fallible closure into a TransformFactory.
FnWorkerInit
Adapts a fallible closure into a WorkerInit.
IdentityTransform
Identity sample transform used by the default factory.
IdentityTransformFactory
Default factory that creates infallible identity transforms.
LoaderCancelled
Error returned by WorkerContext::check after cancellation or expiry.
LoaderConfiguration
Loader settings that must match when a serial checkpoint is resumed.
LoaderIter
One fresh iteration borrowed from an OwnedDataLoader.
LoaderState
Versioned state for the next batch visible from a map loader.
LogicalSampleId
Stable identity used for task randomness and future stream checkpoints.
MemoryDisabled
Builder state for item-bounded loading without byte accounting.
MemoryEnabled
Builder state for post-transform byte-bounded prefetch.
NoBatch
One-sample loading without an automatic batch dimension.
NoWorkerInit
Default worker initializer that performs no work.
OwnedDataLoader
An owned, re-iterable map-style data loader.
PinDisabled
Builder state for loading without recursive batch pinning.
PinEnabled
Builder state for enabled recursive batch pinning.
RandomSampler
A seeded sampler over a finite index range.
RandomSamplerState
Serialized configuration and cursor for RandomSampler.
ReplaySafeMap
Explicit exact-checkpoint adapter for a replay-safe map dataset.
ReplaySafeTensorDataset
Tensor dataset with private backing storage and independently owned rows.
SequenceId
A record’s position in one ordered global stream generation.
SequentialSampler
An allocation-free sampler that yields indices in ascending order.
SequentialSamplerState
Serialized cursor for SequentialSampler.
SerialExecution
Default execution capability preserving fully local serial data types.
StackDataset
A dataset that returns aligned samples from a tuple of child datasets.
StatelessWorker
Explicit unit-state adapter for a transform the caller asserts is stateless.
StreamCheckpointConfiguration
Exact streaming configuration, with no sampler state.
StreamDataLoader
Owned, re-iterable explicitly sharded stream loader.
StreamDataLoaderBuilder
Builder for an explicitly sharded positive-worker stream loader.
StreamLaneState
Paired source and transform boundary for one worker shard.
StreamLoaderIter
One borrowing iterator generation from a StreamDataLoader.
StreamLoaderState
Versioned exact stream state at the next consumer-visible batch.
Subset
A dataset containing selected indices from another dataset.
SubsetRandomSampler
A seeded sampler that shuffles a caller-provided subset of indices.
SubsetRandomSamplerState
Serialized configuration and cursor for SubsetRandomSampler.
TaskContext
Logical identity used to seed one transformation occurrence.
TensorDataset
A map-style dataset backed by tensors with a shared first dimension.
TransactionalMap
Explicit serial exact-checkpoint adapter for a transactional dataset.
TransactionalWorker
Explicit transactional adapter for a stateful transform.
VecCollate
Moves samples unchanged into their existing vector.
WeightedRandomSampler
A seeded sampler that draws indices according to nonnegative weights.
WeightedRandomSamplerState
Serialized configuration and cursor for WeightedRandomSampler.
WorkerContext
Cancellation and deadline state passed to worker lifecycle callbacks.
WorkerExecution
Execution capability selected by DataLoaderBuilder::workers.
WorkerInfo
Stable information visible while a loader worker callback is active.
WorkerLaneState
One deterministic worker lane’s committed transform state.
WorkerLoaderIter
One positive-worker iteration borrowed from an OwnedDataLoader.
WorkerRecord
One typed record produced by an explicitly sharded worker source.
WorkerTransformState
Versioned worker replay envelope within schema-v1 loader state.

Enums§

CheckpointBuildError
Resume-build error preserving either a RustTorch validation error or the transform factory’s original error value.
CheckpointPinRequest
Pinning request recorded in a loader checkpoint.
CheckpointPinStatus
Effective pinning behavior recorded in a loader checkpoint.
CollateError
An error produced by built-in collation or conversion.
LoaderError
A typed failure produced while an owned data loader is active.
PinMemoryStatus
Effective recursive pinning behavior for a built loader.
PipelineError
A preserved failure from one typed data-pipeline stage.
RandomReplacement
Replacement mode stored by RandomSamplerState.
SplitLength
One requested output length for random_split.
StreamCheckpointBuildError
Typed stream construction or transactional resume failure.
WaitOutcome
Outcome of waiting for cooperative cancellation or a deadline.
WorkerTransformLanes
Explicit zero-worker or positive-worker transform state.

Constants§

LOADER_STATE_SCHEMA_VERSION
Schema version emitted by serial map-loader checkpoints.
TASK_RNG_DERIVATION_VERSION
Version of RustTorch’s deterministic task-RNG seed derivation.
WORKER_SEED_DERIVATION_VERSION
Version of RustTorch’s deterministic worker-seed derivation.

Traits§

BatchSource
A reusable, epoch-aware source of finite index batches.
CheckpointSourceFactory
Stable factory identity for explicitly checkpointable stream sources.
Checkpointable
A component whose state can be validated before it is restored.
CheckpointableSource
Replay-safe source with self-contained state captured before each read.
Collate
Converts an owned group of samples into one batch.
Dataset
A finite, indexable collection of samples.
DatasetCheckpoint
Dataset state used by exact serial loader checkpoints.
DefaultCollate
Recursively converts one Rust sample type into its default batch type.
DefaultConvert
Recursively converts one sample without adding a batch dimension.
MemoryFootprint
Conservative logical payload bytes retained by one prefetched value.
PinMemory
Recursively moves a value into host memory pinned for device.
ReplaySafeDataset
Marker for a map dataset whose fixed-identity fetches are replay-safe.
Sampler
A reusable, epoch-aware source of finite sample indices.
StackTuple
A sealed tuple shape accepted by StackDataset.
Stateless
Marker for a component with no mutable checkpoint state.
TransactionalCheckpoint
Transactional state contract for a serial component.
Transform
A fallible sample transformation with task-local deterministic context.
TransformFactory
Constructs one transform for a serial iterator or worker.
WorkerCheckpoint
Checkpoint contract for one transform instance.
WorkerInit
Initializes one worker after its context becomes active.
WorkerSourceFactory
Creates one independently owned stream shard per worker and generation.

Functions§

batches
Batches an ordinary fallible sample iterator into vectors.
batches_with_collate
Batches an ordinary fallible sample iterator with custom collation.
chain_datasets
Flattens datasets or other iterables using the standard iterator adapter.
get_worker_info
Returns worker information for the active callback, if any.
random_split
Randomly partitions a shared dataset into non-overlapping subsets.