Expand description
Typed datasets, samplers, batching, and loading for RustTorch.
§rusttorch-data compatibility
This page is generated from compat/pytorch_api.toml against PyTorch v2.13.0 at commit cf30153, through tch 0.26.0.
Each entry is independently scoped. Supported applies only to its written scope; row counts are not a compatibility score.
§Supported
§data.batched_fetch
- PyTorch:
torch.utils.data.Dataset.__getitems__,torch.utils.data._utils.fetch._MapDatasetFetcher - RustTorch:
rusttorch_data::Dataset::get_batch,rusttorch_data::Dataset::get_batch_with_context,rusttorch::data::Dataset::get_batch,rusttorch::data::Dataset::get_batch_with_context - Implementation: Implemented by RustTorch
- Scope: Source-compatible batched map-dataset fetch with an ordered per-index default, one Dataset::get_batch call per DataLoader index batch, and an overridable context-aware hook for cooperative worker cancellation and deadlines; incorrect result cardinality is rejected before collation.
- Pinned source:
torch/utils/data/_utils/fetch.py - Evidence:
crates/rusttorch-data/tests/datasets.rs::loader_uses_one_batched_fetch_per_index_batch,crates/rusttorch-data/tests/datasets.rs::loader_preserves_errors_from_a_short_dropped_tail,crates/rusttorch-data/tests/datasets.rs::loader_rejects_wrong_batched_fetch_cardinality_before_collation,crates/rusttorch-data/tests/loader_builder.rs::wrong_batch_cardinality_is_typed_and_precedes_collation,crates/rusttorch-data/tests/worker_lifecycle.rs::early_drop_wakes_context_aware_fetch_and_joins,crates/rusttorch-data/tests/worker_lifecycle.rs::timeout_is_per_next_yielded_once_and_cancels_late_results - Notes: Rust map indices are usize and do not reproduce Python negative indexing. The preserved borrowed loader path treats wrong cardinality as a trait-contract violation, while the owned loader returns LoaderError::InvalidBatchCardinality; ordinary dataset and collation failures remain recoverable.
§data.batches
- PyTorch:
torch.utils.data.IterableDataset,torch.utils.data.DataLoader - RustTorch:
rusttorch_data::batches,rusttorch_data::batches_with_collate,rusttorch::data::batches,rusttorch::data::batches_with_collate - Implementation: Implemented by RustTorch
- Scope: Single-threaded batching of ordinary fallible Rust iterators, with explicit nonzero batch size, drop-last behavior, identity Vec collation or fallible custom collation, and one-error-then-exhaust semantics.
- Pinned source:
torch/utils/data/_utils/fetch.py - Evidence:
tests/data.rs::stream_batches_keep_a_short_tail,tests/data.rs::stream_batches_drop_a_short_tail,tests/data.rs::stream_batches_reject_zero_batch_size_with_a_structured_error,tests/data.rs::stream_batches_apply_fallible_collation,tests/data.rs::stream_batches_report_a_partial_drop_last_failure_once_then_exhaust,tests/data.rs::stream_batches_report_a_collation_failure_once_then_exhaust - Notes: rusttorch-data owns the direct surface and rusttorch::data preserves it. An ordinary Rust iterator is the streaming surface; RustTorch does not require an IterableDataset wrapper or claim Python multiprocessing behavior.
§data.chain_dataset
- PyTorch:
torch.utils.data.ChainDataset - RustTorch:
rusttorch_data::chain_datasets,rusttorch::data::chain_datasets - Implementation: Implemented by RustTorch
- Scope: Lazy sequential chaining for datasets or other Rust iterables through the standard IntoIterator flatten adapter, without custom buffering.
- Pinned source:
torch/utils/data/dataset.py - Evidence:
crates/rusttorch-data/tests/datasets.rs::chain_datasets_is_the_standard_flattened_iterator - Notes: RustTorch uses the standard typed iterator item in place of Python’s dynamically typed dataset protocol.
§data.concat_dataset
- PyTorch:
torch.utils.data.ConcatDataset - RustTorch:
rusttorch_data::ConcatDataset,rusttorch::data::ConcatDataset - Implementation: Implemented by RustTorch
- Scope: Nonempty concatenation of same-typed map datasets with checked cumulative sizes and binary-search global index routing.
- Pinned source:
torch/utils/data/dataset.py - Evidence:
crates/rusttorch-data/tests/datasets.rs::concat_dataset_uses_global_indices_and_rejects_empty_input - Notes: Indices are usize, so Python negative-index translation is intentionally absent; all children share one Rust sample and error type.
§data.dataset
- PyTorch:
torch.utils.data.Dataset - RustTorch:
rusttorch_data::Dataset,rusttorch_data::DatasetSamples,rusttorch::data::Dataset,rusttorch::data::DatasetSamples - Implementation: Implemented by RustTorch
- Scope: Finite map-style datasets with owned samples, fallible indexed access, length and empty queries, and a lazy borrowing iterator over samples in index order.
- Pinned source:
torch/utils/data/dataset.py - Evidence:
tests/data.rs::dataset_default_is_empty_follows_len,tests/data.rs::dataset_samples_borrow_and_fetch_lazily_in_order - Notes: rusttorch-data owns the direct surface and rusttorch::data preserves it. DatasetSamples performs no dataset clone or precollection; format-specific decoding belongs in optional packages.
§data.loader
- PyTorch:
torch.utils.data.DataLoader - RustTorch:
rusttorch_data::DataLoader,rusttorch_data::DataLoaderBuilder,rusttorch_data::OwnedDataLoader,rusttorch_data::LoaderIter,rusttorch_data::WorkerLoaderIter,rusttorch_data::SerialExecution,rusttorch_data::WorkerExecution,rusttorch_data::AutoBatch,rusttorch_data::ExplicitBatches,rusttorch_data::NoBatch,rusttorch_data::LoaderError,rusttorch_data::PipelineError,rusttorch::data::DataLoader,rusttorch::data::DataLoaderBuilder,rusttorch::data::OwnedDataLoader,rusttorch::data::LoaderIter,rusttorch::data::WorkerLoaderIter,rusttorch::data::SerialExecution,rusttorch::data::WorkerExecution,rusttorch::data::AutoBatch,rusttorch::data::ExplicitBatches,rusttorch::data::NoBatch,rusttorch::data::LoaderError,rusttorch::data::PipelineError - Implementation: Implemented by RustTorch
- Scope: Preserved borrowed single-pass loading plus owned, fresh epoch-aware serial or bounded map-worker iteration with default or explicit typed collation, optional no-batch conversion, sampler or batch-source plans, optional exact length, checked batch cardinality, typed five-stage one-error-then-exhaust failures, deterministic transforms, ordered or completion-order delivery, cooperative per-next timeout, optional loader-owned persistent worker pools, and order-independent PyTorch argument exclusions.
- Pinned source:
torch/utils/data/dataloader.py - Evidence:
tests/data.rs::loader_keeps_a_short_tail,tests/data.rs::loader_drops_a_short_tail,tests/data.rs::loader_rejects_zero_batch_size_with_a_structured_error,tests/data.rs::loader_applies_fallible_collation,tests/data.rs::loader_moves_non_clone_samples_into_batches,tests/data.rs::loader_discards_a_partial_batch_on_dataset_failure_then_exhausts,tests/data.rs::loader_yields_a_collation_failure_once_then_exhausts,tests/data.rs::facade_exposes_the_owned_loader_builder,tests/data.rs::facade_reexports_task_transform_and_worker_context_contracts,crates/rusttorch-data/tests/current_api.rs::owned_loader_generic_defaults_remain_source_compatible,crates/rusttorch-data/tests/loader_builder.rs::defaults_batch_scalars_and_fresh_iterations_restart,crates/rusttorch-data/tests/loader_builder.rs::batching_controls_custom_sources_and_collators,crates/rusttorch-data/tests/loader_builder.rs::no_batching_uses_default_and_custom_conversion,crates/rusttorch-data/tests/loader_builder.rs::optional_lengths_never_gate_iteration,crates/rusttorch-data/tests/loader_builder.rs::set_epoch_forwards_through_every_plan,crates/rusttorch-data/tests/loader_builder.rs::pipeline_failures_remain_typed_and_terminate_iteration,crates/rusttorch-data/tests/loader_builder.rs::conflicts_are_order_independent,crates/rusttorch-data/tests/loader_builder.rs::batch_sampler_conflicts_do_not_require_discarded_operational_bounds,crates/rusttorch-data/tests/loader_builder.rs::unallocatable_batch_size_is_a_typed_build_error_without_panicking,crates/rusttorch-data/tests/transform.rs::every_serial_stage_keeps_its_concrete_error_and_metadata,crates/rusttorch-data/tests/transform.rs::every_pipeline_variant_preserves_the_standard_error_source_chain - Notes: rusttorch-data owns the direct loader and rusttorch::data preserves it. Owned loading defaults to batch size one, sequential sampling, zero workers, ordered output, no pinning, no timeout, and no persistence. Positive map workers share an Arc dataset, fetch and transform on bounded deterministic Rust-thread lanes, and collate only on the coordinator. Rust threads replace Python multiprocessing_context, subprocesses, and pickling. Rust cannot force-cancel non-cooperative dataset or native calls, so iterator and owner drop wait for those calls before joining. Explicit stream workers are tracked separately in data.stream_workers; byte-budget, pinning, and exact serial checkpoint details are tracked in data.loader.prefetch, data.loader.pinning, and data.loader.checkpoint.
§data.loader.checkpoint_stream
- PyTorch:
No PyTorch symbol (RustTorch stream checkpoint extension) - RustTorch:
rusttorch_data::CheckpointableSource,rusttorch_data::CheckpointSourceFactory,rusttorch_data::StreamLoaderState,rusttorch_data::StreamLaneState,rusttorch_data::StreamCheckpointConfiguration,rusttorch_data::StreamCheckpointBuildError,rusttorch_data::StreamDataLoaderBuilder::checkpointable,rusttorch_data::StreamDataLoaderBuilder::resume,rusttorch_data::ExactStreamDataLoader,rusttorch_data::ExactStreamLoaderIter::checkpoint,rusttorch::data::StreamLoaderState - Implementation: Implemented by RustTorch
- Scope: RustTorch-native exact next-visible-batch replay for explicitly checkpointable sharded stream factories. Ordered shards emit strictly increasing global sequence IDs. A barrier cancels active reads, restores paired pre-attempt source/transform snapshots, drains unpublished records, and advances transport generation while retaining logical factory identity. Resume validates the envelope and all suspended source/transform/coordinator components before applying state. Original continuation, serde round trips, unequal/empty shards, global tail policy, terminal sources, and replay of advanced source/transform failures are covered.
- Pinned source:
torch/utils/data/dataloader.py - Evidence:
crates/rusttorch-data/tests/checkpoint_stream.rs::checkpoints_replay_every_boundary_and_original_continuation,crates/rusttorch-data/tests/checkpoint_stream.rs::resume_validation_is_atomic_and_static_rejection_precedes_callbacks,crates/rusttorch-data/tests/checkpoint_stream.rs::retained_source_gets_fresh_context_and_cancels_at_each_barrier_and_drop,crates/rusttorch-data/tests/checkpoint_stream.rs::active_transform_rolls_back_to_paired_pre_source_boundary,crates/rusttorch-data/tests/checkpoint_stream.rs::source_error_rolls_back_pre_attempt_and_preserves_error_chain,crates/rusttorch-data/tests/checkpoint_stream.rs::transform_failure_restores_paired_state_and_nonfused_end_replays,crates/rusttorch-data/tests/checkpoint_stream.rs::malformed_protocol_errors_once_and_ordinary_modes_stay_available,crates/rusttorch-data/tests/checkpoint_stream.rs::init_errors_and_panics_preserve_context_and_join,crates/rusttorch-data/tests/checkpoint_stream.rs::inline_snapshot_shapes_participate_in_aggregate_retention_limit,crates/rusttorch-data/tests/checkpoint_stream.rs::terminal_shards_do_not_hide_a_full_active_lane_with_a_gap,crates/rusttorch-data/tests/checkpoint_stream.rs::global_duplicate_and_state_configuration_drift_are_rejected,crates/rusttorch-data/tests/checkpoint_stream.rs::capacity_overflow_and_unsupported_settings_reject_before_workers,crates/rusttorch-data/tests/checkpoint_stream.rs::faster_later_failures_wait_for_lower_records_and_replay_in_order,crates/rusttorch-data/tests/checkpoint_stream.rs::source_failure_positions_obey_monotonic_and_bounded_gap_checks - Notes: This does not claim Python iterator-checkpoint parity. Exact streams require MemoryDisabled, NoWorkerInit, nonpersistent workers, and no timeout; ordinary stream defaults and generic auto-traits remain unchanged. Sources retaining context must implement set_run_context. Bounded journals retain at most prefetch_factor + batch_size + 1 entries per lane under the checked 64 MiB aggregate allocation ceiling; nested user-state heap contents are not a total-RSS bound. No samples are serialized or discarded records collated/pinned. Blocking callbacks remain cooperatively cancellable and drop joins. Persistence format and policy belong to the application.
§data.random_split
- PyTorch:
torch.utils.data.random_split - RustTorch:
rusttorch_data::SplitLength,rusttorch_data::random_split,rusttorch::data::SplitLength,rusttorch::data::random_split - Implementation: Implemented by RustTorch
- Scope: Seeded non-overlapping splits from exact usize counts or finite fractions summing to one, with fractional floors and round-robin remainder distribution.
- Pinned source:
torch/utils/data/dataset.py - Evidence:
crates/rusttorch-data/tests/datasets.rs::random_split_requires_integer_lengths_to_sum_to_dataset_length,crates/rusttorch-data/tests/datasets.rs::random_split_floors_fractions_and_distributes_remainder_round_robin,crates/rusttorch-data/tests/datasets.rs::random_split_rejects_fraction_floors_larger_than_the_dataset,crates/rusttorch-data/tests/datasets.rs::random_split_is_deterministic_and_allows_zero_length_parts - Notes: SplitLength is a typed Rust enum replacing Python’s mixed numeric list, and produced subset indices are usize; the seeded ChaCha12 permutation is reproducible within RustTorch rather than claimed PyTorch RNG identity.
§data.sampler
- PyTorch:
torch.utils.data.Sampler,torch.utils.data.BatchSampler,torch.utils.data.SequentialSampler,torch.utils.data.RandomSampler,torch.utils.data.SubsetRandomSampler,torch.utils.data.WeightedRandomSampler - RustTorch:
rusttorch_data::Sampler,rusttorch_data::BatchSource,rusttorch_data::FnSampler,rusttorch_data::FnBatchSource,rusttorch_data::BatchSampler,rusttorch_data::SequentialSampler,rusttorch_data::RandomSampler,rusttorch_data::SubsetRandomSampler,rusttorch_data::WeightedRandomSampler,rusttorch::data::Sampler,rusttorch::data::BatchSource,rusttorch::data::FnSampler,rusttorch::data::FnBatchSource,rusttorch::data::BatchSampler,rusttorch::data::SequentialSampler,rusttorch::data::RandomSampler,rusttorch::data::SubsetRandomSampler,rusttorch::data::WeightedRandomSampler - Implementation: Implemented by RustTorch
- Scope: Fresh finite epoch-aware index and index-batch iterator contracts with optional exact lengths; direct fixed-size batching with explicit tail policy; direct sequential, seeded replacement and repeated-permutation random, subset, and weighted sampling; and sized or unsized closure factories. Caller-controlled storage dimensions fail with typed configuration errors rather than capacity-overflow panics. Weighted sampling validates positive sample count, nonempty finite nonnegative weights, positive finite total weight, replacement cardinality, and unique numerically stable positive-first selection with deterministic zero-weight fill without replacement.
- Pinned source:
torch/utils/data/sampler.py - Evidence:
tests/data.rs::sequential_sampler_yields_every_index_in_order,tests/data.rs::random_sampler_is_seeded_and_yields_a_permutation,tests/data_libtorch.rs::random_sampler_does_not_change_libtorch_global_rng,crates/rusttorch-data/tests/samplers.rs::random_sampler_preserves_the_empty_length_error_contract,crates/rusttorch-data/tests/samplers.rs::random_sampler_with_replacement_is_seeded_and_bounded,crates/rusttorch-data/tests/samplers.rs::random_sampler_without_replacement_repeats_complete_permutations,crates/rusttorch-data/tests/samplers.rs::random_sampler_rejects_unrepresentable_storage_without_panicking,crates/rusttorch-data/tests/samplers.rs::subset_random_sampler_yields_one_seeded_permutation,crates/rusttorch-data/tests/samplers.rs::weighted_random_sampler_validates_inputs,crates/rusttorch-data/tests/samplers.rs::weighted_random_sampler_with_replacement_is_deterministic_and_has_cardinality,crates/rusttorch-data/tests/samplers.rs::weighted_random_sampler_rejects_unrepresentable_storage_without_panicking,crates/rusttorch-data/tests/samplers.rs::weighted_random_sampler_without_replacement_is_unique_and_fills_zero_weights,crates/rusttorch-data/tests/samplers.rs::weighted_random_sampler_keeps_equal_subnormal_weights_random,crates/rusttorch-data/tests/samplers.rs::concrete_samplers_create_fresh_epoch_aware_iterators,crates/rusttorch-data/tests/samplers.rs::subset_and_weighted_samplers_create_fresh_epoch_iterators,crates/rusttorch-data/tests/samplers.rs::fn_sampler_factories_are_fresh_and_length_is_optional,crates/rusttorch-data/tests/samplers.rs::fn_batch_source_factories_are_fresh_and_length_is_optional,crates/rusttorch-data/tests/batch_sampler.rs::batch_sampler_keeps_or_drops_the_exact_tail,crates/rusttorch-data/tests/batch_sampler.rs::batch_sampler_accepts_a_one_shot_custom_iterator,crates/rusttorch-data/tests/batch_sampler.rs::batch_sampler_validates_public_storage_dimensions_without_panicking,crates/rusttorch-data/tests/batch_sampler.rs::batch_sampler_recreates_sized_sampler_iterations_and_forwards_epoch,crates/rusttorch-data/tests/batch_sampler.rs::batch_sampler_preserves_an_unsized_sampler_source,crates/rusttorch-data/tests/batch_sampler.rs::fn_batch_source_recreates_nonuniform_batches_for_each_epoch - Notes: rusttorch-data owns the direct samplers and rusttorch::data preserves them. Concrete samplers and BatchSampler over an iterator remain ordinary one-shot Iterator values for source compatibility, while Sampler and BatchSource create fresh iterators for owned-loader epochs. Randomness uses sampler-local ChaCha12 state and never LibTorch’s global RNG; deterministic RustTorch seeds do not provide exact PyTorch Philox sequence parity.
§data.sampler.distributed
- PyTorch:
torch.utils.data.DistributedSampler - RustTorch:
rusttorch_data::DistributedSampler,rusttorch::data::DistributedSampler - Implementation: Implemented by RustTorch
- Scope: Explicit replica/rank sharding over finite usize index ranges, with equal per-rank lengths, cyclic padding or global-tail truncation, deterministic epoch-aware optional shuffling, fresh reusable iterations, and direct-iterator position reporting. Empty datasets yield no indices on every valid rank, and invalid or unrepresentable dimensions return typed configuration errors.
- Pinned source:
torch/utils/data/distributed.py - Evidence:
crates/rusttorch-data/tests/distributed_sampler.rs::distributed_sampler_cyclically_pads_equal_rank_lengths,crates/rusttorch-data/tests/distributed_sampler.rs::distributed_sampler_truncates_to_disjoint_rank_strides,crates/rusttorch-data/tests/distributed_sampler.rs::distributed_sampler_validates_replica_and_rank_configuration,crates/rusttorch-data/tests/distributed_sampler.rs::distributed_sampler_rejects_unrepresentable_padding_and_storage_without_panicking,crates/rusttorch-data/tests/distributed_sampler.rs::distributed_sampler_is_deterministic_per_epoch_and_creates_fresh_iterations,crates/rusttorch-data/tests/distributed_sampler.rs::distributed_sampler_tracks_direct_iteration_position_and_resets_on_epoch,crates/rusttorch-data/tests/distributed_sampler.rs::distributed_sampler_accepts_empty_datasets_for_every_valid_rank,crates/rusttorch-data/tests/distributed_sampler.rs::distributed_sampler_does_not_change_libtorch_global_rng - Notes: RustTorch requires explicit replica and rank values instead of consulting a process group. Shuffle uses sampler-local ChaCha12 state and never LibTorch’s global RNG; deterministic RustTorch seeds do not claim exact PyTorch Philox sequence parity. Versioned distributed sampler configuration and active cursor state are proven for exact serial and ordered prefetched worker checkpoints in data.loader.checkpoint.
§data.stack_dataset
- PyTorch:
torch.utils.data.StackDataset - RustTorch:
rusttorch_data::StackDataset,rusttorch_data::StackTuple,rusttorch::data::StackDataset,rusttorch::data::StackTuple - Implementation: Implemented by RustTorch
- Scope: Equal-length typed dataset tuples at arities two through eight, producing aligned sample tuples whose children share one error type.
- Pinned source:
torch/utils/data/dataset.py - Evidence:
crates/rusttorch-data/tests/datasets.rs::stack_dataset_checks_lengths_and_returns_tuples,crates/rusttorch-data/tests/datasets.rs::stack_dataset_supports_tuple_arities_two_through_eight - Notes: Typed Rust tuples replace Python’s positional tuple or string-keyed dictionary StackDataset forms; indices are usize and dictionary stacking is represented by an application-defined typed struct instead of dynamic keys.
§data.subset
- PyTorch:
torch.utils.data.Subset - RustTorch:
rusttorch_data::Subset,rusttorch::data::Subset - Implementation: Implemented by RustTorch
- Scope: Validated selection of usize source indices with shared ownership available through blanket Dataset delegation for Arc<D>.
- Pinned source:
torch/utils/data/dataset.py - Evidence:
crates/rusttorch-data/tests/datasets.rs::subset_validates_source_indices_and_arc_delegates_dataset - Notes: Construction rejects out-of-range usize indices, so Python negative indexing is intentionally not reproduced.
§data.tensor_dataset
- PyTorch:
torch.utils.data.TensorDataset - RustTorch:
rusttorch_data::TensorDataset,rusttorch::data::TensorDataset - Implementation: RustTorch frontend backed by LibTorch
- Scope: Nonempty tensors with equal first dimensions and ordered Vec<Tensor> row samples whose ordinary indexed views share LibTorch storage with the source tensors.
- Pinned source:
torch/utils/data/dataset.py - Evidence:
crates/rusttorch-data/tests/datasets.rs::tensor_dataset_returns_one_row_from_each_tensor,crates/rusttorch-data/tests/datasets.rs::tensor_dataset_rejects_mismatched_first_dimensions,crates/rusttorch-data/tests/datasets.rs::tensor_dataset_get_returns_storage_sharing_views,crates/rusttorch-data/tests/datasets.rs::tensor_dataset_get_batch_returns_storage_sharing_views - Notes: A typed Vec<Tensor> replaces Python’s dynamically sized tuple, indices are usize, and storage-sharing mutation means this adapter is not automatically replay-safe.
§Partial
§data.collation
- PyTorch:
torch.utils.data.default_collate,torch.utils.data.default_convert,torch.utils.data._utils.collate.collate - RustTorch:
rusttorch_data::Collate,rusttorch_data::VecCollate,rusttorch_data::FnCollate,rusttorch_data::DefaultCollator,rusttorch_data::DefaultCollate,rusttorch_data::DefaultConverter,rusttorch_data::DefaultConvert,rusttorch_data::Bytes,rusttorch_data::CollateError,rusttorch::data::Collate,rusttorch::data::VecCollate,rusttorch::data::FnCollate,rusttorch::data::DefaultCollator,rusttorch::data::DefaultCollate,rusttorch::data::DefaultConverter,rusttorch::data::DefaultConvert,rusttorch::data::Bytes,rusttorch::data::CollateError - Implementation: RustTorch frontend backed by LibTorch
- Scope: Typed Rust collation stacks tensors on dimension zero, creates exact-kind tensors from supported Rust scalars, preserves strings and Bytes records, and recursively collates uniform options, equal-length vectors, tuples of arity two through eight, and BTreeMap values with identical keys. Typed no-batching conversion preserves leaves and recursively converts those containers without vector transposition. VecCollate and fallible FnCollate provide explicit escape hatches.
- Pinned source:
torch/utils/data/_utils/collate.py - Evidence:
crates/rusttorch-data/tests/collate.rs::tensors_stack_on_a_new_leading_dimension,crates/rusttorch-data/tests/collate.rs::numeric_scalars_use_their_exact_libtorch_kinds,crates/rusttorch-data/tests/collate.rs::strings_and_bytes_remain_records,crates/rusttorch-data/tests/collate.rs::tuples_collate_each_field_through_arity_eight,crates/rusttorch-data/tests/collate.rs::equal_vectors_transpose_before_recursive_collation,crates/rusttorch-data/tests/collate.rs::mismatched_vector_lengths_are_typed_errors,crates/rusttorch-data/tests/collate.rs::options_require_uniform_presence_and_recurse,crates/rusttorch-data/tests/collate.rs::maps_require_identical_keys_and_recurse_over_values,crates/rusttorch-data/tests/collate.rs::empty_batches_and_incompatible_tensors_return_errors,crates/rusttorch-data/tests/collate.rs::default_converter_preserves_leaves_and_recurses_without_transposing,crates/rusttorch-data/tests/collate.rs::default_converter_collate_requires_exactly_one_sample,crates/rusttorch-data/tests/collate.rs::custom_and_vector_collators_are_explicit_escape_hatches,tests/data.rs::facade_reexports_typed_collation_contracts - Notes: RustTorch uses associated Rust types and a Bytes newtype instead of Python’s runtime type registry. It does not support NumPy arrays/scalars, Python mappings/sequences/namedtuples, mutable registry extension, dynamic coercion between unrelated Rust types, or every Python scalar type. Tensor construction and stacking delegate fallibly to LibTorch; structural mismatches and checked allocation boundaries return non-exhaustive CollateError variants.
§data.dataset.datapipes
- PyTorch:
torch.utils.data.DataChunk,torch.utils.data.DFIterDataPipe,torch.utils.data.IterDataPipe,torch.utils.data.MapDataPipe - RustTorch:
std::iter::Iterator,rusttorch_data::Dataset,rusttorch_data::WorkerSourceFactory,rusttorch_data::Transform,rusttorch::data::Dataset,rusttorch::data::WorkerSourceFactory,rusttorch::data::Transform - Implementation: Implemented by RustTorch
- Scope: Ordinary typed Rust iterators replace basic iterable DataPipe traversal, Dataset replaces indexed MapDataPipe access, WorkerSourceFactory creates independent sharded iterators, and Transform composes typed sample operations.
- Pinned source:
torch/utils/data/datapipes/datapipe.py - Evidence:
crates/rusttorch-data/tests/stream_workers.rs::four_explicit_modulo_shards_produce_each_logical_record_once,tests/data.rs::dataset_samples_borrow_and_fetch_lazily_in_order - Notes: RustTorch does not reproduce functional-name registration, reset/single-iterator hooks, Python serialization hooks, dataframe tracing, or DFIterDataPipe. Vec is the closest DataChunk container but does not retain a separate original-items raw iterator, so DataChunk is not claimed as supported.
§data.loader.checkpoint
- PyTorch:
No PyTorch symbol (RustTorch checkpoint extension) - RustTorch:
rusttorch_data::LOADER_STATE_SCHEMA_VERSION,rusttorch_data::LoaderState,rusttorch_data::LoaderConfiguration,rusttorch_data::CheckpointPinRequest,rusttorch_data::CheckpointPinStatus,rusttorch_data::CheckpointBuildError,rusttorch_data::Checkpointable,rusttorch_data::ReplaySafeDataset,rusttorch_data::DatasetCheckpoint,rusttorch_data::Stateless,rusttorch_data::TransactionalCheckpoint,rusttorch_data::WorkerCheckpoint,rusttorch_data::ReplaySafeMap,rusttorch_data::TransactionalMap,rusttorch_data::ReplaySafeTensorDataset,rusttorch_data::StatelessWorker,rusttorch_data::TransactionalWorker,rusttorch_data::DataLoaderBuilder::dataset_identity,rusttorch_data::DataLoaderBuilder::resume_from,rusttorch_data::DataLoaderBuilder::checkpoint_stateless,rusttorch_data::DataLoaderBuilder::checkpoint_transactional,rusttorch_data::LoaderIter::checkpoint,rusttorch_data::WorkerLoaderIter::checkpoint,rusttorch_data::WorkerTransformState,rusttorch_data::WorkerTransformLanes,rusttorch_data::WorkerLaneState,rusttorch_data::SequentialSamplerState,rusttorch_data::RandomSamplerState,rusttorch_data::SubsetRandomSamplerState,rusttorch_data::WeightedRandomSamplerState,rusttorch_data::DistributedSamplerState,rusttorch_data::BatchSamplerState,rusttorch::data::LoaderState,rusttorch::data::CheckpointBuildError,rusttorch::data::ReplaySafeMap,rusttorch::data::TransactionalMap,rusttorch::data::ReplaySafeTensorDataset - Implementation: Implemented by RustTorch
- Scope: Versioned serde state and exact next-visible-batch resume for serial map loaders and ordered prefetched map workers. Serial replay supports explicit replay-safe or transactional datasets. Positive workers require ReplaySafeMap (or a copy-safe tensor wrapper satisfying native Send/Sync), MemoryDisabled, NoWorkerInit, nonpersistent workers, and no timeout. Every built-in checkpointable sampler, automatic/explicit/no batching, transactional or explicitly stateless transforms, coordinator state, and pin identity are supported. Worker barriers cancel and account dispatched tasks, restore bounded per-lane pre-task snapshots, discard unpublished samples, rewind the sampler, and replay in deterministic lanes. Resume validates the envelope and all components before any apply and retains suspended worker instances.
- Pinned source:
torch/utils/data/dataloader.py - Evidence:
crates/rusttorch-data/tests/checkpoint_map.rs::prefetched_every_boundary_continues_and_resumes,crates/rusttorch-data/tests/checkpoint_map.rs::prefetched_transactional_lanes_and_zero_workers_restore_every_boundary,crates/rusttorch-data/tests/checkpoint_map.rs::prefetched_delayed_low_lane_keeps_earliest_unconsumed_snapshot,crates/rusttorch-data/tests/checkpoint_map.rs::prefetched_hidden_transform_failure_replays_after_lower_batches,crates/rusttorch-data/tests/checkpoint_map.rs::prefetched_fetch_and_cardinality_errors_recur_at_hidden_batch,crates/rusttorch-data/tests/checkpoint_map.rs::prefetched_sampler_and_stateless_task_rng_replay,crates/rusttorch-data/tests/checkpoint_map.rs::prefetched_explicit_and_no_batch_plans_restore,crates/rusttorch-data/tests/checkpoint_map.rs::prefetched_validation_is_transactional_and_threads_join,crates/rusttorch-data/tests/checkpoint_map.rs::prefetched_capability_drift_and_factory_sources_are_explicit,crates/rusttorch-data/tests/checkpoint_map.rs::prefetched_factory_identity_survives_transport_generation_changes,crates/rusttorch-data/tests/checkpoint_map.rs::json_round_trip_resumes_every_serial_boundary_exactly,crates/rusttorch-data/tests/checkpoint_map.rs::short_tail_drop_last_and_invalid_boundaries_are_explicit,crates/rusttorch-data/tests/checkpoint_map.rs::every_concrete_sampler_and_epoch_restores_its_active_cursor,crates/rusttorch-data/tests/checkpoint_map.rs::explicit_batch_sampler_and_no_batch_converter_resume,crates/rusttorch-data/tests/checkpoint_map.rs::stateful_transform_and_coordinator_restore_after_last_visible_batch,crates/rusttorch-data/tests/checkpoint_map.rs::deterministic_transform_failure_recurs_at_the_same_logical_batch,crates/rusttorch-data/tests/checkpoint_map.rs::transactional_dataset_restores_serial_fetch_state,crates/rusttorch-data/tests/checkpoint_map.rs::resume_validates_every_component_before_any_apply,crates/rusttorch-data/tests/checkpoint_map.rs::envelope_and_configuration_drift_reject_before_iteration,crates/rusttorch-data/tests/checkpoint_map.rs::replay_safe_tensor_dataset_owns_backing_and_each_fetched_row,crates/rusttorch-data/tests/checkpoint_map.rs::no_batch_state_targets_the_effective_converter,crates/rusttorch-data/tests/checkpoint_map.rs::resume_build_error_preserves_configuration_and_factory_sources,crates/rusttorch-data/tests/checkpoint_map.rs::pin_request_and_effective_status_are_checkpoint_identity,crates/rusttorch-data/tests/checkpoint_map.rs::sampler_and_distributed_semantic_corruption_rejects_without_panicking,crates/rusttorch-data/tests/checkpoint_map.rs::malformed_json_is_a_storage_error_not_a_loader_fallback,tests/data.rs::facade_reexports_exact_serial_checkpoint_contracts - Notes: This is a RustTorch-native replay capability, not Python iterator-checkpoint parity. Serial schema v1 remains unchanged; worker transform state carries a separate versioned lane/configuration and factory/transport-generation envelope. Each map lane retains at most prefetch_factor + 1 snapshots. Typed checkpoint queues/rings and coordinator ledgers participate in the checked 64 MiB allocation preflight; arbitrary state heap contents are item-bounded, not a total-RSS guarantee. Blocking callbacks remain cooperatively cancellable and drop joins. Transactional datasets remain serial-only. Persistent/initialized/byte-enabled/timeout worker replay remains unsupported. Exact sharded streams are tracked in data.loader.checkpoint_stream. Storage format and persistence policy are caller-owned.
§data.loader.pinning
- PyTorch:
torch.utils.data.DataLoader(pin_memory=...) - RustTorch:
rusttorch_data::PinMemory,rusttorch_data::PinMemoryStatus,rusttorch_data::PinDisabled,rusttorch_data::PinEnabled,rusttorch_data::Auto,rusttorch_data::Explicit,rusttorch_data::DataLoaderBuilder::pin_memory,rusttorch_data::DataLoaderBuilder::pin_memory_for,rusttorch_data::OwnedDataLoader::pin_memory_enabled,rusttorch_data::OwnedDataLoader::pin_memory_status,rusttorch_data::StreamDataLoaderBuilder::pin_memory,rusttorch_data::StreamDataLoaderBuilder::pin_memory_for,rusttorch_data::StreamDataLoader::pin_memory_enabled,rusttorch_data::StreamDataLoader::pin_memory_status,rusttorch::data::PinMemory,rusttorch::data::PinMemoryStatus,rusttorch::data::DataLoaderBuilder::pin_memory,rusttorch::data::DataLoaderBuilder::pin_memory_for,rusttorch::data::OwnedDataLoader::pin_memory_status,rusttorch::data::StreamDataLoaderBuilder::pin_memory,rusttorch::data::StreamDataLoaderBuilder::pin_memory_for,rusttorch::data::StreamDataLoader::pin_memory_status - Implementation: Implemented by RustTorch
- Scope: Type-state-enabled recursive post-collation pinning for map and explicitly sharded stream loaders across Tensor, Vec, Option, tuples of arity two through eight, and BTreeMap values, with identity scalar/string/Bytes leaves, exact batch types, typed source-preserving failures, automatic CUDA-device-zero selection, and explicit available-CUDA validation before callbacks.
- Pinned source:
torch/utils/data/dataloader.py - Evidence:
crates/rusttorch-data/tests/pin_memory.rs::recursive_pin_memory_preserves_identity_leaves_keys_and_shape_without_clone,crates/rusttorch-data/tests/pin_memory.rs::tensor_and_nested_cuda_pinning_is_checked_when_cuda_exists,crates/rusttorch-data/tests/pin_memory.rs::automatic_pin_status_is_effective_and_preserves_the_exact_batch_type,crates/rusttorch-data/tests/pin_memory.rs::disabled_custom_batch_has_no_pin_memory_bound,crates/rusttorch-data/tests/pin_memory.rs::explicit_unsupported_devices_reject_before_factory_callbacks,crates/rusttorch-data/tests/pin_memory.rs::pinning_runs_after_collation_in_both_setter_orders_when_cuda_exists,crates/rusttorch-data/tests/pin_memory.rs::pin_failure_is_typed_once_when_cuda_exists,crates/rusttorch-data/tests/pin_memory.rs::stream_auto_pin_status_and_batch_type_are_preserved,tests/data.rs::facade_reexports_memory_and_pinning_contracts - Notes: Automatic pinning uses CUDA device zero because RustTorch has no current-CUDA-index API. CPU-only and MPS-only runtimes report DisabledNoAccelerator and perform a no-op; explicit CPU, MPS, Vulkan, other devices, and unavailable CUDA indices are rejected. CUDA storage assertions and forced pin-failure evidence are conditional on the linked runtime exposing CUDA. Deprecated Python pin_memory_device is not reproduced; pin_memory_for(Device) is the typed replacement. This does not claim PyTorch custom pin_memory duck typing, accelerator selection beyond CUDA, or asynchronous transfer benchmarks.
§data.loader.prefetch
- PyTorch:
torch.utils.data.DataLoader(prefetch_factor=...) - RustTorch:
rusttorch_data::MemoryFootprint,rusttorch_data::MemoryDisabled,rusttorch_data::MemoryEnabled,rusttorch_data::DataLoaderBuilder::prefetch_factor,rusttorch_data::DataLoaderBuilder::prefetch_bytes,rusttorch_data::OwnedDataLoader::effective_prefetch_factor,rusttorch_data::OwnedDataLoader::effective_prefetch_bytes,rusttorch_data::StreamDataLoaderBuilder::prefetch_factor,rusttorch_data::StreamDataLoaderBuilder::prefetch_bytes,rusttorch_data::StreamDataLoader::effective_prefetch_factor,rusttorch_data::StreamDataLoader::effective_prefetch_bytes,rusttorch::data::MemoryFootprint,rusttorch::data::DataLoaderBuilder::prefetch_factor,rusttorch::data::DataLoaderBuilder::prefetch_bytes,rusttorch::data::OwnedDataLoader::effective_prefetch_bytes,rusttorch::data::StreamDataLoaderBuilder::prefetch_bytes,rusttorch::data::StreamDataLoader::effective_prefetch_bytes - Implementation: Implemented by RustTorch
- Scope: Positive map workers use checked bounded task/control/result storage and a global item-credit limit of workers multiplied by prefetch_factor. Optional nonzero prefetch_bytes adds a strict generation-scoped post-transform logical-payload budget to map batches and stream records through result queues and ordered reassembly, with cancellation-safe RAII permits, exact oversize metadata, sequence-aware ordered admission, bounded stream front waiters, and one-error-then-exhaust behavior.
- Pinned source:
torch/utils/data/dataloader.py - Evidence:
crates/rusttorch-data/tests/loader_builder.rs::scalar_configuration_is_validated_and_prefetch_is_normalized,crates/rusttorch-data/tests/map_workers.rs::global_outstanding_work_uses_default_and_explicit_checked_credits,crates/rusttorch-data/tests/map_workers.rs::build_rejects_before_sampler_or_batch_source_epoch_callbacks,crates/rusttorch-data/tests/memory_budget.rs::footprints_are_recursive_capacity_aware_and_saturating,crates/rusttorch-data/tests/memory_budget.rs::byte_accounting_uses_the_final_transform_type_in_either_setter_order,crates/rusttorch-data/tests/memory_budget.rs::disabled_custom_types_remain_item_bounded_without_new_trait_bounds,crates/rusttorch-data/tests/memory_budget.rs::map_zero_workers_reject_byte_prefetch_before_iteration_callbacks,crates/rusttorch-data/tests/memory_budget.rs::oversized_transformed_map_batch_has_exact_metadata_once,crates/rusttorch-data/tests/memory_budget.rs::ordered_map_byte_admission_blocks_high_until_delayed_low_is_consumable,crates/rusttorch-data/tests/memory_budget.rs::unordered_unsequenced_oversize_preserves_exact_stream_metadata_once,crates/rusttorch-data/tests/memory_budget.rs::ordered_oversize_preserves_stream_identity_without_guessing_a_batch,crates/rusttorch-data/tests/memory_budget.rs::ordered_byte_stream_rejects_missing_and_nonmonotonic_same_shard_sequences,crates/rusttorch-data/tests/memory_budget.rs::ordered_byte_admission_allows_delayed_low_progress_and_persistent_reuse,crates/rusttorch-data/tests/memory_budget.rs::early_drop_cancels_a_blocked_permit_and_persistent_restart_has_no_leak,crates/rusttorch-data/tests/memory_budget.rs::timeout_cancels_ordered_budget_waiters_before_persistent_restart,crates/rusttorch-data/tests/worker_lifecycle.rs::full_result_queue_and_idle_task_waits_wake_on_drop - Notes: MemoryFootprint is an infallible saturating logical-payload estimate, not whole-process RSS: tensor views/shared backing storage and allocator/node overhead are outside the contract. The coordinator releases permits when moving values into its one active item-bounded collation batch, which is deliberately outside prefetch_bytes. Byte accounting is opt-in and requires the final post-transform type; disabled custom types retain the item-only contract.
§data.loader.transforms
- PyTorch:
torch.utils.data.Dataset transform conventions,torch.utils.data.DataLoader(generator=...) - RustTorch:
rusttorch_data::TaskContext,rusttorch_data::CancellationToken,rusttorch_data::Deadline,rusttorch_data::WaitOutcome,rusttorch_data::LoaderCancelled,rusttorch_data::Transform,rusttorch_data::FnTransform,rusttorch_data::TransformFactory,rusttorch_data::FnTransformFactory,rusttorch_data::IdentityTransformFactory,rusttorch_data::DataLoaderBuilder::transform,rusttorch_data::DataLoaderBuilder::transform_factory,rusttorch_data::DataLoaderBuilder::seed,rusttorch_data::DataLoaderBuilder::epoch,rusttorch_data::DataLoaderBuilder::rank,rusttorch_data::TASK_RNG_DERIVATION_VERSION,rusttorch::data::TaskContext,rusttorch::data::CancellationToken,rusttorch::data::Deadline,rusttorch::data::WaitOutcome,rusttorch::data::LoaderCancelled,rusttorch::data::Transform,rusttorch::data::FnTransform,rusttorch::data::TransformFactory,rusttorch::data::FnTransformFactory,rusttorch::data::IdentityTransformFactory,rusttorch::data::DataLoaderBuilder::transform,rusttorch::data::DataLoaderBuilder::transform_factory,rusttorch::data::DataLoaderBuilder::seed,rusttorch::data::DataLoaderBuilder::epoch,rusttorch::data::DataLoaderBuilder::rank,rusttorch::data::TASK_RNG_DERIVATION_VERSION - Implementation: Implemented by RustTorch
- Scope: Owned loading constructs one typed transform per serial iterator or per worker, reuses it for that lifecycle, supplies monotonic logical occurrence IDs plus cooperative cancellation and deadline state, and preserves dataset, transform, collation, transform-construction, and worker-initialization error types through PipelineError. Version-1 task seeds deterministically mix loader seed, epoch, rank, logical sample, and stage into task-local ChaCha12 state without worker assignment.
- Pinned source:
torch/utils/data/dataloader.py - Evidence:
crates/rusttorch-data/tests/transform.rs::task_rng_is_versioned_schedule_independent_and_has_a_locked_seed,crates/rusttorch-data/tests/transform.rs::task_rng_does_not_change_libtorch_global_rng,crates/rusttorch-data/tests/transform.rs::fn_transform_preserves_output_and_error_types,crates/rusttorch-data/tests/transform.rs::serial_factory_creates_one_transform_per_iterator_and_context_ids_restart,crates/rusttorch-data/tests/transform.rs::transform_builder_clones_fresh_state_for_each_iterator,crates/rusttorch-data/tests/transform.rs::every_serial_stage_keeps_its_concrete_error_and_metadata,crates/rusttorch-data/tests/transform.rs::pipeline_error_generic_positions_are_distinct,crates/rusttorch-data/tests/worker_lifecycle.rs::context_aware_transform_wakes_on_drop_and_deadline,tests/data.rs::facade_reexports_task_transform_and_worker_context_contracts - Notes: The fixed SplitMix64-style mixer and ChaCha12 sequence are versioned RustTorch contracts; they do not reproduce PyTorch generator or Philox sequences. Deterministic modulo lanes give stateful worker transforms a reproducible history.
§data.loader.workers
- PyTorch:
torch.utils.data.DataLoader(num_workers=...),torch.utils.data.get_worker_info - RustTorch:
rusttorch_data::DataLoaderBuilder::workers,rusttorch_data::DataLoaderBuilder::persistent_workers,rusttorch_data::DataLoaderBuilder::timeout,rusttorch_data::DataLoaderBuilder::worker_init,rusttorch_data::OwnedDataLoader::workers,rusttorch_data::WorkerInfo,rusttorch_data::WorkerContext,rusttorch_data::CancellationToken,rusttorch_data::Deadline,rusttorch_data::WaitOutcome,rusttorch_data::LoaderCancelled,rusttorch_data::WorkerInit,rusttorch_data::FnWorkerInit,rusttorch_data::NoWorkerInit,rusttorch_data::get_worker_info,rusttorch_data::WORKER_SEED_DERIVATION_VERSION,rusttorch::data::DataLoaderBuilder::workers,rusttorch::data::DataLoaderBuilder::persistent_workers,rusttorch::data::DataLoaderBuilder::timeout,rusttorch::data::DataLoaderBuilder::worker_init,rusttorch::data::OwnedDataLoader::workers,rusttorch::data::WorkerInfo,rusttorch::data::WorkerContext,rusttorch::data::CancellationToken,rusttorch::data::Deadline,rusttorch::data::WaitOutcome,rusttorch::data::LoaderCancelled,rusttorch::data::WorkerInit,rusttorch::data::FnWorkerInit,rusttorch::data::NoWorkerInit,rusttorch::data::get_worker_info,rusttorch::data::WORKER_SEED_DERIVATION_VERSION - Implementation: Implemented by RustTorch
- Scope: Positive map-worker counts create joinable bounded Rust threads sharing an Arc dataset, route batch sequence s to worker s modulo worker count, initialize and construct one transform per worker, fetch and transform with cooperative generation cancellation and per-next deadlines, collate on the coordinator, preserve sampler order by default, optionally yield completion order, convert worker panics, and terminate after one typed failure. Optional persistent pools reuse threads, initial seeds, initializer calls, and factory-created transforms while quiescing and discarding tagged stale work between fresh generations.
- Pinned source:
torch/utils/data/_utils/worker.py - Evidence:
crates/rusttorch-data/tests/map_workers.rs::positive_workers_fetch_batches_concurrently_on_distinct_threads,crates/rusttorch-data/tests/map_workers.rs::ordered_and_completion_modes_follow_the_selected_contract,crates/rusttorch-data/tests/map_workers.rs::batch_sequence_routes_to_its_modulo_worker_across_iterations,crates/rusttorch-data/tests/map_workers.rs::factory_and_initializer_run_once_per_worker_with_fresh_generations,crates/rusttorch-data/tests/map_workers.rs::worker_and_coordinator_errors_are_typed_visible_once_and_contextual,crates/rusttorch-data/tests/map_workers.rs::worker_panics_are_converted_with_worker_and_batch_context,crates/rusttorch-data/tests/map_workers.rs::early_drop_disconnects_saturated_work_and_joins_every_worker,crates/rusttorch-data/tests/map_workers.rs::overflowing_timeout_rejects_before_plan_or_worker_callbacks,crates/rusttorch-data/tests/worker_context.rs::worker_info_scopes_are_nested_and_panic_safe,crates/rusttorch-data/tests/worker_context.rs::worker_info_is_isolated_between_concurrent_threads,crates/rusttorch-data/tests/worker_context.rs::worker_info_validates_identity_and_locks_versioned_seed_derivation,crates/rusttorch-data/tests/worker_context.rs::worker_seed_derivation_does_not_change_libtorch_global_rng,crates/rusttorch-data/tests/worker_lifecycle.rs::public_cancellation_and_deadline_waits_are_notification_driven,crates/rusttorch-data/tests/worker_lifecycle.rs::timeout_deadline_is_fresh_for_each_blocking_next,crates/rusttorch-data/tests/worker_lifecycle.rs::unordered_ready_batch_does_not_block_on_saturated_lane_refill,crates/rusttorch-data/tests/worker_lifecycle.rs::first_error_cancels_siblings_and_every_worker_exits_once,crates/rusttorch-data/tests/worker_lifecycle.rs::worker_init_error_and_panic_are_visible_once,crates/rusttorch-data/tests/worker_lifecycle.rs::full_result_queue_and_idle_task_waits_wake_on_drop,crates/rusttorch-data/tests/worker_lifecycle.rs::nonpersistent_iterations_use_fresh_consecutive_seed_ranges,crates/rusttorch-data/tests/worker_lifecycle.rs::serial_factory_gets_none_and_worker_callbacks_get_lifecycle_context,crates/rusttorch-data/tests/worker_lifecycle.rs::persistent_workers_reuse_threads_seeds_and_callbacks_across_epochs,crates/rusttorch-data/tests/worker_lifecycle.rs::persistent_early_drop_discards_stale_sequence_zero_in_ordered_and_unordered_modes,crates/rusttorch-data/tests/worker_lifecycle.rs::non_cooperative_fetch_makes_drop_wait_until_explicit_release,tests/data.rs::facade_reexports_task_transform_and_worker_context_contracts - Notes: Rust threads replace Python subprocesses: map datasets are shared rather than copied, collation stays on the coordinator, and WorkerInfo intentionally omits an erased dataset. Worker and task seeds are versioned local derivations with no PyTorch generator or Philox identity. Persistent factory and initializer contexts use the pool-lifetime shutdown token; fetch and task contexts use a fresh generation token and dynamic deadline. The compiling WorkerContext rustdoc demonstrates both cancellation and deadline outcomes. Nonzero loader timeouts must fit the platform monotonic clock; public overflowing deadlines fail closed as expired. Rust cannot force-cancel a non-cooperative dataset or native call, so drop waits until it returns before joining. Optional post-collation recursive pinning is tracked separately in data.loader.pinning.
§data.stream_workers
- PyTorch:
torch.utils.data.IterableDataset,torch.utils.data.DataLoader(num_workers=...),torch.utils.data.DataLoader(in_order=...),torch.utils.data.DataLoader(persistent_workers=...) - RustTorch:
rusttorch_data::SequenceId,rusttorch_data::LogicalSampleId,rusttorch_data::WorkerRecord,rusttorch_data::WorkerSourceFactory,rusttorch_data::StreamDataLoaderBuilder,rusttorch_data::StreamDataLoader,rusttorch_data::StreamLoaderIter,rusttorch::data::SequenceId,rusttorch::data::LogicalSampleId,rusttorch::data::WorkerRecord,rusttorch::data::WorkerSourceFactory,rusttorch::data::StreamDataLoaderBuilder,rusttorch::data::StreamDataLoader,rusttorch::data::StreamLoaderIter - Implementation: Implemented by RustTorch
- Scope: Explicitly sharded positive-worker Rust streams with one owned iterator per worker and generation, stable logical sample identity, optional exact global length, ordered contiguous global sequence reassembly or completion-order delivery, coordinator-side global batching and collation, checked bounded result/preallocated-flat-reassembly/batch allocation, typed pipeline/protocol/panic failures, ready-completion-first cooperative per-next timeout and cancellation, and optional loader-owned persistent worker pools.
- Pinned source:
torch/utils/data/dataloader.py - Evidence:
crates/rusttorch-data/tests/stream_workers.rs::four_explicit_modulo_shards_produce_each_logical_record_once,crates/rusttorch-data/tests/stream_workers.rs::ordered_sequence_protocol_rejects_every_gap_and_duplicate_shape_once,crates/rusttorch-data/tests/stream_workers.rs::ordered_full_credit_window_reports_first_gap_instead_of_timing_out,crates/rusttorch-data/tests/stream_workers.rs::ordered_internal_gap_beyond_same_worker_quota_is_protocol_error,crates/rusttorch-data/tests/stream_workers.rs::unordered_records_may_be_unsequenced_and_task_rng_uses_logical_id,crates/rusttorch-data/tests/stream_workers.rs::coordinator_batches_globally_and_drops_only_one_global_tail,crates/rusttorch-data/tests/stream_workers.rs::per_worker_credits_bound_fast_shards_without_blocking_their_progress,crates/rusttorch-data/tests/stream_workers.rs::every_stream_stage_failure_is_typed_contextual_and_visible_once,crates/rusttorch-data/tests/stream_workers.rs::queued_low_record_beats_an_expired_deadline_after_high_reassembly,crates/rusttorch-data/tests/stream_workers.rs::queued_protocol_error_beats_expiry_after_a_high_record,crates/rusttorch-data/src/stream.rs::queued_error_and_end_beat_an_expired_receive_deadline,crates/rusttorch-data/tests/stream_workers.rs::cooperative_stream_source_wakes_on_timeout_and_early_drop,crates/rusttorch-data/tests/stream_workers.rs::drop_waits_for_a_noncooperative_source_instead_of_detaching_it,crates/rusttorch-data/tests/stream_workers.rs::full_result_queue_drop_wakes_and_joins_every_worker,crates/rusttorch-data/tests/stream_workers.rs::persistent_ordered_and_unordered_workers_recreate_sources_but_keep_pool_state,crates/rusttorch-data/tests/stream_workers.rs::persistent_early_drop_drains_stale_results_before_both_delivery_modes_restart,crates/rusttorch-data/tests/stream_workers.rs::persistent_exhaustion_waits_until_the_generation_source_is_dropped,crates/rusttorch-data/tests/stream_workers.rs::persistent_source_destructor_panic_is_visible_and_poisons_the_pool,crates/rusttorch-data/tests/stream_workers.rs::first_protocol_failure_cancels_before_releasing_reassembly_credits,crates/rusttorch-data/tests/stream_workers.rs::iterator_drop_cancels_before_releasing_reassembly_credits,crates/rusttorch-data/tests/stream_workers.rs::invalid_stream_builders_fail_before_factory_or_worker_side_effects,crates/rusttorch-data/tests/stream_workers.rs::large_inline_batch_and_reassembly_storage_are_rejected_before_callbacks,crates/rusttorch-data/tests/stream_workers.rs::sparse_one_entry_reassembly_uses_one_concrete_vec_slot,crates/rusttorch-data/tests/stream_workers.rs::oversized_one_entry_vec_aggregate_is_rejected_before_every_callback,crates/rusttorch-data/tests/current_api.rs::stream_loader_remains_sync_for_send_but_not_sync_output,tests/data.rs::facade_exposes_explicit_stream_worker_contracts - Notes: RustTorch uses threads and typed explicitly sharded factories rather than Python worker processes or a shared IterableDataset object. Ordered sources must provide a unique zero-based contiguous global sequence; unordered sources may omit sequence but must provide stable logical IDs. Byte-enabled ordered shards additionally emit strictly increasing IDs per shard; bounded front-waiter state detects a globally missing ID without byte deadlock. Ordered reassembly is one loader-owned preallocated flat vector with linear lookup in the bounded credit window; its actual retained capacity is validated before callbacks and reused across generations. RustTorch merges before batching, so drop_last removes at most one global tail instead of PyTorch iterable-style per-process tails. Cancellation is cooperative and drop joins rather than force-cancelling a blocking native source. Checkpoint/resume remains outside this row.
§Planned
No entries.
§Python-only
§data.dataset.datapipes.decorators
- PyTorch:
torch.utils.data.argument_validation,torch.utils.data.functional_datapipe,torch.utils.data.guaranteed_datapipes_determinism,torch.utils.data.non_deterministic,torch.utils.data.runtime_validation,torch.utils.data.runtime_validation_disabled - RustTorch: —
- Implementation: Not implemented
- Scope: Python decorators dynamically register DataPipe functions, label determinism, and toggle runtime validation or construction-time determinism contexts.
- Pinned source:
torch/utils/data/datapipes/_decorator.py - Evidence: —
- Notes: Rust traits and the type system provide compile-time composition and validation but are not semantic implementations of these mutable Python registries and context managers.
§data.internal.exports
- PyTorch:
torch.utils.data._DatasetKind - RustTorch: —
- Implementation: Not implemented
- Scope: The private loader-kind dispatch enum exported by the pinned module remains a Python implementation detail.
- Pinned source:
torch/utils/data/dataloader.py - Evidence: —
- Notes: RustTorch uses type-state builders instead of exposing Python’s internal enum. argument_validation is separately attributed to the pinned DataPipe decorator module.
§Not supported
No entries.
Structs§
- Auto
- Automatic pinning mode selecting CUDA device zero when available.
- Auto
Batch - Automatic fixed-size batching over a reusable sampler.
- Batch
Sampler - Groups sampler indices into fixed-size batches.
- Batch
Sampler State - Serialized cursor for a compositional
BatchSampler. - Bytes
- A byte-string record that is not treated as a recursive numeric sequence.
- Cancellation
Token - Cloneable cooperative cancellation notification.
- Checkpoint
Disabled - Default builder/loader state with exact checkpointing disabled.
- Checkpoint
Fresh - Builder state after a dataset identity enables a fresh exact checkpoint.
- Checkpoint
Resume - Builder state carrying a typed checkpoint selected for resume.
- Clone
Transform Factory - Factory used by
crate::DataLoaderBuilder::transformto clone a template. - Concat
Dataset - A dataset that maps one global index across consecutive child datasets.
- Data
Loader - A single-threaded iterator over batches from a map-style
Dataset. - Data
Loader Builder - Builder for an owned, re-iterable map-style data loader.
- Dataset
Samples - A borrowing, sequential iterator over a
Dataset. - Deadline
- Cloneable monotonic deadline visible to cooperative worker code.
- Default
Collator - Applies
DefaultCollateto an owned group of samples. - Default
Converter - Applies
DefaultConvertin no-auto-batching mode. - Distributed
Sampler - Deterministically shards a finite index range across distributed ranks.
- Distributed
Sampler State - Serialized configuration and cursor for
DistributedSampler. - Exact
Stream Data Loader - Exact stream owner with suspended, transactionally validated worker sources.
- Exact
Stream Loader Iter - Iterator whose checkpoint rolls unpublished reads back to the visible boundary.
- Explicit
- Explicit pinning mode validated against an available CUDA device.
- Explicit
Batches - Explicit reusable batches of indices.
- FnBatch
Source - Adapts an epoch-aware batch-iterator factory into a reusable
BatchSource. - FnCollate
- Adapts an ordinary fallible closure to
Collate. - FnSampler
- Adapts an epoch-aware iterator factory into a reusable
Sampler. - FnTransform
- Adapts a fallible closure into a
Transform. - FnTransform
Factory - Adapts a fallible closure into a
TransformFactory. - FnWorker
Init - Adapts a fallible closure into a
WorkerInit. - Identity
Transform - Identity sample transform used by the default factory.
- Identity
Transform Factory - Default factory that creates infallible identity transforms.
- Loader
Cancelled - Error returned by
WorkerContext::checkafter cancellation or expiry. - Loader
Configuration - Loader settings that must match when a serial checkpoint is resumed.
- Loader
Iter - One fresh iteration borrowed from an
OwnedDataLoader. - Loader
State - Versioned state for the next batch visible from a map loader.
- Logical
Sample Id - Stable identity used for task randomness and future stream checkpoints.
- Memory
Disabled - Builder state for item-bounded loading without byte accounting.
- Memory
Enabled - Builder state for post-transform byte-bounded prefetch.
- NoBatch
- One-sample loading without an automatic batch dimension.
- NoWorker
Init - Default worker initializer that performs no work.
- Owned
Data Loader - An owned, re-iterable map-style data loader.
- PinDisabled
- Builder state for loading without recursive batch pinning.
- PinEnabled
- Builder state for enabled recursive batch pinning.
- Random
Sampler - A seeded sampler over a finite index range.
- Random
Sampler State - Serialized configuration and cursor for
RandomSampler. - Replay
Safe Map - Explicit exact-checkpoint adapter for a replay-safe map dataset.
- Replay
Safe Tensor Dataset - Tensor dataset with private backing storage and independently owned rows.
- Sequence
Id - A record’s position in one ordered global stream generation.
- Sequential
Sampler - An allocation-free sampler that yields indices in ascending order.
- Sequential
Sampler State - Serialized cursor for
SequentialSampler. - Serial
Execution - Default execution capability preserving fully local serial data types.
- Stack
Dataset - A dataset that returns aligned samples from a tuple of child datasets.
- Stateless
Worker - Explicit unit-state adapter for a transform the caller asserts is stateless.
- Stream
Checkpoint Configuration - Exact streaming configuration, with no sampler state.
- Stream
Data Loader - Owned, re-iterable explicitly sharded stream loader.
- Stream
Data Loader Builder - Builder for an explicitly sharded positive-worker stream loader.
- Stream
Lane State - Paired source and transform boundary for one worker shard.
- Stream
Loader Iter - One borrowing iterator generation from a
StreamDataLoader. - Stream
Loader State - Versioned exact stream state at the next consumer-visible batch.
- Subset
- A dataset containing selected indices from another dataset.
- Subset
Random Sampler - A seeded sampler that shuffles a caller-provided subset of indices.
- Subset
Random Sampler State - Serialized configuration and cursor for
SubsetRandomSampler. - Task
Context - Logical identity used to seed one transformation occurrence.
- Tensor
Dataset - A map-style dataset backed by tensors with a shared first dimension.
- Transactional
Map - Explicit serial exact-checkpoint adapter for a transactional dataset.
- Transactional
Worker - Explicit transactional adapter for a stateful transform.
- VecCollate
- Moves samples unchanged into their existing vector.
- Weighted
Random Sampler - A seeded sampler that draws indices according to nonnegative weights.
- Weighted
Random Sampler State - Serialized configuration and cursor for
WeightedRandomSampler. - Worker
Context - Cancellation and deadline state passed to worker lifecycle callbacks.
- Worker
Execution - Execution capability selected by
DataLoaderBuilder::workers. - Worker
Info - Stable information visible while a loader worker callback is active.
- Worker
Lane State - One deterministic worker lane’s committed transform state.
- Worker
Loader Iter - One positive-worker iteration borrowed from an
OwnedDataLoader. - Worker
Record - One typed record produced by an explicitly sharded worker source.
- Worker
Transform State - Versioned worker replay envelope within schema-v1 loader state.
Enums§
- Checkpoint
Build Error - Resume-build error preserving either a RustTorch validation error or the transform factory’s original error value.
- Checkpoint
PinRequest - Pinning request recorded in a loader checkpoint.
- Checkpoint
PinStatus - Effective pinning behavior recorded in a loader checkpoint.
- Collate
Error - An error produced by built-in collation or conversion.
- Loader
Error - A typed failure produced while an owned data loader is active.
- PinMemory
Status - Effective recursive pinning behavior for a built loader.
- Pipeline
Error - A preserved failure from one typed data-pipeline stage.
- Random
Replacement - Replacement mode stored by
RandomSamplerState. - Split
Length - One requested output length for
random_split. - Stream
Checkpoint Build Error - Typed stream construction or transactional resume failure.
- Wait
Outcome - Outcome of waiting for cooperative cancellation or a deadline.
- Worker
Transform Lanes - Explicit zero-worker or positive-worker transform state.
Constants§
- LOADER_
STATE_ SCHEMA_ VERSION - Schema version emitted by serial map-loader checkpoints.
- TASK_
RNG_ DERIVATION_ VERSION - Version of RustTorch’s deterministic task-RNG seed derivation.
- WORKER_
SEED_ DERIVATION_ VERSION - Version of RustTorch’s deterministic worker-seed derivation.
Traits§
- Batch
Source - A reusable, epoch-aware source of finite index batches.
- Checkpoint
Source Factory - Stable factory identity for explicitly checkpointable stream sources.
- Checkpointable
- A component whose state can be validated before it is restored.
- Checkpointable
Source - Replay-safe source with self-contained state captured before each read.
- Collate
- Converts an owned group of samples into one batch.
- Dataset
- A finite, indexable collection of samples.
- Dataset
Checkpoint - Dataset state used by exact serial loader checkpoints.
- Default
Collate - Recursively converts one Rust sample type into its default batch type.
- Default
Convert - Recursively converts one sample without adding a batch dimension.
- Memory
Footprint - Conservative logical payload bytes retained by one prefetched value.
- PinMemory
- Recursively moves a value into host memory pinned for
device. - Replay
Safe Dataset - Marker for a map dataset whose fixed-identity fetches are replay-safe.
- Sampler
- A reusable, epoch-aware source of finite sample indices.
- Stack
Tuple - A sealed tuple shape accepted by
StackDataset. - Stateless
- Marker for a component with no mutable checkpoint state.
- Transactional
Checkpoint - Transactional state contract for a serial component.
- Transform
- A fallible sample transformation with task-local deterministic context.
- Transform
Factory - Constructs one transform for a serial iterator or worker.
- Worker
Checkpoint - Checkpoint contract for one transform instance.
- Worker
Init - Initializes one worker after its context becomes active.
- Worker
Source Factory - Creates one independently owned stream shard per worker and generation.
Functions§
- batches
- Batches an ordinary fallible sample iterator into vectors.
- batches_
with_ collate - Batches an ordinary fallible sample iterator with custom collation.
- chain_
datasets - Flattens datasets or other iterables using the standard iterator adapter.
- get_
worker_ info - Returns worker information for the active callback, if any.
- random_
split - Randomly partitions a shared dataset into non-overlapping subsets.