pub struct StreamDataLoaderBuilder<S, C = DefaultCollator, F = IdentityTransformFactory, I = NoWorkerInit, M = MemoryDisabled, N = PinDisabled, K = CheckpointDisabled> { /* private fields */ }Expand description
Merge lazy worker-owned streams into batches for training or inference.
Use this when records are read sequentially from independent shards rather
than fetched by index. Each WorkerSourceFactory call returns one owned
iterator. Workers read and transform records; the calling thread merges and
collates them. For a single existing iterator, crate::batches is simpler.
§Read disjoint shards without collecting them first
This factory assigns every workers-th record to one worker. Together, the
shards provide contiguous global sequence IDs, so the loader restores input
order even if a later record finishes first. Five records produce two full
batches and one final short batch across the entire source.
use std::{convert::Infallible, iter::StepBy, ops::Range};
use rusttorch_data::{
LogicalSampleId, SequenceId, StreamDataLoaderBuilder, VecCollate,
WorkerContext, WorkerRecord, WorkerSourceFactory,
};
struct Rows(usize);
struct Shard(StepBy<Range<usize>>);
impl Iterator for Shard {
type Item = Result<WorkerRecord<usize>, Infallible>;
fn next(&mut self) -> Option<Self::Item> {
self.0.next().map(|row| Ok(WorkerRecord {
sequence: Some(SequenceId::new(row as u64)),
logical_id: LogicalSampleId::new(row as u64),
sample: row,
}))
}
}
impl WorkerSourceFactory for Rows {
type Sample = usize;
type Error = Infallible;
type Source = Shard;
fn create(&self, worker: WorkerContext) -> Result<Shard, Infallible> {
Ok(Shard((worker.info.id..self.0).step_by(worker.info.num_workers)))
}
fn exact_len(&self) -> Option<usize> { Some(self.0) }
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut loader = StreamDataLoaderBuilder::new(Rows(5))
.workers(2).prefetch_factor(2).batch_size(2)
.collate(VecCollate).build()?;
let batches = loader.iter().collect::<Result<Vec<_>, _>>()?;
assert_eq!(batches, [vec![0, 1], vec![2, 3], vec![4]]);
Ok(())
}§Select ordering and resource bounds
- Defaults: one worker, ordered delivery, batch size one, keeping the final tail, and prefetch factor two. Unlike map loaders, workers must be positive.
Self::drop_lastapplies after all shards are merged: uneven shards do not each lose a separate tail.Self::prefetch_factorlimits outstanding records per worker. Ordered IDs must start at zero without gaps or duplicates. If higher IDs fill the whole window before the next ID can be reached, iteration returns a protocol error; adjust sharding or the factor to keep that next record reachable.Self::ordered(false)permits completion-order delivery and omitted sequence IDs; logical sample IDs remain required.Self::prefetch_bytesadditionally limits transformed payload bytes. Ordered shards must then increase strictly in sequence order. The active collation batch and allocations inside your callbacks are outside this cap.
iter() opens fresh shards for another pass. StreamDataLoader::set_epoch
changes sample-transform randomness for the next pass. Persistent workers
reuse threads and transforms, while each pass still opens a fresh source.
Dropping an iterator cancels its work and waits for it to stop; dropping the
owner joins persistent threads too. Blocking sources must cooperate with
cancellation if prompt shutdown matters.
For restartable streams, implement crate::CheckpointableSource and
crate::CheckpointSourceFactory, then use Self::checkpointable.
ExactStreamDataLoader contains a complete JSON checkpoint/restore example.
Implementations§
Source§impl<S, C, F, I, M, N> StreamDataLoaderBuilder<S, C, F, I, M, N, CheckpointDisabled>
impl<S, C, F, I, M, N> StreamDataLoaderBuilder<S, C, F, I, M, N, CheckpointDisabled>
Sourcepub fn checkpointable(
self,
identity: impl Into<String>,
) -> StreamDataLoaderBuilder<S, C, F, I, M, N, CheckpointFresh>
pub fn checkpointable( self, identity: impl Into<String>, ) -> StreamDataLoaderBuilder<S, C, F, I, M, N, CheckpointFresh>
Enables exact checkpoints for an explicit replay-safe source identity.
The identity must change when the source contents change. Implement
CheckpointSourceFactory and CheckpointableSource for the factory
and reader; ExactStreamDataLoader provides an executable cursor and
JSON restore example. Capture on the iterator after using a batch, then
recreate the same builder and call Self::resume to continue.
Building requires a checkpoint factory, checkpointable sources/transforms, ordered delivery, no byte budget, no custom initializer, no persistence, and no timeout. Ordinary iterator batching has no checkpoint method.
use rusttorch_data::batches;
let mut iter = batches((0..4).map(Ok::<_, std::convert::Infallible>), 2, false).unwrap();
iter.checkpoint().unwrap();Sourcepub fn resume<SS, TS, CS>(
self,
identity: impl Into<String>,
state: StreamLoaderState<SS, TS, CS>,
) -> StreamDataLoaderBuilder<S, C, F, I, M, N, CheckpointResume<StreamLoaderState<SS, TS, CS>>>
pub fn resume<SS, TS, CS>( self, identity: impl Into<String>, state: StreamLoaderState<SS, TS, CS>, ) -> StreamDataLoaderBuilder<S, C, F, I, M, N, CheckpointResume<StreamLoaderState<SS, TS, CS>>>
Restores the input position saved after a successful batch.
Pass the deserialized StreamLoaderState and the same content identity
used when enabling checkpointing. Builder settings, source kind and state
must match; build() validates the envelope and components before applying
any restore. The first iter() resumes that cursor, while later calls
start fresh passes. See ExactStreamDataLoader for a complete example.
Source§impl<S, C, F, N> StreamDataLoaderBuilder<S, C, F, NoWorkerInit, MemoryDisabled, N, CheckpointFresh>where
S: CheckpointSourceFactory,
S::Source: CheckpointableSource<Sample = S::Sample, Error = S::Error>,
<<S as WorkerSourceFactory>::Source as CheckpointableSource>::State: Send + 'static,
F: TransformFactory<S::Sample> + Send + Sync + 'static,
F::Transform: WorkerCheckpoint + Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as WorkerCheckpoint>::State: Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output: Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Error: Send + 'static,
F::Error: Send + 'static,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output> + Checkpointable,
N: StreamPinPolicy<S, C, F>,
impl<S, C, F, N> StreamDataLoaderBuilder<S, C, F, NoWorkerInit, MemoryDisabled, N, CheckpointFresh>where
S: CheckpointSourceFactory,
S::Source: CheckpointableSource<Sample = S::Sample, Error = S::Error>,
<<S as WorkerSourceFactory>::Source as CheckpointableSource>::State: Send + 'static,
F: TransformFactory<S::Sample> + Send + Sync + 'static,
F::Transform: WorkerCheckpoint + Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as WorkerCheckpoint>::State: Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output: Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Error: Send + 'static,
F::Error: Send + 'static,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output> + Checkpointable,
N: StreamPinPolicy<S, C, F>,
Sourcepub fn build(
self,
) -> Result<ExactStreamDataLoader<S, C, F, N>, StreamCheckpointBuildError<<S as WorkerSourceFactory>::Error, <F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Error>>
pub fn build( self, ) -> Result<ExactStreamDataLoader<S, C, F, N>, StreamCheckpointBuildError<<S as WorkerSourceFactory>::Error, <F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Error>>
Validates exact capabilities and constructs suspended worker instances.
Source§impl<S, C, F, N> StreamDataLoaderBuilder<S, C, F, NoWorkerInit, MemoryDisabled, N, CheckpointResume<StreamLoaderState<<<S as WorkerSourceFactory>::Source as CheckpointableSource>::State, <<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as WorkerCheckpoint>::State, <C as Checkpointable>::State>>>where
S: CheckpointSourceFactory,
S::Source: CheckpointableSource<Sample = S::Sample, Error = S::Error>,
<<S as WorkerSourceFactory>::Source as CheckpointableSource>::State: Send + 'static,
F: TransformFactory<S::Sample> + Send + Sync + 'static,
F::Transform: WorkerCheckpoint + Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as WorkerCheckpoint>::State: Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output: Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Error: Send + 'static,
F::Error: Send + 'static,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output> + Checkpointable,
N: StreamPinPolicy<S, C, F>,
impl<S, C, F, N> StreamDataLoaderBuilder<S, C, F, NoWorkerInit, MemoryDisabled, N, CheckpointResume<StreamLoaderState<<<S as WorkerSourceFactory>::Source as CheckpointableSource>::State, <<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as WorkerCheckpoint>::State, <C as Checkpointable>::State>>>where
S: CheckpointSourceFactory,
S::Source: CheckpointableSource<Sample = S::Sample, Error = S::Error>,
<<S as WorkerSourceFactory>::Source as CheckpointableSource>::State: Send + 'static,
F: TransformFactory<S::Sample> + Send + Sync + 'static,
F::Transform: WorkerCheckpoint + Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as WorkerCheckpoint>::State: Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output: Send + 'static,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Error: Send + 'static,
F::Error: Send + 'static,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output> + Checkpointable,
N: StreamPinPolicy<S, C, F>,
Sourcepub fn build(
self,
) -> Result<ExactStreamDataLoader<S, C, F, N>, StreamCheckpointBuildError<<S as WorkerSourceFactory>::Error, <F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Error>>
pub fn build( self, ) -> Result<ExactStreamDataLoader<S, C, F, N>, StreamCheckpointBuildError<<S as WorkerSourceFactory>::Error, <F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Error>>
Validates the envelope before callbacks, then all components before apply.
Source§impl<S> StreamDataLoaderBuilder<S>where
S: WorkerSourceFactory,
impl<S> StreamDataLoaderBuilder<S>where
S: WorkerSourceFactory,
Source§impl<S, C, F, I, M, N, K> StreamDataLoaderBuilder<S, C, F, I, M, N, K>
impl<S, C, F, I, M, N, K> StreamDataLoaderBuilder<S, C, F, I, M, N, K>
Sourcepub fn batch_size(self, batch_size: usize) -> Self
pub fn batch_size(self, batch_size: usize) -> Self
Sets the global coordinator batch size.
Sourcepub fn drop_last(self, drop_last: bool) -> Self
pub fn drop_last(self, drop_last: bool) -> Self
Selects whether the one final global short batch is omitted.
Sourcepub fn prefetch_factor(self, factor: usize) -> Self
pub fn prefetch_factor(self, factor: usize) -> Self
Sets bounded unpublished records permitted per worker.
In ordered mode, filling the complete global reassembly window with higher IDs before the next ID is observable is a stream protocol error.
Sourcepub fn prefetch_bytes(
self,
limit: NonZeroUsize,
) -> StreamDataLoaderBuilder<S, C, F, I, MemoryEnabled, N, K>
pub fn prefetch_bytes( self, limit: NonZeroUsize, ) -> StreamDataLoaderBuilder<S, C, F, I, MemoryEnabled, N, K>
Enables a nonzero byte budget for final post-transform records.
Ordered byte-bounded shards must emit strictly increasing sequence IDs. The coordinator tracks one bounded front waiter per shard and reports a protocol error when every active shard has advanced beyond a missing ID.
Sourcepub fn pin_memory(
self,
) -> StreamDataLoaderBuilder<S, C, F, I, M, PinEnabled<Auto>, K>
pub fn pin_memory( self, ) -> StreamDataLoaderBuilder<S, C, F, I, M, PinEnabled<Auto>, K>
Enables automatic recursive pinning for CUDA device zero when available.
Sourcepub fn pin_memory_for(
self,
device: Device,
) -> StreamDataLoaderBuilder<S, C, F, I, M, PinEnabled<Explicit>, K>
pub fn pin_memory_for( self, device: Device, ) -> StreamDataLoaderBuilder<S, C, F, I, M, PinEnabled<Explicit>, K>
Enables recursive pinning for one explicit, available CUDA device.
Sourcepub fn ordered(self, ordered: bool) -> Self
pub fn ordered(self, ordered: bool) -> Self
Selects globally sequenced or completion-order delivery.
Sourcepub fn in_order(self, in_order: bool) -> Self
pub fn in_order(self, in_order: bool) -> Self
Selects globally sequenced or completion-order delivery.
Sourcepub fn rank(self, rank: usize) -> Self
pub fn rank(self, rank: usize) -> Self
Sets the distributed rank included in worker and task contexts.
Sourcepub fn timeout(self, timeout: Duration) -> Self
pub fn timeout(self, timeout: Duration) -> Self
Sets the cooperative timeout for each blocking iterator call.
Sourcepub fn persistent_workers(self, persistent: bool) -> Self
pub fn persistent_workers(self, persistent: bool) -> Self
Selects loader-owned persistent worker threads.
Sourcepub fn collate<C2>(
self,
collator: C2,
) -> StreamDataLoaderBuilder<S, C2, F, I, M, N, K>
pub fn collate<C2>( self, collator: C2, ) -> StreamDataLoaderBuilder<S, C2, F, I, M, N, K>
Replaces coordinator collation.
Sourcepub fn transform<T>(
self,
transform: T,
) -> StreamDataLoaderBuilder<S, C, CloneTransformFactory<T>, I, M, N, K>
pub fn transform<T>( self, transform: T, ) -> StreamDataLoaderBuilder<S, C, CloneTransformFactory<T>, I, M, N, K>
Clones one transform template for each worker lifecycle.
Sourcepub fn transform_factory<F2>(
self,
factory: F2,
) -> StreamDataLoaderBuilder<S, C, F2, I, M, N, K>
pub fn transform_factory<F2>( self, factory: F2, ) -> StreamDataLoaderBuilder<S, C, F2, I, M, N, K>
Replaces worker transform construction.
Sourcepub fn worker_init<I2>(
self,
worker_init: I2,
) -> StreamDataLoaderBuilder<S, C, F, I2, M, N, K>
pub fn worker_init<I2>( self, worker_init: I2, ) -> StreamDataLoaderBuilder<S, C, F, I2, M, N, K>
Replaces worker initialization.
Source§impl<S, C, F, I> StreamDataLoaderBuilder<S, C, F, I, MemoryDisabled, PinDisabled>
impl<S, C, F, I> StreamDataLoaderBuilder<S, C, F, I, MemoryDisabled, PinDisabled>
Sourcepub fn build(
self,
) -> Result<StreamDataLoader<S, C, F, I, MemoryDisabled, PinDisabled>>where
S: WorkerSourceFactory,
F: TransformFactory<S::Sample>,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output>,
I: WorkerInit,
pub fn build(
self,
) -> Result<StreamDataLoader<S, C, F, I, MemoryDisabled, PinDisabled>>where
S: WorkerSourceFactory,
F: TransformFactory<S::Sample>,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output>,
I: WorkerInit,
Validates configuration and builds an item-bounded, unpinned stream.
Source§impl<S, C, F, I> StreamDataLoaderBuilder<S, C, F, I, MemoryEnabled, PinDisabled>
impl<S, C, F, I> StreamDataLoaderBuilder<S, C, F, I, MemoryEnabled, PinDisabled>
Sourcepub fn build(
self,
) -> Result<StreamDataLoader<S, C, F, I, MemoryEnabled, PinDisabled>>where
S: WorkerSourceFactory,
F: TransformFactory<S::Sample>,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output: MemoryFootprint,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output>,
I: WorkerInit,
pub fn build(
self,
) -> Result<StreamDataLoader<S, C, F, I, MemoryEnabled, PinDisabled>>where
S: WorkerSourceFactory,
F: TransformFactory<S::Sample>,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output: MemoryFootprint,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output>,
I: WorkerInit,
Validates configuration and builds a byte-bounded, unpinned stream.
Source§impl<S, C, F, I, Q> StreamDataLoaderBuilder<S, C, F, I, MemoryDisabled, PinEnabled<Q>>
impl<S, C, F, I, Q> StreamDataLoaderBuilder<S, C, F, I, MemoryDisabled, PinEnabled<Q>>
Sourcepub fn build(
self,
) -> Result<StreamDataLoader<S, C, F, I, MemoryDisabled, PinEnabled<Q>>>where
S: WorkerSourceFactory,
F: TransformFactory<S::Sample>,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output>,
C::Batch: PinMemory,
I: WorkerInit,
pub fn build(
self,
) -> Result<StreamDataLoader<S, C, F, I, MemoryDisabled, PinEnabled<Q>>>where
S: WorkerSourceFactory,
F: TransformFactory<S::Sample>,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output>,
C::Batch: PinMemory,
I: WorkerInit,
Validates configuration and builds an item-bounded pinned stream.
Source§impl<S, C, F, I, Q> StreamDataLoaderBuilder<S, C, F, I, MemoryEnabled, PinEnabled<Q>>
impl<S, C, F, I, Q> StreamDataLoaderBuilder<S, C, F, I, MemoryEnabled, PinEnabled<Q>>
Sourcepub fn build(
self,
) -> Result<StreamDataLoader<S, C, F, I, MemoryEnabled, PinEnabled<Q>>>where
S: WorkerSourceFactory,
F: TransformFactory<S::Sample>,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output: MemoryFootprint,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output>,
C::Batch: PinMemory,
I: WorkerInit,
pub fn build(
self,
) -> Result<StreamDataLoader<S, C, F, I, MemoryEnabled, PinEnabled<Q>>>where
S: WorkerSourceFactory,
F: TransformFactory<S::Sample>,
<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output: MemoryFootprint,
C: Collate<<<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as Transform<<S as WorkerSourceFactory>::Sample>>::Output>,
C::Batch: PinMemory,
I: WorkerInit,
Validates configuration and builds a byte-bounded pinned stream.