Skip to main content

StreamDataLoaderBuilder

Struct StreamDataLoaderBuilder 

Source
pub struct StreamDataLoaderBuilder<S, C = DefaultCollator, F = IdentityTransformFactory, I = NoWorkerInit, M = MemoryDisabled, N = PinDisabled, K = CheckpointDisabled> { /* private fields */ }
Expand description

Merge lazy worker-owned streams into batches for training or inference.

Use this when records are read sequentially from independent shards rather than fetched by index. Each WorkerSourceFactory call returns one owned iterator. Workers read and transform records; the calling thread merges and collates them. For a single existing iterator, crate::batches is simpler.

§Read disjoint shards without collecting them first

This factory assigns every workers-th record to one worker. Together, the shards provide contiguous global sequence IDs, so the loader restores input order even if a later record finishes first. Five records produce two full batches and one final short batch across the entire source.

use std::{convert::Infallible, iter::StepBy, ops::Range};
use rusttorch_data::{
    LogicalSampleId, SequenceId, StreamDataLoaderBuilder, VecCollate,
    WorkerContext, WorkerRecord, WorkerSourceFactory,
};
struct Rows(usize);
struct Shard(StepBy<Range<usize>>);
impl Iterator for Shard {
    type Item = Result<WorkerRecord<usize>, Infallible>;
    fn next(&mut self) -> Option<Self::Item> {
        self.0.next().map(|row| Ok(WorkerRecord {
            sequence: Some(SequenceId::new(row as u64)),
            logical_id: LogicalSampleId::new(row as u64),
            sample: row,
        }))
    }
}
impl WorkerSourceFactory for Rows {
    type Sample = usize;
    type Error = Infallible;
    type Source = Shard;
    fn create(&self, worker: WorkerContext) -> Result<Shard, Infallible> {
        Ok(Shard((worker.info.id..self.0).step_by(worker.info.num_workers)))
    }
    fn exact_len(&self) -> Option<usize> { Some(self.0) }
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut loader = StreamDataLoaderBuilder::new(Rows(5))
        .workers(2).prefetch_factor(2).batch_size(2)
        .collate(VecCollate).build()?;
    let batches = loader.iter().collect::<Result<Vec<_>, _>>()?;
    assert_eq!(batches, [vec![0, 1], vec![2, 3], vec![4]]);
    Ok(())
}

§Select ordering and resource bounds

  • Defaults: one worker, ordered delivery, batch size one, keeping the final tail, and prefetch factor two. Unlike map loaders, workers must be positive.
  • Self::drop_last applies after all shards are merged: uneven shards do not each lose a separate tail.
  • Self::prefetch_factor limits outstanding records per worker. Ordered IDs must start at zero without gaps or duplicates. If higher IDs fill the whole window before the next ID can be reached, iteration returns a protocol error; adjust sharding or the factor to keep that next record reachable.
  • Self::ordered(false) permits completion-order delivery and omitted sequence IDs; logical sample IDs remain required.
  • Self::prefetch_bytes additionally limits transformed payload bytes. Ordered shards must then increase strictly in sequence order. The active collation batch and allocations inside your callbacks are outside this cap.

iter() opens fresh shards for another pass. StreamDataLoader::set_epoch changes sample-transform randomness for the next pass. Persistent workers reuse threads and transforms, while each pass still opens a fresh source. Dropping an iterator cancels its work and waits for it to stop; dropping the owner joins persistent threads too. Blocking sources must cooperate with cancellation if prompt shutdown matters.

For restartable streams, implement crate::CheckpointableSource and crate::CheckpointSourceFactory, then use Self::checkpointable. ExactStreamDataLoader contains a complete JSON checkpoint/restore example.

Implementations§

Source§

impl<S, C, F, I, M, N> StreamDataLoaderBuilder<S, C, F, I, M, N, CheckpointDisabled>

Source

pub fn checkpointable( self, identity: impl Into<String>, ) -> StreamDataLoaderBuilder<S, C, F, I, M, N, CheckpointFresh>

Enables exact checkpoints for an explicit replay-safe source identity.

The identity must change when the source contents change. Implement CheckpointSourceFactory and CheckpointableSource for the factory and reader; ExactStreamDataLoader provides an executable cursor and JSON restore example. Capture on the iterator after using a batch, then recreate the same builder and call Self::resume to continue.

Building requires a checkpoint factory, checkpointable sources/transforms, ordered delivery, no byte budget, no custom initializer, no persistence, and no timeout. Ordinary iterator batching has no checkpoint method.

ⓘ
use rusttorch_data::batches;
let mut iter = batches((0..4).map(Ok::<_, std::convert::Infallible>), 2, false).unwrap();
iter.checkpoint().unwrap();
Source

pub fn resume<SS, TS, CS>( self, identity: impl Into<String>, state: StreamLoaderState<SS, TS, CS>, ) -> StreamDataLoaderBuilder<S, C, F, I, M, N, CheckpointResume<StreamLoaderState<SS, TS, CS>>>

Restores the input position saved after a successful batch.

Pass the deserialized StreamLoaderState and the same content identity used when enabling checkpointing. Builder settings, source kind and state must match; build() validates the envelope and components before applying any restore. The first iter() resumes that cursor, while later calls start fresh passes. See ExactStreamDataLoader for a complete example.

Source§

impl<S, C, F, N> StreamDataLoaderBuilder<S, C, F, NoWorkerInit, MemoryDisabled, N, CheckpointFresh>

Source

pub fn build( self, ) -> Result<ExactStreamDataLoader<S, C, F, N>, StreamCheckpointBuildError<<S as WorkerSourceFactory>::Error, <F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Error>>

Validates exact capabilities and constructs suspended worker instances.

Source§

impl<S, C, F, N> StreamDataLoaderBuilder<S, C, F, NoWorkerInit, MemoryDisabled, N, CheckpointResume<StreamLoaderState<<<S as WorkerSourceFactory>::Source as CheckpointableSource>::State, <<F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Transform as WorkerCheckpoint>::State, <C as Checkpointable>::State>>>

Source

pub fn build( self, ) -> Result<ExactStreamDataLoader<S, C, F, N>, StreamCheckpointBuildError<<S as WorkerSourceFactory>::Error, <F as TransformFactory<<S as WorkerSourceFactory>::Sample>>::Error>>

Validates the envelope before callbacks, then all components before apply.

Source§

impl<S> StreamDataLoaderBuilder<S>

Source

pub fn new(factory: S) -> Self

Starts a builder with one worker, ordered delivery, batch size one, prefetch factor two, no timeout, and no persistence.

Source§

impl<S, C, F, I, M, N, K> StreamDataLoaderBuilder<S, C, F, I, M, N, K>

Source

pub fn workers(self, workers: usize) -> Self

Sets the positive stream worker count.

Source

pub fn batch_size(self, batch_size: usize) -> Self

Sets the global coordinator batch size.

Source

pub fn drop_last(self, drop_last: bool) -> Self

Selects whether the one final global short batch is omitted.

Source

pub fn prefetch_factor(self, factor: usize) -> Self

Sets bounded unpublished records permitted per worker.

In ordered mode, filling the complete global reassembly window with higher IDs before the next ID is observable is a stream protocol error.

Source

pub fn prefetch_bytes( self, limit: NonZeroUsize, ) -> StreamDataLoaderBuilder<S, C, F, I, MemoryEnabled, N, K>

Enables a nonzero byte budget for final post-transform records.

Ordered byte-bounded shards must emit strictly increasing sequence IDs. The coordinator tracks one bounded front waiter per shard and reports a protocol error when every active shard has advanced beyond a missing ID.

Source

pub fn pin_memory( self, ) -> StreamDataLoaderBuilder<S, C, F, I, M, PinEnabled<Auto>, K>

Enables automatic recursive pinning for CUDA device zero when available.

Source

pub fn pin_memory_for( self, device: Device, ) -> StreamDataLoaderBuilder<S, C, F, I, M, PinEnabled<Explicit>, K>

Enables recursive pinning for one explicit, available CUDA device.

Source

pub fn ordered(self, ordered: bool) -> Self

Selects globally sequenced or completion-order delivery.

Source

pub fn in_order(self, in_order: bool) -> Self

Selects globally sequenced or completion-order delivery.

Source

pub fn seed(self, seed: u64) -> Self

Sets the deterministic loader seed.

Source

pub fn epoch(self, epoch: u64) -> Self

Sets the initial stream epoch.

Source

pub fn rank(self, rank: usize) -> Self

Sets the distributed rank included in worker and task contexts.

Source

pub fn timeout(self, timeout: Duration) -> Self

Sets the cooperative timeout for each blocking iterator call.

Source

pub fn persistent_workers(self, persistent: bool) -> Self

Selects loader-owned persistent worker threads.

Source

pub fn collate<C2>( self, collator: C2, ) -> StreamDataLoaderBuilder<S, C2, F, I, M, N, K>

Replaces coordinator collation.

Source

pub fn transform<T>( self, transform: T, ) -> StreamDataLoaderBuilder<S, C, CloneTransformFactory<T>, I, M, N, K>

Clones one transform template for each worker lifecycle.

Source

pub fn transform_factory<F2>( self, factory: F2, ) -> StreamDataLoaderBuilder<S, C, F2, I, M, N, K>

Replaces worker transform construction.

Source

pub fn worker_init<I2>( self, worker_init: I2, ) -> StreamDataLoaderBuilder<S, C, F, I2, M, N, K>

Replaces worker initialization.

Source§

impl<S, C, F, I> StreamDataLoaderBuilder<S, C, F, I, MemoryDisabled, PinDisabled>

Source

pub fn build( self, ) -> Result<StreamDataLoader<S, C, F, I, MemoryDisabled, PinDisabled>>

Validates configuration and builds an item-bounded, unpinned stream.

Source§

impl<S, C, F, I> StreamDataLoaderBuilder<S, C, F, I, MemoryEnabled, PinDisabled>

Source

pub fn build( self, ) -> Result<StreamDataLoader<S, C, F, I, MemoryEnabled, PinDisabled>>

Validates configuration and builds a byte-bounded, unpinned stream.

Source§

impl<S, C, F, I, Q> StreamDataLoaderBuilder<S, C, F, I, MemoryDisabled, PinEnabled<Q>>

Source

pub fn build( self, ) -> Result<StreamDataLoader<S, C, F, I, MemoryDisabled, PinEnabled<Q>>>

Validates configuration and builds an item-bounded pinned stream.

Source§

impl<S, C, F, I, Q> StreamDataLoaderBuilder<S, C, F, I, MemoryEnabled, PinEnabled<Q>>

Source

pub fn build( self, ) -> Result<StreamDataLoader<S, C, F, I, MemoryEnabled, PinEnabled<Q>>>

Validates configuration and builds a byte-bounded pinned stream.

Auto Trait Implementations§

§

impl<S, C, F, I, M, N, K> Freeze for StreamDataLoaderBuilder<S, C, F, I, M, N, K>
where S: Freeze, C: Freeze, F: Freeze, I: Freeze, K: Freeze, PhantomData<(M, N)>: Freeze,

§

impl<S, C, F, I, M, N, K> RefUnwindSafe for StreamDataLoaderBuilder<S, C, F, I, M, N, K>

§

impl<S, C, F, I, M, N, K> Send for StreamDataLoaderBuilder<S, C, F, I, M, N, K>
where S: Send, C: Send, F: Send, I: Send, K: Send, PhantomData<(M, N)>: Send,

§

impl<S, C, F, I, M, N, K> Sync for StreamDataLoaderBuilder<S, C, F, I, M, N, K>
where S: Sync, C: Sync, F: Sync, I: Sync, K: Sync, PhantomData<(M, N)>: Sync,

§

impl<S, C, F, I, M, N, K> Unpin for StreamDataLoaderBuilder<S, C, F, I, M, N, K>
where S: Unpin, C: Unpin, F: Unpin, I: Unpin, K: Unpin, PhantomData<(M, N)>: Unpin,

§

impl<S, C, F, I, M, N, K> UnsafeUnpin for StreamDataLoaderBuilder<S, C, F, I, M, N, K>

§

impl<S, C, F, I, M, N, K> UnwindSafe for StreamDataLoaderBuilder<S, C, F, I, M, N, K>

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V