Skip to main content

ExactStreamDataLoader

Struct ExactStreamDataLoader 

Source
pub struct ExactStreamDataLoader<S, C, F = IdentityTransformFactory, N = PinDisabled>{ /* private fields */ }
Expand description

A reusable sharded stream whose unread position can survive a restart.

Build it with StreamDataLoaderBuilder::checkpointable. Use this for versioned data whose source and transform state can be restored before the next unread record. Prefetched work is rolled back to the last visible batch and replayed; the checkpoint contains cursors and component state, not samples.

§Save a lazy shard cursor and resume the next batch

Each shard below saves the number of its own records already read. Validation rejects an out-of-range cursor without changing the reader. set_run_context replaces cancellation state after a checkpoint barrier without resetting that cursor. The example uses JSON; applications that choose this format need a serde_json dependency.

use std::io;
use rusttorch_data::{
    CheckpointSourceFactory, CheckpointableSource, LogicalSampleId, SequenceId,
    StreamDataLoaderBuilder, StreamLoaderState, VecCollate, WorkerContext,
    WorkerRecord, WorkerSourceFactory,
};

const ROWS: usize = 11;
struct Rows;
struct Shard { cursor: usize, context: WorkerContext }
impl Shard {
    fn len(&self) -> usize {
        ROWS.saturating_sub(self.context.info.id).div_ceil(self.context.info.num_workers)
    }
}
impl Iterator for Shard {
    type Item = Result<WorkerRecord<usize>, io::Error>;
    fn next(&mut self) -> Option<Self::Item> {
        if self.context.cancellation.is_cancelled() || self.cursor == self.len() {
            return None;
        }
        let row = self.context.info.id + self.cursor * self.context.info.num_workers;
        self.cursor += 1;
        Some(Ok(WorkerRecord {
            sequence: Some(SequenceId::new(row as u64)),
            logical_id: LogicalSampleId::new(row as u64),
            sample: row,
        }))
    }
}
impl WorkerSourceFactory for Rows {
    type Sample = usize;
    type Error = io::Error;
    type Source = Shard;
    fn create(&self, context: WorkerContext) -> Result<Shard, io::Error> {
        Ok(Shard { cursor: 0, context })
    }
    fn exact_len(&self) -> Option<usize> { Some(ROWS) }
}
impl CheckpointSourceFactory for Rows {
    const CHECKPOINT_KIND: &'static str = "example.modulo-rows.v1";
}
impl CheckpointableSource for Shard {
    type Sample = usize;
    type Error = io::Error;
    type State = usize;
    fn snapshot(&self) -> usize { self.cursor }
    fn validate_snapshot(&self, cursor: &usize) -> Result<(), io::Error> {
        if *cursor > self.len() {
            return Err(io::Error::new(io::ErrorKind::InvalidData, "invalid shard cursor"));
        }
        Ok(())
    }
    fn restore_validated(&mut self, cursor: &usize) { self.cursor = *cursor; }
    fn set_run_context(&mut self, context: WorkerContext) { self.context = context; }
    fn error_sequence(&self, _: &io::Error) -> SequenceId {
        unreachable!("this in-memory source never returns a read error")
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let build = || StreamDataLoaderBuilder::new(Rows)
        .workers(3).prefetch_factor(2).batch_size(4).collate(VecCollate);
    let mut loader = build().checkpointable("rows-0-through-10.v1").build()?;
    let mut iteration = loader.iter();
    assert_eq!(iteration.next().unwrap()?, [0, 1, 2, 3]);
    let saved = serde_json::to_string(&iteration.checkpoint()?)?;
    let uninterrupted = iteration.collect::<Result<Vec<_>, _>>()?;

    let state: StreamLoaderState<usize> = serde_json::from_str(&saved)?;
    let mut restored = build().resume("rows-0-through-10.v1", state).build()?;
    let remaining = restored.iter().collect::<Result<Vec<_>, _>>()?;
    assert_eq!(remaining, [vec![4, 5, 6, 7], vec![8, 9, 10]]);
    assert_eq!(remaining, uninterrupted);
    Ok(())
}

For a file decoder, include byte offsets and all decoder state affecting future reads. Real read failures must report their reproducible global position through CheckpointableSource::error_sequence; this in-memory example has none. Use a source kind/version for the cursor format and a separate content identity for the underlying records. All shards must emit increasing sequence IDs that together cover the global sequence without gaps.

Exact streams require ordered delivery and checkpointable transforms/collation, with no byte budget, timeout, custom initializer or persistent workers. Save model, optimizer and training counters at the same completed batch. Application storage owns atomic writes and retention; decoding or validation errors should be propagated rather than treated as permission to restart from zero.

Implementations§

Source§

impl<S, C, F, N> ExactStreamDataLoader<S, C, F, N>

Source

pub fn len(&self) -> Option<usize>

Returns the exact global batch count, when known.

Source

pub fn is_empty(&self) -> Option<bool>

Returns whether the known batch count is zero.

Source

pub fn workers(&self) -> usize

Returns the shard count.

Source

pub fn pin_memory_status(&self) -> PinMemoryStatus

Returns the effective pinning policy.

Source

pub fn set_epoch(&mut self, epoch: u64)

Selects a fresh epoch and discards a retained resume.

Source

pub fn iter(&mut self) -> ExactStreamLoaderIter<'_, S, C, F, N> ⓘ

Starts a pass, consuming a pending resume only on the first call.

Following passes reopen the source from the beginning. Calling Self::set_epoch before that first pass discards its saved cursor and starts the selected epoch afresh.

Auto Trait Implementations§

§

impl<S, C, F = IdentityTransformFactory, N = PinDisabled> !RefUnwindSafe for ExactStreamDataLoader<S, C, F, N>

§

impl<S, C, F = IdentityTransformFactory, N = PinDisabled> !UnwindSafe for ExactStreamDataLoader<S, C, F, N>

§

impl<S, C, F, N> Forget for ExactStreamDataLoader<S, C, F, N>

§

impl<S, C, F, N> Freeze for ExactStreamDataLoader<S, C, F, N>
where Arc<S>: Freeze, Arc<F>: Freeze, C: Freeze, Option<Pool<S, F>>: Freeze, PhantomData<N>: Freeze,

§

impl<S, C, F, N> Send for ExactStreamDataLoader<S, C, F, N>
where Arc<S>: Send, Arc<F>: Send, C: Send, Option<Pool<S, F>>: Send, PhantomData<N>: Send,

§

impl<S, C, F, N> Sync for ExactStreamDataLoader<S, C, F, N>
where Arc<S>: Sync, Arc<F>: Sync, C: Sync, Option<Pool<S, F>>: Sync, PhantomData<N>: Sync,

§

impl<S, C, F, N> Unpin for ExactStreamDataLoader<S, C, F, N>
where Arc<S>: Unpin, Arc<F>: Unpin, C: Unpin, Option<Pool<S, F>>: Unpin, PhantomData<N>: Unpin,

§

impl<S, C, F, N> UnsafeUnpin for ExactStreamDataLoader<S, C, F, N>

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V