Struct StreamingDataLoader 

pub struct StreamingDataLoader { /* private fields */ }

A data loader that lazily reads batches from disk, using a background thread to prefetch the next batch while the current batch is being processed.

The loader scans a directory for numbered batch files (batch_0000.bin, batch_0001.bin, etc.). Each file stores a pair of tensors (inputs + targets) in a simple binary format:

[input_ndims: u32] [input_shape...] [input_data: f32...]
[target_ndims: u32] [target_shape...] [target_data: f32...]

Implementations

impl StreamingDataLoader

pub fn new( path: impl Into<PathBuf>, batch_size: usize, ) -> Result<Self, ModelError>

Create a new streaming data loader that reads batch files from path.

The directory is scanned for files matching batch_NNNN.bin. If no batch files are found, an empty loader is returned.

pub fn next_batch(&mut self) -> Option<(Tensor, Tensor)>

Returns the next batch of (inputs, targets), or None if all batches have been consumed for this epoch.

pub fn reset(&mut self)

Reset the loader to the beginning so it can be iterated again.

pub fn len(&self) -> usize

Returns the total number of batch files available.

pub fn is_empty(&self) -> bool

Returns true if there are no batch files.

pub fn batch_size(&self) -> usize

Returns the configured batch size.

pub fn path(&self) -> &Path

Returns the directory path.

pub fn write_batch_file( path: &Path, inputs: &Tensor, targets: &Tensor, ) -> Result<(), ModelError>

Write a tensor pair to the simple binary format.