Struct MemoryMappedFileDataset 

pub struct MemoryMappedFileDataset<T> { /* private fields */ }

Enhanced memory-mapped dataset that loads data directly from files This provides true zero-copy access to very large datasets that exceed available RAM

Implementations

impl<T> MemoryMappedFileDataset<T>

where T: Clone + Default + Zero + Send + Sync + 'static,

pub fn from_file<P: AsRef<Path>>( file_path: P, num_samples: usize, feature_shape: Vec<usize>, label_shape: Vec<usize>, ) -> Result<Self>

Create a memory-mapped dataset directly from a file The file should contain binary data in the format: [sample0_features, sample0_labels, sample1_features, sample1_labels, …]

pub fn auto_detect<P: AsRef<Path>>( file_path: P, feature_shape: Vec<usize>, label_shape: Vec<usize>, ) -> Result<Self>

Create a memory-mapped dataset from an existing file with automatic shape detection Assumes all samples have the same shape

pub fn file_stats(&self) -> MemoryMappedFileStats

Get file statistics for monitoring

pub fn get_batch_view( &self, start_index: usize, batch_size: usize, ) -> Result<&[u8]>

Create a view that spans multiple samples for batch processing

Trait Implementations

impl<T> Dataset<T> for MemoryMappedFileDataset<T>

where T: Clone + Default + Zero + Send + Sync + Pod + 'static,

Available on crate feature mmap only.

fn len(&self) -> usize

fn get(&self, index: usize) -> Result<(Tensor<T>, Tensor<T>)>

fn is_empty(&self) -> bool

fn batch(self, batch_size: usize) -> BatchedDataset<T, Self>

where Self: Sized,