Skip to main content

BatchDataSet

flodl::data

Trait BatchDataSet 

Source 
pub trait BatchDataSet: Send + Sync {
    // Required methods
    fn len(&self) -> usize;
    fn get_batch(&self, indices: &[usize]) -> Result<Vec<Tensor>>;

    // Provided method
    fn is_empty(&self) -> bool { ... }
}
Expand description

A dataset that provides entire batches at once.

Implement this when your storage can produce a batch more efficiently than N individual gets (e.g., contiguous memory-mapped arrays, database bulk reads, or pre-stacked tensors).

DataSet is automatically promoted to BatchDataSet via DataSetAdapter (call get() N times and stack position-wise).

Requires Send + Sync for background prefetch (see DataSet docs).

§Contract

Each tensor in the returned Vec must have dimension 0 as the batch dimension, with length equal to indices.len(). The number of tensors and their shapes (beyond dim 0) must be consistent across calls.

Required Methods§

Source

fn len(&self) -> usize

Number of samples in the dataset.

Source

fn get_batch(&self, indices: &[usize]) -> Result<Vec<Tensor>>

Fetch a batch of samples by indices.

Returns Vec<Tensor> where each tensor has indices.len() rows along dimension 0. Position i must have consistent shape[1..] across calls.

Provided Methods§

Source

fn is_empty(&self) -> bool

Whether the dataset is empty.

Implementors§