Skip to main content

Crate incin_data

Crate incin_data 

Source
Expand description

§Incin Data

Provides dataset and dataloader abstractions for the Incin framework. Handles batching, shuffling, and data transformation pipelines.

§Examples

A Dataset supplies items by index; a Collate can turn a batch into the model’s input type; a DataLoader joins the two and yields typed Result values so worker failures cannot look like end-of-data. Use DataLoader::builder for the default scalar, tuple, and tensor batching policy.

use incin_data::{Collate, DataLoader, Dataset};

struct Squares;

impl Dataset for Squares {
    type Item = u32;

    fn len(&self) -> usize {
        100
    }

    fn get(&self, index: usize) -> Result<Option<Self::Item>, incin_data::DataError> {
        Ok(u32::try_from(index).ok().filter(|_| index < self.len()).map(|i| i * i))
    }
}

struct IntoBatch;

impl Collate<u32> for IntoBatch {
    type Output = Vec<u32>;

    fn collate(&self, batch: Vec<u32>) -> incin_data::BatchResult<Self::Output> {
        Ok(batch)
    }
}

// Batches of 32, shuffled. Loading happens on worker threads; iterating
// borrows the loader, so the same one can be iterated each epoch.
let loader = DataLoader::new(Squares, IntoBatch, 32).unwrap().with_shuffle(true);

let batches: Vec<Vec<u32>> = (&loader).into_iter().collect::<Result<_, _>>().unwrap();
assert_eq!(batches.len(), 4); // 100 items is three full batches and a short one
assert_eq!(batches.iter().map(Vec::len).sum::<usize>(), 100);

Re-exports§

pub use dataset::Dataset;
pub use downloader::Downloader;
pub use loader::BatchResult;
pub use loader::Collate;
pub use loader::DataError;
pub use loader::DataLoader;
pub use loader::DataLoaderBuilder;
pub use loader::DefaultCollate;
pub use transforms::CenterCrop;
pub use transforms::Compose;
pub use transforms::Normalize;
pub use transforms::RandomHorizontalFlip;
pub use transforms::Scale;
pub use transforms::Transform;

Modules§

dataset
Dataset.
downloader
Downloader.
hub
Hub.
loader
Loader.
prelude
Prelude.
transforms
Transforms and data processing pipelines. Data transformation and augmentation pipeline.
vision
Vision.