tenflowers-dataset 0.1.1

Data pipeline and dataset utilities for TenfloweRS
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

//! DataLoader Module
//!
//! This module provides comprehensive data loading utilities for machine learning workflows.
//! It has been refactored into specialized submodules for better organization and maintainability.
//!
//! # Available Components
//!
//! - **Batch Results**: Types for handling both individual samples and collated tensor batches
//! - **Collation**: Various strategies for combining samples into batches with padding support
//! - **Samplers**: Different sampling strategies including sequential, random, distributed, stratified, and importance-based
//! - **Core DataLoader**: Multi-threaded data loader with prefetching and NUMA-aware scheduling
//! - **Builder**: Fluent API for configuring and creating DataLoader instances
//!
//! # Usage
//!
//! ```rust,ignore
//! use tenflowers_dataset::dataloader::{DataLoaderBuilder, SequentialSampler};
//!
//! // Create a DataLoader with custom configuration
//! let dataloader = DataLoaderBuilder::new(dataset)
//!     .batch_size(32)
//!     .num_workers(4)
//!     .prefetch_factor(2)
//!     .drop_last(true)
//!     .build(SequentialSampler::new());
//!
//! // Iterate over batches
//! for batch in dataloader.iter() {
//!     let batch = batch?;
//!     // Process batch...
//! }
//! ```

// Import all specialized dataloader modules
pub mod batch_result;
pub mod builder;
pub mod collate;
pub mod core;
pub mod samplers;

// Re-export all public types for backward compatibility
// This ensures existing code continues to work without modification

// Batch Result Types
pub use batch_result::BatchResult;

// Collation Functions
pub use collate::{BucketCollate, CollateFn, DefaultCollate, PaddingCollate, PaddingStrategy};

// Sampling Strategies
pub use samplers::{
    DistributedSampler, ImportanceSampler, RandomSampler, Sampler, SequentialSampler,
    StratifiedSampler,
};

// Core DataLoader
pub use core::{DataLoader, DataLoaderConfig, DataLoaderIterator};

// Builder Pattern
pub use builder::DataLoaderBuilder;