burn-dataset 0.22.0

Library with simple dataset APIs for creating ML data pipelines
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

#![warn(missing_docs)]
#![cfg_attr(docsrs, feature(doc_cfg))]

//! # Burn Dataset
//!
//! Datasets, transformations and data sources for Burn.
//!
//! Applications enable the `dataset` feature of `burn` (also enabled by `train`) and use this crate as
//! `burn::data::dataset`.
//!
//! - [`Dataset`] is random access by index. [`InMemDataset`] holds items in memory, and the
//!   `sqlite` feature adds `SqliteDataset` for datasets larger than memory.
//! - [`transform`]: mapping, sampling, shuffling, windowing, partial views, selection and
//!   composition of datasets, applied lazily.
//! - [`source`]: dataset downloads, including Hugging Face datasets.
//! - `vision`, `nlp`, `audio`: ready-made datasets such as MNIST, CIFAR, image folders, AG News
//!   and Speech Commands, behind the features of the same names.
//!
//! # Feature flags
//!
//! - `sqlite`: SQLite-backed datasets, using the [Turso](https://turso.tech/) engine.
//!   `sqlite-bundled` is a deprecated alias.
//! - `vision`, `nlp`, `audio`: domain datasets and loaders. `builtin-sources` enables the
//!   downloadable vision and NLP datasets.
//! - `dataframe`: datasets backed by a Polars `DataFrame`.
//! - `network`: file downloads.
//! - `fake`: generated datasets for tests.
//! - `tracing`: instrument operations with the `tracing` crate.

#[macro_use]
extern crate derive_new;

extern crate alloc;
extern crate dirs;

/// Sources for datasets.
pub mod source;

pub mod transform;

/// Audio datasets.
#[cfg(feature = "audio")]
pub mod audio;

/// Vision datasets.
#[cfg(feature = "vision")]
pub mod vision;

/// Natural language processing datasets.
#[cfg(feature = "nlp")]
pub mod nlp;

/// Network dataset utilities.
#[cfg(feature = "network")]
pub mod network {
    pub use burn_std::network::*;
}

mod dataset;
pub use dataset::*;
#[cfg(feature = "sqlite")]
pub use source::huggingface::downloader::*;

#[cfg(test)]
mod test_data {
    pub fn string_items() -> Vec<String> {
        vec![
            "1 Item".to_string(),
            "2 Items".to_string(),
            "3 Items".to_string(),
            "4 Items".to_string(),
        ]
    }
}