简体中文 | English

dataset-ml

Ready-to-use loaders for classic machine learning datasets, built on dataset-core.

Overview

dataset-ml includes loaders for 29 classic ML datasets. Each loader:

• Downloads the source file on first access with ureq, and retries transient network failures.
• Verifies a pinned SHA-256 hash to detect corruption or upstream changes.
• Parses the source (CSV, or raw documents extracted from an archive for the text corpora) into ndarray Array1 / Array2.
• Caches the parsed result in memory using dataset_core::Dataset<T, E>. Later accesses return a & reference with zero I/O.

Each module is also a complete reference implementation of the pattern for wrapping Dataset<T, E> for a concrete data source.

Two modules apply to every dataset rather than to one of them:

• preprocessing: seeded train/test and k-fold splits (plain or class-stratified), feature scaling, one-hot encoding, and label encoding.
• traits: the MlDataset trait every loader implements, for code written generically over "some dataset".

Installation

[dependencies]
dataset-ml = "0.4"

Datasets

The crate also re-exports all structs at the crate root, so dataset_ml::Iris, dataset_ml::RedWineQuality, etc. work too.

Usage

use dataset_ml::iris::Iris;

fn main() {
    let iris = Iris::new("./data");

    // Lazy: downloads and parses on first access, then cached.
    let features = iris.features().unwrap();  // &Array2<f64>
    let labels   = iris.labels().unwrap();    // &Array1<&'static str>

    // Or get both at once:
    let (features, labels) = iris.data().unwrap();

    assert_eq!(features.shape(), &[150, 4]);
    assert_eq!(labels.len(), 150);

    // Call .to_owned() when you need a mutable copy.
    let mut owned = features.to_owned();
    owned[[0, 0]] = 5.5;
}

Each dataset struct follows the same pattern:

• new(storage_dir): create instance (no I/O)
• features(): reference to feature matrix
• labels() / targets(): reference to label/target vector
• data(): all references at once

The text loaders SmsSpam, YoutubeSpam, SentimentSentences, Newsgroups20, and MovieReviewPolarity are the exception. A text corpus has no fixed feature matrix, so instead of features() they expose texts() (an Array1<String> of raw documents). SentimentSentences also exposes sources() (the review site each sentence came from). Newsgroups20 is the only multi-class text loader (20 classes) and offers new/new_test/new_all subset constructors.

The MlDataset trait

Every loader implements dataset_ml::traits::MlDataset, which covers the container operations that are the same whatever the loader parses into. This lets you write a function over "some dataset" instead of one concrete struct:

use dataset_ml::traits::MlDataset;
use dataset_ml::{Iris, SmsSpam};

fn describe<D: MlDataset>(dataset: &D) -> String {
    format!("{} ({} samples)", D::NAME, dataset.n_samples().unwrap())
}

fn main() {
    println!("{}", describe(&Iris::new("./data")));     // iris (150 samples)
    println!("{}", describe(&SmsSpam::new("./data")));  // sms_spam (5574 samples)
}

The trait's names deliberately differ from the inherent data() / get_data() / take_data(), so neither set ever shadows the other. Both are always available and always agree.

Preprocessing

dataset_ml::preprocessing turns what the loaders return into what a model consumes. Everything is deterministic given a seed and needs no extra crates.

use dataset_ml::preprocessing::{stratified_split, standardize, label_encode};
use dataset_ml::Iris;
use ndarray::Axis;

fn main() {
    let iris = Iris::new("./data");
    let (features, labels) = iris.data().unwrap();

    // Split with each species proportionally represented on both sides.
    let (train, test) = stratified_split(labels.as_slice().unwrap(), 0.2, 42).unwrap();

    // Fit the scaler on the training rows only, then replay it on the test rows.
    let (train_x, scaler) = standardize(&features.select(Axis(0), &train)).unwrap();
    let (codes, classes) = label_encode(&labels.select(Axis(0), &train)).unwrap();

    assert_eq!(train_x.nrows(), 120);
    assert_eq!(classes.len(), 3);
}

The splitting functions return row indices, not arrays, because a sample spans two or three parallel arrays. One index list keeps every sample aligned across them. To get arrays, use ndarray's select(Axis(0), &indices). The scalers compute their statistics over the finite values of each column. So the NaN that marks a missing value in Titanic, PalmerPenguins, and HeartDisease stays missing, instead of skewing the column's statistics.

Migration from dataset-core 0.1.x

If you used the datasets feature of dataset-core 0.1.x, switch to this crate:

- dataset-core = { version = "0.1", features = ["datasets"] }
+ dataset-ml = "0.4"

dataset_core::utils::* and dataset_core::DatasetError are unchanged. They remain in dataset-core under the utils feature.

Performance Considerations

• First access: downloads the file (if not on disk), validates SHA-256, parses, caches in memory.
• Later accesses: return a reference to the cached data, with zero allocation and zero I/O.
• .to_owned(): clones cached data into a new owned value. Use it only when you need to mutate the data.
• Offline: after the initial download, datasets stay on disk. Later runs need no network access.

License

This project is licensed under the MIT License. See LICENSE for details.

Author

SomeB1oody: stanyin64@gmail.com