Skip to main content

Crate dataset_ml

Crate dataset_ml 

Source
Expand description

Built-in dataset implementations for machine learning.

dataset-ml provides ready-to-use loaders for classic ML datasets, built on top of dataset_core::Dataset. Every loader lives in a module under dataset. Each one is a worked example that shows how to wrap Dataset<T, E> for one data source. The steps are the same each time:

  1. Download from a URL.
  2. Verify a SHA-256 hash.
  3. Parse the source: CSV records, raw documents from an archive, or binary IDX images.
  4. Return a Table of named, typed columns.

§Feature flags

Both features are on by default. Turn one off to leave out what you do not use.

FeatureWhat it enables
datasetThe dataset module and its loaders, the crate-root re-export of every loader struct, and DOWNLOAD_RETRIES. It adds the csv, serde, and tempfile dependencies.
preprocessingThe preprocessing module: seeded train/test and k-fold splits, feature scaling, one-hot encoding, and label encoding. It adds no dependencies.

The table and traits modules are always available, whichever features you pick. They hold Table and MlDataset, so you can write a loader of your own against the same interface with both features off.

The dataset module lists every built-in dataset with its sample count, feature count, and task type.

§Example

This example needs the dataset feature.

use dataset_ml::Iris;

let iris = Iris::new("./data");
let table = iris.data().unwrap();

// Every loader returns a `Table`. Name the columns you want in the matrix.
let features = table.numeric_matrix(&Iris::FEATURE_NAMES).unwrap();
assert_eq!(features.shape(), &[150, 4]);

// Or reach one column by name.
let species = table.column(Iris::TARGET).unwrap().as_string().unwrap();
assert_eq!(species.len(), 150);

The crate root re-exports every loader struct, so dataset_ml::Iris and dataset_ml::dataset::iris::Iris name the same type. Use whichever path reads better.

All loaders are lazy. The first call downloads and parses the file. Every later call returns a cached reference. See the individual module docs for features, target, sample count, and source.

§Beyond the loaders

Two modules apply to every dataset here rather than to one of them:

  • preprocessing: seeded train/test and k-fold splits (plain or class-stratified), feature scaling, one-hot encoding, and label encoding. You can feed the arrays a loader returns straight to a model, without writing that glue code by hand.
  • table: the Table every loader returns, and the Column and ColumnData types it holds.
  • traits: the MlDataset trait every loader implements. It lets you write code generically over “some dataset”: cache inspection and invalidation, plus a uniform n_samples().

This example needs the dataset and preprocessing features.

use dataset_ml::preprocessing::{standardize, stratified_split};
use dataset_ml::traits::MlDataset;
use dataset_ml::Iris;
use ndarray::Axis;

let iris = Iris::new("./data");
let table = iris.data().unwrap();

let features = table.numeric_matrix(&Iris::FEATURE_NAMES).unwrap();
let species = table.column(Iris::TARGET).unwrap().as_string().unwrap();

// Split with each species proportionally represented on both sides.
let (train, test) = stratified_split(species.as_slice().unwrap(), 0.2, 42).unwrap();
let (scaled_train, scaler) = standardize(&features.select(Axis(0), &train)).unwrap();

assert_eq!(scaled_train.nrows(), 120);
assert_eq!(iris.n_samples().unwrap(), 150); // from the `MlDataset` trait

Re-exports§

pub use dataset::abalone::Abalone;
pub use dataset::adult::Adult;
pub use dataset::bank_marketing::BankMarketing;
pub use dataset::banknote_authentication::BanknoteAuthentication;
pub use dataset::bike_sharing::bike_sharing_daily::BikeSharingDaily;
pub use dataset::bike_sharing::bike_sharing_hourly::BikeSharingHourly;
pub use dataset::boston_housing::BostonHousing;
pub use dataset::breast_cancer::BreastCancer;
pub use dataset::california_housing::CaliforniaHousing;
pub use dataset::car_evaluation::CarEvaluation;
pub use dataset::covtype::Covtype;
pub use dataset::diabetes::Diabetes;
pub use dataset::digits::Digits;
pub use dataset::fashion_mnist::FashionMnist;
pub use dataset::heart_disease::HeartDisease;
pub use dataset::ionosphere::Ionosphere;
pub use dataset::iris::Iris;
pub use dataset::kddcup99::Kddcup99;
pub use dataset::letter_recognition::LetterRecognition;
pub use dataset::linnerud::Linnerud;
pub use dataset::mnist::Mnist;
pub use dataset::movie_review_polarity::MovieReviewPolarity;
pub use dataset::movielens_100k::MovieLens100k;
pub use dataset::mushroom::Mushroom;
pub use dataset::newsgroups20::Newsgroups20;
pub use dataset::palmer_penguins::PalmerPenguins;
pub use dataset::sentiment_sentences::SentimentSentences;
pub use dataset::sms_spam::SmsSpam;
pub use dataset::spambase::Spambase;
pub use dataset::titanic::Titanic;
pub use dataset::wholesale_customers::WholesaleCustomers;
pub use dataset::wine_quality::red_wine_quality::RedWineQuality;
pub use dataset::wine_quality::white_wine_quality::WhiteWineQuality;
pub use dataset::wine_recognition::WineRecognition;
pub use dataset::youtube_spam::YoutubeSpam;
pub use table::Column;
pub use table::ColumnData;
pub use table::Table;
pub use traits::MlDataset;

Modules§

dataset
Every built-in dataset loader, one module per data source.
preprocessing
Preprocessing helpers.
table
The Table every loader parses into.
traits
The traits::MlDataset trait implemented by every loader in this crate.

Constants§

DOWNLOAD_RETRIES
How many extra download tries every loader in this crate makes before it stops retrying.