Module preprocessing 

Preprocessing helpers.

Turns what the loaders return into what a model consumes. It covers seeded train/test and k-fold splits (plain or class-stratified), feature scaling, one-hot encoding of the categorical matrices, and label encoding. Everything is deterministic given a seed and depends on no extra crates.

Needs the preprocessing feature, which is on by default. Preprocessing helpers for the loaded datasets.

Every loader in this crate returns a Table of raw ndarray columns: numbers exactly as the source published them, and categorical values as strings. Model input usually needs four steps: split off an evaluation set, scale the numeric columns, encode the categorical columns, and encode the labels. This module provides those steps, so a user does not need to reimplement them or add a framework dependency just to run a baseline.

Splitting is index-based

The splitting functions (train_test_split, stratified_split, k_fold_indices, shuffled_indices) return row indices, not arrays. That is deliberate. A sample spans every column of the loader’s Table. One index list keeps them aligned. Convert indices to arrays with ndarray’s own select. The example below also needs the dataset feature:

use dataset_ml::Iris;
use dataset_ml::preprocessing::train_test_split;
use ndarray::Axis;

let dataset = Iris::new("./data");
let table = dataset.data().unwrap();

let features = table.numeric_matrix(&Iris::FEATURE_NAMES).unwrap();
let species = table.column(Iris::TARGET).unwrap().as_string().unwrap();

let (train, test) = train_test_split(features.nrows(), 0.2, 42).unwrap();

let train_x = features.select(Axis(0), &train);
let train_y = species.select(Axis(0), &train);
let test_x = features.select(Axis(0), &test);
let test_y = species.select(Axis(0), &test);

assert_eq!(train_x.nrows(), 120);
assert_eq!(test_x.nrows(), 30);

Determinism

Everything that shuffles takes an explicit u64 seed and uses a SplitMix64 generator built into this crate. It has no rand dependency and no hidden global state. The same seed and the same inputs always produce the same split, on every platform and every release of this crate.

Missing values

Several loaders encode a missing number as NaN (titanic, palmer_penguins, heart_disease). The scalers here compute their statistics over the finite values of each column. Non-finite entries stay untouched, so a missing value stays missing instead of corrupting the whole column’s statistics. Decide how to impute it yourself.

Structs

Scaler

Fitting a scaler produces these per-column statistics. Reuse them to replay the same transform on new data.

Functions

apply_scaler

Apply an already-fitted Scaler to a feature matrix.

class_counts

Count how many samples carry each label.

k_fold_indices

Partition 0..n_samples into k cross-validation folds.

label_encode

Map labels of any type to consecutive integer codes.

min_max_scale

Rescale each feature column into the [0, 1] range.

one_hot_encode

One-hot encode a matrix of categorical string features.

shuffled_indices

Return 0..n_samples in a deterministic pseudo-random order.

standardize

Standardize each feature column to zero mean and unit variance.

stratified_split

Split into train and test index lists that preserve each class’s proportion.

train_test_split

Split 0..n_samples into shuffled train and test index lists.

Type Aliases

IndexSplit

A pair of disjoint row-index lists that a splitting function produces.