# 4. Data Preprocessing
Raw data almost never arrives in the shape a model wants. Feature columns can span very different scales. Class labels can come as strings or as non-consecutive integers. You also need an honest held-out set before you fit anything. This chapter covers the preprocessing helpers in `rustyml::utils`: splitting, standardizing, normalizing, and label encoding. They all live behind the [`utils` feature flag](../Chapter-01/1.2._Installation_and_Feature_Flags.md), and operate directly on the [ndarray](../Chapter-01/1.3._Working_with_ndarray.md) arrays your models already consume.
If you come from scikit-learn, note this module's shape. Most of it is plain functions, not stateful `fit`/`transform` estimators. Each call computes its statistics from the array you pass it, and returns a fresh array. The exceptions are the scalers: `StandardScaler`, `MinMaxScaler`, `MaxAbsScaler`, `RobustScaler`, and `Normalizer`. These do carry the `fit`/`transform` contract across a train/test boundary. Each stores what it learned from the training matrix, so later batches go through the identical map. Either way, the ordering discipline is yours. Split your data first, then fit the scaling. This keeps anything derived from test rows out of training. Every function returns `Result<_, Error>` and validates its input first. Each function rejects an empty dataset, a shape mismatch, or a non-finite value, where that check applies. Read [Error Handling](../Chapter-01/1.6._Error_Handling.md) if you have not.
Read the sections in pipeline order. Split first (4.1). Scale the training features next (4.2). Then encode labels, if your model or loss needs one-hot targets (4.3). The example below runs that exact sequence:
```rust
use ndarray::{Array1, array};
use rustyml::utils::train_test_split::train_test_split;
use rustyml::utils::standardize::{StandardizationAxis, standardize};
use rustyml::utils::label_encoding::to_categorical;
fn main() {
// Six samples, two features on very different scales, two classes.
let x = array![
[1.0, 20.0],
[2.0, 21.0],
[3.0, 19.0],
[4.0, 22.0],
[5.0, 18.0],
[6.0, 23.0],
];
let y: Array1<i32> = array![0, 1, 0, 1, 0, 1];
// 1. Split first, so scaling never sees the test rows.
let (x_train, _x_test, y_train, _y_test) =
train_test_split(x, y, Some(0.34), Some(42)).unwrap();
// 2. Standardize each feature column to zero mean, unit variance.
let x_train = standardize(&x_train, StandardizationAxis::Column).unwrap();
// 3. One-hot encode the integer labels for a categorical loss.
let y_train = to_categorical(&y_train, None).unwrap();
println!("x_train shape: {:?}", x_train.shape());
println!("y_train shape: {:?}", y_train.shape());
}
```
## 4.1. Train-Test Split
[Train-Test Split](./4.1._Train_Test_Split.md) carves your feature matrix and labels into disjoint training and test subsets. This lets evaluation numbers reflect unseen data. `train_test_split` does a plain random partition. `train_test_split_stratified` splits each class independently, so both sides keep the same label proportions. Choose the stratified variant whenever a class is rare, since a plain split can drop that class from one side entirely. `test_size` defaults to `0.3`. The label array is generic over its element type. Passing a `random_state` seed makes the shuffle reproducible (see [Reproducibility and Random Seeds](../Chapter-07/7.1._Reproducibility_and_Random_Seeds.md)).
## 4.2. Standardization and Normalization
[Standardization and Normalization](./4.2._Standardization_and_Normalization.md) are 2 different rescalings that people often confuse. `standardize` computes z-scores. It subtracts the mean, then divides by the standard deviation, so each feature ends up with zero mean and unit variance. `normalize` rescales each row or column to unit norm, under an L1, L2, Max, or custom Lp order.
Each operation also has a stateful form. A stateful form remembers its training statistics, so a test split or a single live sample gets scaled by the numbers the model trained under. The stateful forms are `StandardScaler` and `Normalizer`. `MinMaxScaler` (bounded range), `MaxAbsScaler` (magnitude only, keeping structural zeros), and `RobustScaler` (median and IQR, so outliers do not set the scale) join them.
Column standardization is the everyday choice before any distance-based or gradient-based model. [KNN](../Chapter-02/2.3._K_Nearest_Neighbors.md), [SVM](../Chapter-02/2.5._Support_Vector_Machines.md), [PCA](../Chapter-02/2.10._Principal_Component_Analysis.md), and neural networks all misbehave when features live on different scales. Per-row unit-norm normalization instead suits direction-only vectors, such as TF-IDF rows compared by cosine similarity. Both operations handle a degenerate lane instead of dividing by a vanishing scale. `standardize` forces the divisor to 1.0, so a constant feature centers to zeros. `normalize` leaves a near-zero lane exactly as it is.
## 4.3. Label Encoding
[Label Encoding](./4.3._Label_Encoding.md) converts between the integer labels a dataset ships with and the one-hot matrices a categorical loss expects. `to_categorical` turns a column of non-negative `i32` labels into a one-hot matrix. `to_categorical_with_mapping` does the same for arbitrary hashable labels, strings included, and also returns the label-to-index map, so you can decode later. `to_sparse_categorical` runs the inverse. It reduces a one-hot or softmax-probability matrix to the per-row argmax, as `i32` labels. Use `to_sparse_categorical` to turn a network's [softmax](../Chapter-03/3.2._Dense_Layers_and_Activations.md) output back into predicted classes. If you know Keras, this trio matches its `to_categorical`. One-hot targets are exactly what [Categorical Cross-Entropy](../Chapter-03/3.3._Loss_Functions.md) consumes.