SciRS2 Transform

Production-ready data transformation library for machine learning in Rust (v0.1.0)

This crate provides comprehensive data transformation utilities for the SciRS2 ecosystem (v0.1.0). Following the SciRS2 POLICY, this module is designed to match and exceed the functionality of scikit-learn's preprocessing module while leveraging Rust's performance and safety guarantees with enhanced distributed processing capabilities.

🚀 Features

Data Normalization & Standardization

• Min-Max Scaling: Scale features to [0, 1] or custom ranges
• Z-Score Standardization: Transform to zero mean and unit variance
• Robust Scaling: Use median and IQR for outlier-resistant scaling
• Max Absolute Scaling: Scale by maximum absolute value
• L1/L2 Normalization: Vector normalization

Feature Engineering

• Polynomial Features: Generate polynomial and interaction features
• Power Transformations: Box-Cox and Yeo-Johnson with optimal λ estimation
• Discretization: Equal-width and equal-frequency binning
• Binarization: Convert continuous features to binary
• Log Transformations: Logarithmic feature scaling

Dimensionality Reduction

• PCA: Principal Component Analysis with centering/scaling options
• Truncated SVD: Memory-efficient singular value decomposition
• LDA: Linear Discriminant Analysis for supervised reduction
• t-SNE: Advanced non-linear embedding with Barnes-Hut optimization

Categorical Encoding

• One-Hot Encoding: Sparse and dense representations
• Ordinal Encoding: Label encoding for ordinal categories
• Target Encoding: Supervised encoding with regularization
• Binary Encoding: Memory-efficient encoding for high-cardinality features

Missing Value Imputation

• Simple Imputation: Mean, median, mode, and constant strategies
• KNN Imputation: K-nearest neighbors with multiple distance metrics
• Iterative Imputation: MICE algorithm for multivariate imputation
• Missing Indicators: Track which values were imputed

Feature Selection

• Variance Threshold: Remove low-variance features
• Integration: Seamless integration with all transformers

📦 Installation

Add this to your Cargo.toml:

[dependencies]
scirs2-transform = "0.1.2"

For parallel processing and enhanced performance:

[dependencies]
scirs2-transform = { version = "0.1.2", features = ["parallel"] }

🎯 Quick Start

Basic Normalization

use ndarray::array;
use scirs2_transform::normalize::{normalize_array, NormalizationMethod, Normalizer};

// One-shot normalization
let data = array![[1.0, 2.0, 3.0], 
                 [4.0, 5.0, 6.0],
                 [7.0, 8.0, 9.0]];
let normalized = normalize_array(&data, NormalizationMethod::MinMax, 0)?;

// Fit-transform workflow for reusable transformations
let mut normalizer = Normalizer::new(NormalizationMethod::ZScore, 0);
let train_transformed = normalizer.fit_transform(&train_data)?;
let test_transformed = normalizer.transform(&test_data)?;

Feature Engineering

use scirs2_transform::features::{PolynomialFeatures, PowerTransformer, binarize};

// Generate polynomial features
let data = array![[1.0, 2.0], [3.0, 4.0]];
let poly = PolynomialFeatures::new(2, false, true);
let poly_features = poly.transform(&data)?;

// Power transformations with optimal lambda
let mut transformer = PowerTransformer::yeo_johnson(true);
let gaussian_data = transformer.fit_transform(&skewed_data)?;

// Binarization
let binary_features = binarize(&data, 0.0)?;

Dimensionality Reduction

use scirs2_transform::reduction::{PCA, TSNE};

// PCA for linear dimensionality reduction
let mut pca = PCA::new(2, true, false);
let reduced_data = pca.fit_transform(&high_dim_data)?;
let explained_variance = pca.explained_variance_ratio().unwrap();

// t-SNE for non-linear visualization
let mut tsne = TSNE::new(2, 30.0, 500)?;
let embedding = tsne.fit_transform(&data)?;

Categorical Encoding

use scirs2_transform::encoding::{OneHotEncoder, TargetEncoder};

// One-hot encoding
let mut encoder = OneHotEncoder::new(false, false)?;
let encoded = encoder.fit_transform(&categorical_data)?;

// Target encoding for supervised learning
let mut target_encoder = TargetEncoder::mean_encoding(1.0);
let encoded = target_encoder.fit_transform(&categories, &targets)?;

Missing Value Imputation

use scirs2_transform::impute::{SimpleImputer, KNNImputer, ImputeStrategy};

// Simple imputation
let mut imputer = SimpleImputer::new(ImputeStrategy::Mean);
let complete_data = imputer.fit_transform(&data_with_missing)?;

// KNN imputation
let mut knn_imputer = KNNImputer::new(5)?;
let imputed_data = knn_imputer.fit_transform(&data_with_missing)?;

🔧 Advanced Usage

Pipeline Integration

// Sequential transformations
let mut scaler = Normalizer::new(NormalizationMethod::ZScore, 0);
let mut pca = PCA::new(50, true, false);

// Preprocessing pipeline
let scaled_data = scaler.fit_transform(&raw_data)?;
let reduced_data = pca.fit_transform(&scaled_data)?;

Custom Transformations

use scirs2_transform::features::PowerTransformer;

// Custom power transformation
let mut transformer = PowerTransformer::new("yeo-johnson", true)?;
transformer.fit(&training_data)?;

// Apply to new data
let transformed_test = transformer.transform(&test_data)?;
let original_test = transformer.inverse_transform(&transformed_test)?;

Performance Optimization

// Enable parallel processing for large datasets
use rayon::prelude::*;

// Most transformers automatically use parallel processing when beneficial
let mut pca = PCA::new(100, true, false);
let result = pca.fit_transform(&large_dataset)?; // Automatically parallelized

📊 Performance

SciRS2 Transform is designed for production workloads:

• Memory Efficient: Zero-copy operations where possible
• Parallel Processing: Multi-core support via Rayon
• SIMD Ready: Integration with vectorized operations
• Large Scale: Handles datasets with 100k+ samples and 10k+ features

Benchmarks

🧪 Testing

Run the comprehensive test suite:

# All tests (100 tests)
cargo test

# With output
cargo test -- --nocapture

# Specific module
cargo test normalize::tests

📚 Documentation

• API Documentation
• SciRS2 Project
• Performance Guide
• Migration from sklearn

🔄 Compatibility

Scikit-learn API Compatibility

SciRS2 Transform follows scikit-learn's API conventions:

• fit() / transform() / fit_transform() pattern
• Consistent parameter naming
• Similar default behaviors
• Compatible data formats (via ndarray)

Migration from Scikit-learn

# Python (scikit-learn)
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

// Rust (SciRS2)
use scirs2_transform::normalize::{Normalizer, NormalizationMethod};
let mut scaler = Normalizer::new(NormalizationMethod::ZScore, 0);
let x_scaled = scaler.fit_transform(&x)?;

🏗️ Architecture

Modular Design

scirs2-transform/
├── normalize/     # Data normalization and standardization
├── features/      # Feature engineering utilities  
├── reduction/     # Dimensionality reduction algorithms
├── encoding/      # Categorical data encoding
├── impute/        # Missing value imputation
├── selection/     # Feature selection methods
└── scaling/       # Advanced scaling transformers

Error Handling

Comprehensive error handling with descriptive messages:

use scirs2_transform::{Result, TransformError};

match normalizer.fit_transform(&data) {
    Ok(transformed) => println!("Success!"),
    Err(TransformError::InvalidInput(msg)) => println!("Input error: {}", msg),
    Err(TransformError::TransformationError(msg)) => println!("Transform error: {}", msg),
    Err(e) => println!("Other error: {}", e),
}

🛠️ Development

Building from Source

git clone https://github.com/cool-japan/scirs
cd scirs/scirs2-transform
cargo build --release

Contributing

1. Fork the repository
2. Create a feature branch
3. Add comprehensive tests
4. Ensure all tests pass: cargo test
5. Run clippy: cargo clippy
6. Submit a pull request

📈 Roadmap

Version 0.1.0 (Next)

• Pipeline API for chaining transformations
• Enhanced SIMD acceleration
• Comprehensive benchmarking suite

Version 0.2.0

• UMAP and manifold learning algorithms
• Advanced matrix decomposition methods
• Time series feature extraction

Version 1.0.0

• GPU acceleration
• Distributed processing
• Production monitoring tools

🤝 License

This project is dual-licensed under either:

• MIT License
• Apache License Version 2.0

You may choose to use either license.

🙏 Acknowledgments

• Inspired by scikit-learn
• Built on ndarray
• Powered by Rayon for parallelization

Ready for Production: SciRS2 Transform v0.1.0 provides production-ready data transformation capabilities with performance that meets or exceeds established Python libraries while offering Rust's safety and performance guarantees.