mlinrust 0.1.0

A machine learning library in Rust from scratch.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106

<div align="center">
    <h1>
        Machine Learning in Rust
	</h1>
</div>


Learn the Rust programming language through implementing classic machine learning algorithms. This project is self-completed without relying on any third-party libraries, serving as a **bootstrap machine learning library**.

❗❗❗：Actively seeking code reviews and welcome suggestions on fixing bugs or code refactoring. Please feel free to share your ideas. Happy to accept advice!

## Basics

1. **NdArray Module**, just as the name. It has implemented ``broadcast``, ``matrix operations``, ``permute`` and etc. in arbitrary dimension. SIMD is used in matrix multiplication thanks to auto vectorizing by Rust.
2. **Dataset Module**, supporting customized loading data, re-format, ``normalize``, ``shuffle`` and ``Dataloader``. Several popular dataset pre-processing recipes are available.

## Algorithms

1. **Decision Tree**, supporting both classification and regression tasks. Info gains like ``gini`` or ``entropy`` are provided.
2. **Logistic Regression**, supporting regularization (``Lasso``, ``Ridge`` and ``L-inf``)
3. **Linear Regression**, same as logistic regression, but for regression tasks.
4. **Naive Bayes**, free to handle discrete or continuous feature values.
5. **SVM**, with linear kernel using SGD and Hinge Loss to optimize.
6. **nn Module**, containing ``linear(MLP)`` and some ``activation`` functions which could be freely stacked and optimized by gradient back propagations.
6. **KNN**, supporting both ``KdTree`` and vanilla ``BruteForceSearch``.
6. **K-Means**, clustering data with an unsupervised learning approach

## Start

Let's use KNN algorithm to solve a classification task. More examples can be found in ``examples`` directory.

1. create some synthetic data for tests

   ```rust
   use std::collections::HashMap;
   
   let features = vec![
       vec![0.6, 0.7, 0.8],
       vec![0.7, 0.8, 0.9],
       vec![0.1, 0.2, 0.3],
   ];
   let labels = vec![0, 0, 1];
   // so it is a binary classifiction task, 0 is for the large label, 1 is for the small label
   let mut label_map = HashMap::new();
   label_map.insert(0, "large".to_string());
   label_map.insert(1, "small".to_string());
   ```

2. convert the data to the ``dataset``

   ```rust
   use mlinrust::dataset::Dataset;
   
   let dataset = Dataset::new(features, labels, Some(label_map));
   ```

3. split the dataset into ``train`` and ``valid`` sets and normalize them by Standard normalization

   ```rust
   let mut temp =  dataset.split_dataset(vec![2.0, 1.0], 0); // [2.0, 1.0] is the split fraction, 0 is the seed
   let (mut train_dataset, mut valid_dataset) = (temp.remove(0), temp.remove(0));
   
   use mlinrust::dataset::utils::{normalize_dataset, ScalerType};
   
   normalize_dataset(&mut train_dataset, ScalerType::Standard);
   normalize_dataset(&mut valid_dataset, ScalerType::Standard);
   ```

4. build and train our KNN model using ``KdTree``

   ```rust
   use mlinrust::model::knn::{KNNAlg, KNNModel, KNNWeighting};
   
   // KdTree is one implementation of KNN; 1 defines the k of neighbours; Weighting decides the way of ensemble prediction; train_dataset is for training KNN; Some(2) is the param of minkowski distance
   let model = KNNModel::new(KNNAlg::KdTree, 1, Some(KNNWeighting::Distance), train_dataset, Some(2));
   ```

5. evaluate the model

   ```rust
   use mlinrust::utils::evaluate;
   
   let (correct, acc) = evaluate(&valid_dataset, &model);
   println!("evaluate results\ncorrect {correct} / total {}, acc = {acc:.5}", test_dataset.len());
   ```
   

## Todo

1. model weights serialization for saving and loading
2. Boosting/bagging
1. matrix multiplication with multi threads
3. refactor codes, sincerely request for comments from senior developers

## Reference

1. [scikit-learn](https://scikit-learn.org/stable/)
2. The book, [机器学习西瓜书](https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/MLbook2016.htm) by [Prof. Zhihua Zhou](https://cs.nju.edu.cn/zhouzh/index.htm)

## Thanks

The rust community. I received many help from [rust-lang Discord](https://discord.gg/rust-lang).

## License

Under GPL-v3 license. And commercial use is strictly prohibited.