ml-cellar 0.2.0

CLI of ML model registry for minimum MLOps
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109


# Documentation for template.md
## Summary

When you run `ml-cellar docs {path}`, `ml-cellar` generates the document of results from `template.md`.
This feature semi-automates the documentation of information about trained models, which is necessary for running MLOps.
As a result, it reduces the manual work required to operate MLOps.
By reducing this manual effort, it also helps create an environment where AI models are less likely to be neglected.

## Get started
### 1. Set configuration

First, please set the configuration `config.toml` in your rack.

```toml
[document]
template_file = "template.md"
result_file = "result.json"
```

If you want to know about `template_file` and `result_file` in this configuration, please see [the docs string](/src/rack.rs).

### 2. Make template file

Please write template file like `template.md`.

- You can use JSON keys by writing `{key}`.
- You can use macros by writing `{{version}}` from the directory name.

```markdown

### {{version}}

- Write the summary for the model
- Why you updated the model
- What dataset you added
- What you fixed in the algorithm
- What you updated in training parameters

<details>
<summary> Evaluation results </summary>

- Performance
  - Device: {performance.device}
  - Inference time(ms): {performance.time_ms}
  - Training time: {performance.gpu_num} GPU * {performance.batch_size} Batchsize * {performance.training_days} days
- Dataset
  - name: {dataset.name}
  - Version: {dataset.version}
  - Frame number: {dataset.frame_number}
- mAP for test dataset

| Metric         | Value              |
| -------------- | ------------------ |
| mAP (0.5:0.95) | {ap.map.50_95.all} |
| mAP (0.5)      | {ap.map.50.all}    |

- Per-Class AP for test dataset

| Class   | Samples                                  | Accuracy               |
| ------- | ---------------------------------------- | ---------------------- |
| person  | {test_dataset.per_class_samples.person}  | {ap.per_class.person}  |
| bicycle | {test_dataset.per_class_samples.bicycle} | {ap.per_class.bicycle} |
| car     | {test_dataset.per_class_samples.car}     | {ap.per_class.car}     |
| dog     | {test_dataset.per_class_samples.dog}     | {ap.per_class.dog}     |

</details>
```

### 3. Generate document of evaluation results

- Please run `ml-cellar docs {path}`, then the documentation is generated automatically as below.

```markdown
### test_docs/base/0.1

- Write the summary for the model
- Why you updated the model
- What dataset you added
- What you fixed in the algorithm
- What you updated in training parameters

<details>
<summary> Evaluation results </summary>

- Performance
  - Device: RTX4090
  - Inference time(ms): 7.87
  - Training time: 2 GPU * 32 Batchsize * 4.7 days
- Dataset
  - name: myCOCO
  - Version: 2.1.1
  - Frame number: 21321
- mAP for test dataset

| Metric         | Value      |
| -------------- | ---------- |
| mAP (0.5:0.95) | 0.40453625 |
| mAP (0.5)      | 0.598      |

- Per-Class AP for test dataset

| Class   | Samples | Accuracy |
| ------- | ------- | -------- |
| person  | 647     | 0.5517   |
| bicycle | 916     | 0.2899   |
| car     | 663     | 0.3941   |
| dog     | 988     | 0.586    |
```