rten 0.25.0

Machine learning runtime
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

# Debugging

This document provides strategies for debugging incorrect/different outputs in
RTen compared to other runtimes.

## Inspecting models

[Netron](https://netron.app) is an online tool, also available as an Electron
app for visualizing ONNX models.

## Comparing against ONNX Runtime

[ONNX Runtime](https://onnxruntime.ai) ("ORT") is the most mature implementation
of the ONNX specification, and is often used as a reference for correctness and
performance testing.

The general steps to use ORT to compare and debug unexpected output from RTen
are:

1. Create a Python script to execute the model with ORT, and a
   corresponding Rust binary to execute the model with RTen.

2. Verify that the model produces the expected results with ORT.

3. Verify that the inputs to the model, after all preprocessing and conversion
   to tensors, are the same in RTen and ORT.

4. Verify that there are significant differences in the RTen vs ORT output.

5. Compare the values of intermediate outputs in the graph to find where
   significant differences begin to arise. For small tensors, the values can
   simply be printed and inspected by eye. Most tensors will be larger however
   and so it is useful to get statistics of a comparison. A typical approach is:

   1. Run the model specifying an intermediate node as an output.

      RTen allows any node in the graph to be specified as an output. ORT on the
      other hand only allows nodes specified in the model's output list to be
      fetched as an output from a model run. The
      `tools/add-node-outputs-to-model.py` script works around this limitation of
      ORT by producing a modified ONNX model that lists every node in the graph
      in the output list.

   2. Write out the resulting intermediate tensors. The `Tensor::write` method
      can be used for this in RTen and the `write_tensor` function in
      `tools/debug_utils.py` in Python.

   3. Compare the resulting tensors. `tools/compare-tensors.py` compares tensor
      shapes and reports statistics on the absolute difference between
      corresponding values.

   Repeat steps 1-3 until you have identified where in the model discrepancies
   begin to arise. Note that very small differences for individual values,
   eg. on the order of 5 or 6 places after the decimal point, are normal for
   certain operations due to implementation differences.