mpi-rma 0.1.0

Typed one-sided communication (RMA) and fixed-slot ring transport on top of rsmpi.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

# Benchmark tables

Median over repeats with a 95% bootstrap CI in brackets. Raw measurements: `compare.tsv`, `loss.tsv`, `scale.tsv`.

### Goodput, quiet, MiB/s delivered

| payload | ring-safe | ring-raw | p2p-send | p2p-bsend | ring-safe / p2p-send |
|--------:|----------:|----------:|----------:|----------:|----------:|
| 64 B | 102 [99.2, 103] | 107 [106, 108] | 63.3 [63.2, 63.6] | 63.1 [62.7, 63.3] | 1.61x |
| 512 B | 577 [568, 582] | 564 [554, 567] | 180 [177, 182] | 308 [306, 308] | 3.20x |
| 4 KiB | 1,135 [1,116, 1,157] | 1,116 [1,099, 1,122] | 626 [620, 629] | 566 [412, 579] | 1.81x |
| 32 KiB | 588 [582, 591] | 883 [719, 1,192] | 2,654 [2,632, 2,663] | 877 [871, 879] | 0.22x |
| 256 KiB | 494 [491, 495] | 463 [455, 465] | 2,867 [2,833, 2,877] | 1,054 [1,046, 1,061] | 0.17x |

### Goodput, under 64 KiB background traffic, MiB/s delivered

| payload | ring-safe | ring-raw | p2p-send | p2p-bsend | ring-safe / p2p-send |
|--------:|----------:|----------:|----------:|----------:|----------:|
| 64 B | 52 [51.6, 53.4] | 51.8 [51.2, 57.6] | 30.5 [28.9, 35.3] | 30.3 [28.9, 35.3] | 1.70x |
| 512 B | 276 [272, 284] | 267 [261, 276] | 69.1 [68.2, 72.3] | 173 [153, 182] | 3.99x |
| 4 KiB | 696 [683, 740] | 716 [691, 762] | 255 [253, 267] | 278 [228, 287] | 2.73x |
| 32 KiB | 354 [347, 359] | 349 [336, 367] | 1,244 [1,203, 1,273] | 568 [544, 640] | 0.28x |
| 256 KiB | 311 [294, 316] | 276 [269, 316] | 1,524 [1,513, 1,535] | 722 [410, 774] | 0.20x |

### One-way latency, quiet, µs

| payload | ring-safe | ring-raw | p2p-send | ring-safe / p2p-send |
|--------:|----------:|----------:|----------:|----------:|
| 64 B | 1.77 [1.75, 1.78] | 1.28 [1.27, 1.35] | 1.89 [1.88, 1.91] | 0.94x |
| 512 B | 2.63 [2.61, 2.65] | 2.12 [2.11, 2.17] | 2.83 [2.81, 2.87] | 0.93x |
| 4 KiB | 7.05 [7.01, 7.14] | 6.5 [6.46, 6.58] | 6.65 [6.6, 6.7] | 1.06x |
| 32 KiB | 49.3 [49.1, 49.8] | 49 [48.9, 49.1] | 12.9 [12.6, 13.1] | 3.82x |
| 256 KiB | 840 [833, 845] | 836 [830, 841] | 64.9 [64.7, 66] | 12.94x |

### One-way latency, under 64 KiB background traffic, µs

| payload | ring-safe | ring-raw | p2p-send | ring-safe / p2p-send |
|--------:|----------:|----------:|----------:|----------:|
| 64 B | 3.29 [3.06, 3.46] | 2.38 [2.25, 2.42] | 4.77 [4.48, 4.83] | 0.69x |
| 512 B | 4.49 [4.15, 4.62] | 3.6 [3.21, 3.67] | 6.38 [5.92, 6.6] | 0.70x |
| 4 KiB | 11.8 [11.2, 12.2] | 10.6 [10.3, 10.9] | 14.2 [13.6, 14.6] | 0.83x |
| 32 KiB | 81.3 [77.8, 84.1] | 78.7 [77.4, 83.6] | 24.7 [23.7, 25.9] | 3.29x |
| 256 KiB | 1,281 [1,263, 1,305] | 1,280 [1,238, 1,303] | 115 [110, 117] | 11.19x |

### Worst one-way latency, quiet, µs

| payload | ring-safe | ring-raw | p2p-send | ring-safe / p2p-send |
|--------:|----------:|----------:|----------:|----------:|
| 64 B | 14.8 [12, 16.9] | 12 [8.6, 15.9] | 7.21 [6.12, 8.81] | 2.06x |
| 512 B | 10.9 [9.62, 12.1] | 13.8 [10.1, 19.6] | 9.06 [7.13, 15.6] | 1.20x |
| 4 KiB | 23.6 [17.6, 31.4] | 16.1 [15.4, 22.9] | 39.5 [36.2, 41.9] | 0.60x |
| 32 KiB | 80.9 [80.6, 82.8] | 80.5 [76.4, 93.4] | 47.2 [44.4, 49.8] | 1.72x |
| 256 KiB | 1,250 [1,236, 1,265] | 1,221 [1,208, 1,227] | 353 [348, 355] | 3.54x |

### Worst one-way latency, under 64 KiB background traffic, µs

| payload | ring-safe | ring-raw | p2p-send | ring-safe / p2p-send |
|--------:|----------:|----------:|----------:|----------:|
| 64 B | 25.1 [11.8, 69.2] | 47.2 [18.3, 96.6] | 51.6 [14.2, 117] | 0.49x |
| 512 B | 66.3 [19.2, 440] | 54.8 [25.4, 68.1] | 56.7 [19.1, 129] | 1.17x |
| 4 KiB | 93.4 [62.5, 750] | 68 [47.1, 82.8] | 48.8 [42.1, 106] | 1.92x |
| 32 KiB | 177 [133, 236] | 227 [131, 967] | 151 [76.1, 201] | 1.17x |
| 256 KiB | 2,353 [1,954, 2,741] | 2,255 [2,081, 3,729] | 1,070 [642, 2,038] | 2.20x |

### Raw mode, quiet: send side vs delivery

| payload | raw inject (msg/s) | raw goodput (msg/s) | delivered (%) | safe goodput (msg/s) |
|---|---:|---:|---:|---:|
| 64 B | 2,321,842 [2,032,540, 2,439,104] | 1,758,047 [1,728,887, 1,770,474] | 75.8 [71.6, 87.4] | 1,668,969 [1,624,999, 1,681,787] |
| 512 B | 1,686,700 [1,605,958, 1,700,694] | 1,154,788 [1,134,646, 1,162,065] | 69.1 [68.5, 70.7] | 1,181,844 [1,163,267, 1,191,316] |
| 4 KiB | 432,290 [429,332, 436,470] | 285,711 [281,389, 287,333] | 66.2 [64.8, 67.1] | 290,519 [285,794, 296,080] |
| 32 KiB | 43,605 [38,789, 47,070] | 28,257 [22,994, 38,133] | 65.6 [49, 100] | 18,811 [18,619, 18,918] |
| 256 KiB | 4,421 [4,362, 4,441] | 1,850 [1,821, 1,858] | 53.1 [46.5, 53.5] | 1,975 [1,964, 1,980] |

### Loss by offered rate, quiet (%)

| depth | flat out | 2.0 M/s | 1.0 M/s | 500 k/s | 200 k/s | 100 k/s | 50 k/s | 20 k/s |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| depth 2 | 64.7 [56.5, 66.7] | 66.3 [48, 68.4] | 60.9 [52.4, 66.5] | 61.1 [52.7, 67.9] | 60.6 [57.3, 64.8] | 23.7 [12.1, 40.1] | 2.01 [0.155, 2.76] | 0.0454 [0.0385, 0.0723] |
| depth 8 | 58.3 [53.9, 69] | 69.2 [63.3, 72.2] | 64.3 [57.9, 69.3] | 67.8 [56.1, 70] | 54.2 [48.8, 65.1] | 15.7 [4.72, 25.3] | 1.74 [0.0852, 2.16] | 0.0101 [0.00152, 0.171] |
| depth 32 | 63 [53.3, 72.8] | 65.8 [55.9, 69.8] | 67.2 [58, 72.2] | 61.2 [57.9, 63.1] | 52.4 [50.8, 61.7] | 23.8 [16.7, 29.8] | 0.192 [0.00125, 2.85] | 0 [0, 0] |
| depth 128 | 65.3 [49.7, 72.8] | 72.9 [58.5, 78.4] | 59.1 [54.9, 64.5] | 69.2 [51.9, 74] | 52.2 [48.7, 58.9] | 16.6 [8.53, 23.9] | 0.917 [0.106, 1.84] | 0 [0, 0] |

### Loss by offered rate, under 64 KiB background traffic (%)

| depth | flat out |
|---:|---:|
| depth 2 | 57.2 [51.9, 61.5] |
| depth 8 | 58.8 [57.3, 60] |
| depth 32 | 58.4 [57.3, 64.5] |
| depth 128 | 59.4 [56.2, 64.7] |

### Safe mode at width, quiet

| ranks | delivered (%) | max blocked sends on one rank |
|---:|---:|---:|
| 2 | 100 [100, 100] | 2618 |
| 4 | 100 [100, 100] | 3319 |
| 8 | 100 [100, 100] | 5428 |
| 16 | 100 [100, 100] | 6192 |
| 24 | 100 [100, 100] | 6804 |

### Safe mode at width, under 64 KiB background traffic

| ranks | delivered (%) | max blocked sends on one rank |
|---:|---:|---:|
| 2 | 100 [100, 100] | 2523 |
| 4 | 100 [100, 100] | 3664 |
| 8 | 100 [100, 100] | 4235 |
| 16 | 100 [100, 100] | 6438 |
| 24 | 100 [100, 100] | 6676 |