downsample 0.0.5

keep downsampled history of data over long period of time
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135

[![Crates.io](https://img.shields.io/crates/v/downsample.svg?label=downsample)](https://crates.io/crates/downsample)
[![docs.rs](https://docs.rs/downsample/badge.svg)](https://docs.rs/downsample)
[![pipeline status](https://gitlab.com/xMAC94x/downsample/badges/master/pipeline.svg)](https://gitlab.com/xMAC94x/downsample/commits/master)
[![coverage report](https://gitlab.com/xMAC94x/downsample/badges/master/coverage.svg?min_good=85&min_acceptable=70)](https://gitlab.com/xMAC94x/downsample/commits/master)
[![license](https://img.shields.io/crates/l/downsample)](https://gitlab.com/xMAC94x/downsample/blob/master/LICENSE-MIT)
[![dependencies](https://deps.rs/repo/gitlab/xMAC94x/downsample/status.svg)](https://deps.rs/repo/gitlab/xMAC94x/downsample)

# downsample

`no_std + alloc + no_deps` storage for keeping downsampled metric history over long periods of time.

Provides a method to track data points over long time, aggregating them eventually to keep storage constant.
Downsampling is configured with `Reducer<T>`, a plain function pointer that receives one aggregation window as up to two slices. 
The aggregation to the next level always happens when one full batch as configured is reached.

## Fixed-frequency usage

```rust
use downsample::{reducers, FixedFrequencyBuilder};
use core::num::NonZeroUsize;

let nz = |value| NonZeroUsize::new(value).unwrap();

// Store:
// - 100 raw samples
// - 59 samples aggregated from 100 raw samples each
// - 59 samples aggregated from 60 previous-level samples each
// - 23 samples aggregated from 60 previous-level samples each
let mut temperature = FixedFrequencyBuilder::new(nz(100))
    .level(nz(59), nz(100), reducers::average_f32())
    .level(nz(59), nz(60), reducers::median_f32())
    .level(nz(23), nz(60), reducers::average_f32())
    .build();

for value in 0..1000 {
    temperature.push(value as f32);
}

let hourly = temperature.storage().level(2).unwrap();
for value in hourly.iter_oldest_first() {
    let _ = value;
}

let storage = temperature.storage();
assert_eq!(storage.level_metadata(0).unwrap().storage_capacity(), 100 + 100);
assert_eq!(storage.level_metadata(1).unwrap().storage_capacity(), 59 + 60);
assert_eq!(storage.level_metadata(2).unwrap().storage_capacity(), 59 + 60);
assert_eq!(storage.level_metadata(3).unwrap().storage_capacity(), 23 + 0);
```

## Custom reducers

Reducers are regular functions that can can do any work to reduce a large set of data points into a single data point.

```rust
use downsample::{FixedFrequencyBuilder, Reducer};
use core::num::NonZeroUsize;

let nz = |value| NonZeroUsize::new(value).unwrap();

fn last(first: &[u32], second: &[u32]) -> u32 {
    second.last().or_else(|| first.last()).copied().unwrap_or_default()
}

let mut samples = FixedFrequencyBuilder::new(nz(16))
    .level(nz(8), nz(4), Reducer::new(last))
    .build();

samples.push(42);
```

Built-in reducers include:

- `reducers::average_f32()` and `reducers::average_f64()`
- integer averages for common integer widths
- `reducers::min::<T>()` / `reducers::minimum::<T>()`
- `reducers::max::<T>()` / `reducers::maximum::<T>()`
- `reducers::median::<T>()` for `Ord` values, returning the lower median for even windows
- `reducers::median_f32()` and `reducers::median_f64()`, ignoring `NaN` values unless all values are `NaN`
- `reducers::first::<T>()`
- `reducers::last::<T>()`

## Read/write split

`FixedWriter<T>` owns the writable storage and the reducer list. The read side is exposed as `FixedStorage<T>`.

```rust,ignore
// Continuing the fixed-frequency example above:
let storage = temperature.storage();
let metadata = storage.level_metadata(0).unwrap();
assert_eq!(metadata.capacity(), 100);
```

## Benchmarks

Representative local results from `cargo bench --bench mod -- --sample-size 10 --measurement-time 1 --warm-up-time 1`.
The time column uses Criterion's middle estimate.

| Benchmark | Work | Time | Throughput |
| --- | --- | ---: | ---: |
| `push/single_steady_state_average` | one fixed-frequency push | 4.23 ns | - |
| `push/batch/day_history_average/100000` | 100,000 pushes with day-history levels | 263.71 us | 379.20 Melem/s |
| `push/batch/dense_rollover_average/100000` | 100,000 pushes with frequent rollovers | 478.36 us | 209.05 Melem/s |
| `read/indexed_level_lookup` | 1,000,000 indexed level reads | 1.38 ms | 723.14 Melem/s |
| `reducers/median_f32/split_1024` | median over a wrapped 1,024-value window | 1.65 us | 620.27 Melem/s |

Recorded on an AMD Ryzen 9 5900X 12-Core Processor.

## Time-based direction

Time-series configuration uses caller-chosen integer ticks for time tacking. This is currently still "under construction".

```rust
use downsample::{reducers, TimeSeriesBuilder};
use core::num::NonZeroUsize;

let nz = |value| NonZeroUsize::new(value).unwrap();

// Here `u64` is nanoseconds, but it could also be milliseconds, hardware ticks,
// or any other integer unit chosen by the caller.
let writer = TimeSeriesBuilder::<u64, f32>::new(nz(128))
    .level(nz(60), 1_000_000_000, reducers::average_f32())
    .build();

assert_eq!(writer.storage().level_count(), 1);
```

The time-series module currently contains the integer-tick storage and writer configuration. Bucket rollover and
aggregation policy still need to be completed.

## Limitations

- Fixed-frequency bucket sizes must be known when the object is built.
- Fixed-frequency downsample factors are whole numbers: e.g. 100Hz -> 25Hz -> 5Hz is possible with factor 4 and 5, but 100Hz -> 25Hz -> 10Hz isn't as factor 2.5 is no integer.
- This crate is still `0.0.x`; API changes are expected.