downsample

no_std + alloc + no_deps storage for keeping downsampled metric history over long periods of time.

Provides a method to track data points over long time, aggregating them eventually to keep storage constant. Downsampling is configured with Reducer<T>, a plain function pointer that receives one aggregation window as up to two slices. The aggregation to the next level always happens when one full batch as configured is reached.

Fixed-frequency usage

use downsample::{reducers, FixedFrequencyBuilder};
use core::num::NonZeroUsize;

let nz = |value| NonZeroUsize::new(value).unwrap();

// Store:
// - 100 raw samples
// - 59 samples aggregated from 100 raw samples each
// - 59 samples aggregated from 60 previous-level samples each
// - 23 samples aggregated from 60 previous-level samples each
let mut temperature = FixedFrequencyBuilder::new(nz(100))
    .level(nz(59), nz(100), reducers::average_f32())
    .level(nz(59), nz(60), reducers::median_f32())
    .level(nz(23), nz(60), reducers::average_f32())
    .build();

for value in 0..1000 {
    temperature.push(value as f32);
}

let hourly = temperature.storage().level(2).unwrap();
for value in hourly.iter_oldest_first() {
    let _ = value;
}

let storage = temperature.storage();
assert_eq!(storage.level_metadata(0).unwrap().storage_capacity(), 100 + 100);
assert_eq!(storage.level_metadata(1).unwrap().storage_capacity(), 59 + 60);
assert_eq!(storage.level_metadata(2).unwrap().storage_capacity(), 59 + 60);
assert_eq!(storage.level_metadata(3).unwrap().storage_capacity(), 23 + 0);

Custom reducers

Reducers are regular functions that can can do any work to reduce a large set of data points into a single data point.

use downsample::{FixedFrequencyBuilder, Reducer};
use core::num::NonZeroUsize;

let nz = |value| NonZeroUsize::new(value).unwrap();

fn last(first: &[u32], second: &[u32]) -> u32 {
    second.last().or_else(|| first.last()).copied().unwrap_or_default()
}

let mut samples = FixedFrequencyBuilder::new(nz(16))
    .level(nz(8), nz(4), Reducer::new(last))
    .build();

samples.push(42);

Built-in reducers include:

• reducers::average_f32() and reducers::average_f64()
• integer averages for common integer widths
• reducers::min::<T>() / reducers::minimum::<T>()
• reducers::max::<T>() / reducers::maximum::<T>()
• reducers::median::<T>() for Ord values, returning the lower median for even windows
• reducers::median_f32() and reducers::median_f64(), ignoring NaN values unless all values are NaN
• reducers::first::<T>()
• reducers::last::<T>()

Read/write split

FixedWriter<T> owns the writable storage and the reducer list. The read side is exposed as FixedStorage<T>.

// Continuing the fixed-frequency example above:
let storage = temperature.storage();
let metadata = storage.level_metadata(0).unwrap();
assert_eq!(metadata.capacity(), 100);

Benchmarks

Representative local results from cargo bench --bench mod -- --sample-size 10 --measurement-time 1 --warm-up-time 1. The time column uses Criterion's middle estimate.

Recorded on an AMD Ryzen 9 5900X 12-Core Processor.

Time-based direction

Time-series configuration uses caller-chosen integer ticks for time tacking. This is currently still "under construction".

use downsample::{reducers, TimeSeriesBuilder};
use core::num::NonZeroUsize;

let nz = |value| NonZeroUsize::new(value).unwrap();

// Here `u64` is nanoseconds, but it could also be milliseconds, hardware ticks,
// or any other integer unit chosen by the caller.
let writer = TimeSeriesBuilder::<u64, f32>::new(nz(128))
    .level(nz(60), 1_000_000_000, reducers::average_f32())
    .build();

assert_eq!(writer.storage().level_count(), 1);

The time-series module currently contains the integer-tick storage and writer configuration. Bucket rollover and aggregation policy still need to be completed.

Limitations

• Fixed-frequency bucket sizes must be known when the object is built.
• Fixed-frequency downsample factors are whole numbers: e.g. 100Hz -> 25Hz -> 5Hz is possible with factor 4 and 5, but 100Hz -> 25Hz -> 10Hz isn't as factor 2.5 is no integer.
• This crate is still 0.0.x; API changes are expected.