Crate reductionist

source ·
Expand description

Reductionist is a web service that provides an API to perform reductions on numerical data stored in an S3-compatible object store. By implementing these reductions in the storage system the volume of data that needs to be transferred to the end user is vastly reduced, leading to faster computations.

The work is funded by the ExCALIBUR project and is done in collaboration with the University of Reading.

This site provides documentation for the Reductionist source code. Documentation for the Reductionist application is hosted on GitHub.

This is a performant implementation of the Active Storage Server. The original Python functional prototype is available here.

§Features

Reductionist provides the following features:

  • HTTP(S) API with JSON request data
  • Access to data stored in S3-compatible storage
  • Basic numerical operations on multi-dimensional arrays (count, min, max, select, sum)
  • Perform calculations on a selection/slice of an array
  • Perform calculations allowing for missing data
  • Compressed data (GZip, Zlib)
  • Filtered data (byte shuffle)
  • Data with non-native byte order (endianness)
  • Server resource (CPU, memory, files) management
  • Prometheus metrics
  • Tracing with an option to send data to Jaeger
  • Ansible-based containerised deployment

Reductionist is built on top of a number of open source components.

  • Tokio, the most popular asynchronous Rust runtime.
  • Axum web framework, built by the Tokio team. Axum performs well in various benchmarks and is built on top of various popular components, including the hyper HTTP library.
  • Serde performs (de)serialisation of JSON request and response data.
  • AWS SDK for S3 is used to interact with S3-compatible object stores.
  • ndarray provides NumPy-like n-dimensional arrays used in numerical computation.
  • PyActiveStorage is a Python library which performs reductions on numerical data in data sources such as netCDF4. It has support for delegating computation to Reductionist when the data is stored in an S3-compatible object store.

Modules§

  • app
    Active Storage server API
  • array
    Functions and utilities for working with ndarray objects.
  • cli
    Command Line Interface (CLI) arguments.
  • compression
    (De)compression support.
  • error
    Error handling.
  • filter_pipeline
    Compression and filter pipeline.
  • filters
    Filter implementations.
  • metrics
    Prometheus metrics
  • models
    Data types and associated functions and methods
  • operation
    Interface for Active Storage operations
  • operations
    Numerical operations.
  • resource_manager
    Resource management
  • s3_client
    A simplified S3 client that supports downloading objects. It attempts to hide the complexities of working with the AWS SDK for S3.
  • server
    Web server
  • tracing
    Tracing (logging)
  • types
    Various common types
  • validated_json
    Axum extractor that deserialises and validates JSON