Crate reductionist

Expand description

Reductionist is a web service that provides an API to perform reductions on numerical data stored in an S3-compatible object store. By implementing these reductions in the storage system the volume of data that needs to be transferred to the end user is vastly reduced, leading to faster computations.

The work is funded by the ExCALIBUR project and is done in collaboration with the University of Reading.

This site provides documentation for the Reductionist source code. Documentation for the Reductionist application is hosted on GitHub.

This is a performant implementation of the Active Storage Server. The original Python functional prototype is available here.

§Features

Reductionist provides the following features:

HTTP(S) API with JSON request data
Access to data stored in S3-compatible storage
Basic numerical operations on multi-dimensional arrays (count, min, max, select, sum)
Perform calculations on a selection/slice of an array
Perform calculations allowing for missing data
Compressed data (GZip, Zlib)
Filtered data (byte shuffle)
Data with non-native byte order (endianness)
Server resource (CPU, memory, files) management
Prometheus metrics
Tracing with an option to send data to Jaeger
Ansible-based containerised deployment

Reductionist is built on top of a number of open source components.

Tokio, the most popular asynchronous Rust runtime.
Axum web framework, built by the Tokio team. Axum performs well in various benchmarks and is built on top of various popular components, including the hyper HTTP library.
Serde performs (de)serialisation of JSON request and response data.
AWS SDK for S3 is used to interact with S3-compatible object stores.
ndarray provides NumPy-like n-dimensional arrays used in numerical computation.

PyActiveStorage is a Python library which performs reductions on numerical data in data sources such as netCDF4. It has support for delegating computation to Reductionist when the data is stored in an S3-compatible object store.

Modules§

app: Active Storage server API
app_state
array: Functions and utilities for working with ndarray objects.
chunk_cache
chunk_downloader_http
chunk_downloader_s3
chunk_store
cli: Command Line Interface (CLI) arguments.
compression: (De)compression support.
error: Error handling.
filter_pipeline: Compression and filter pipeline.
filters: Filter implementations.
metrics: Prometheus metrics
models: Data types and associated functions and methods
operation: Interface for Active Storage operations
operations: Numerical operations.
resource_manager: Resource management
s3_client: A simplified S3 client that supports downloading objects. It attempts to hide the complexities of working with the AWS SDK for S3.
server: Web server
tracing: Tracing (logging)
types: Various common types
validated_json: Axum extractor that deserialises and validates JSON