h5rio

A simple HDF5 Rust IO package

h5rio

A small Rust library for writing and reading appendable HDF5 datasets.

h5rio provides a compact interface for two common data-acquisition and simulation output patterns:

• Tables: append typed records to a one-dimensional HDF5 dataset.
• Arrays: append fixed-shape ndarray values along a resizable first axis.

Datasets are buffered in memory and written in chunks, using Blosc/Zlib compression through hdf5-metno. A convenience attribute macro, #[h5type], is provided for defining HDF5-compatible table records.

Features

• Append-only HDF5 table writer for compound Rust types.
• Append-only HDF5 array writer for fixed-shape ndarray entries.
• Configurable buffering through the number of entries stored per chunk.
• Read helpers for complete table and array datasets.
• #[h5type] macro for HDF5-compatible row types.

Data model

TableHdf5Writer<T> creates a dataset with shape

(n_rows,)

where each call to write(value) appends one record of type T.

ArrayHdf5Writer<T> creates a dataset with shape

(n_entries, *item_shape)

where item_shape is fixed at construction time and the leading dimension grows as arrays are appended. For example, a writer created with shape = vec![2, 3] stores successive 2 x 3 arrays in a dataset with shape (n_entries, 2, 3).

Both writers use a resizable leading axis and Blosc/Zlib compression.

Installation

To use the current repository version, depend on the Git repository:

[dependencies]
h5rio = "0.1.0"

# Needed to create/open HDF5 files and by the #[h5type] macro expansion.
hdf5_metno = { package = "hdf5-metno", version = "0.12.3", features = ["blosc-zlib"] }

# Needed when writing ndarray values.
ndarray = "0.17.2"

The library requires an HDF5 installation available to hdf5-metno. The repository includes a Nix development shell that provides HDF5 and the pinned Rust toolchain.

Quick start: writing a table

use std::rc::Rc;

use h5rio::{h5type, read_table, TableHdf5Writer};
use hdf5_metno as hdf5;


#[h5type]
struct Hit {
    event_id: u64,
    sensor_id: u32,
    charge: f32,
}


fn main() -> Result<(), Box<dyn std::error::Error>> {
    let filename = "hits.h5";
    let file = Rc::new(hdf5::File::create(filename)?);

    // Flush every 1024 rows. The dataset is created at /hits.
    let writer = TableHdf5Writer::<Hit>::new(file, "/hits", 1024)?;

    writer.write(Hit {
        event_id: 0,
        sensor_id: 12,
        charge: 18.4,
    })?;

    writer.write(Hit {
        event_id: 0,
        sensor_id: 19,
        charge: 4.7,
    })?;

    // Optional: data are also flushed when the writer is dropped.
    writer.flush()?;

    let hits = read_table::<Hit>(filename, "/hits")?;

    println!("Read {} hits", hits.len());

    Ok(())
}

The #[h5type] attribute expands the struct definition with the derives and representation required for table storage:

#[derive(hdf5::H5Type, Clone, PartialEq, Debug)]
#[repr(C)]

Writing fixed-shape arrays

ArrayHdf5Writer is useful for storing successive waveforms, images, response maps, or any stream of arrays with the same shape.

use std::rc::Rc;

use h5rio::{read_array, ArrayHdf5Writer};
use hdf5_metno as hdf5;
use ndarray::arr2;


fn main() -> Result<(), Box<dyn std::error::Error>> {
    let filename = "waveforms.h5";
    let file = Rc::new(hdf5::File::create(filename)?);

    // Each appended entry has shape (2, 3); two entries are buffered per chunk.
    let writer = ArrayHdf5Writer::<f32>::new(
        file,
        "/waveforms",
        2,
        vec![2, 3],
    )?;

    writer.write(arr2(&[
        [0.0, 1.0, 0.5],
        [0.2, 0.7, 0.1],
    ]))?;

    writer.write(arr2(&[
        [1.1, 1.4, 1.0],
        [0.8, 0.3, 0.0],
    ]))?;

    writer.flush()?;

    let waveforms = read_array::<f32>(filename, "/waveforms")?;

    assert_eq!(waveforms.shape(), &[2, 2, 3]);

    Ok(())
}

API overview

Writers

The chunk_size constructor argument is the number of appended entries buffered before a write to disk. For table datasets, one entry is one record. For array datasets, one entry is one full array with the configured item_shape.

Readers

pub fn read_table<T>(filename: &str, dataset: &str) -> hdf5::Result<Vec<T>>
pub fn read_array<T>(filename: &str, dataset: &str) -> hdf5::Result<ArrayD<T>>

The current readers load the complete dataset into memory.

Macro

#[h5type]
struct Row {
    value: f64,
}

#[h5type] is intended for plain record structs used with TableHdf5Writer<T> and read_table<T>.

Development

With Nix

The recommended development environment is provided by the flake. It includes the pinned Rust toolchain, HDF5, cargo-nextest, just, bacon, and Rust Analyzer support.

nix develop
just build
just test

Without Nix

Install HDF5 and a compatible Rust toolchain, then run:

cargo build
cargo test

The repository pins Rust 1.95.0 in rust-toolchain.toml. The project's justfile uses cargo nextest for its test recipes.

Testing and continuous integration

Tests cover:

• construction of valid and invalid writers;
• table and array round trips;
• flushing incomplete chunks when writers are dropped;
• the derives and C-compatible memory layout generated by #[h5type].

On every push and pull request, the GitHub Actions workflow builds and tests the project inside the Nix development environment with warnings treated as errors.

License

This project is licensed under the terms of the GNU General Public License v3.0.