# h5rio
A simple HDF5 Rust IO package
# h5rio
[](https://github.com/gonzaponte/h5rio/actions/workflows/test.yml)
[](LICENSE)
A small Rust library for writing and reading appendable HDF5 datasets.
`h5rio` provides a compact interface for two common data-acquisition and
simulation output patterns:
- **Tables**: append typed records to a one-dimensional HDF5 dataset.
- **Arrays**: append fixed-shape `ndarray` values along a resizable first axis.
Datasets are buffered in memory and written in chunks, using Blosc/Zlib
compression through [`hdf5-metno`](https://github.com/matthias314/hdf5-metno).
A convenience attribute macro, `#[h5type]`, is provided for defining
HDF5-compatible table records.
## Features
- Append-only HDF5 table writer for compound Rust types.
- Append-only HDF5 array writer for fixed-shape `ndarray` entries.
- Configurable buffering through the number of entries stored per chunk.
- Read helpers for complete table and array datasets.
- `#[h5type]` macro for HDF5-compatible row types.
## Data model
`TableHdf5Writer<T>` creates a dataset with shape
```text
(n_rows,)
```
where each call to `write(value)` appends one record of type `T`.
`ArrayHdf5Writer<T>` creates a dataset with shape
```text
(n_entries, *item_shape)
```
where `item_shape` is fixed at construction time and the leading dimension
grows as arrays are appended. For example, a writer created with
`shape = vec![2, 3]` stores successive `2 x 3` arrays in a dataset with shape
`(n_entries, 2, 3)`.
Both writers use a resizable leading axis and Blosc/Zlib compression.
## Installation
To use the current repository version, depend on the Git repository:
```toml
[dependencies]
h5rio = "0.1.0"
# Needed to create/open HDF5 files and by the #[h5type] macro expansion.
hdf5_metno = { package = "hdf5-metno", version = "0.12.3", features = ["blosc-zlib"] }
# Needed when writing ndarray values.
ndarray = "0.17.2"
```
The library requires an HDF5 installation available to `hdf5-metno`. The
repository includes a Nix development shell that provides HDF5 and the pinned
Rust toolchain.
## Quick start: writing a table
```rust
use std::rc::Rc;
use h5rio::{h5type, read_table, TableHdf5Writer};
use hdf5_metno as hdf5;
#[h5type]
struct Hit {
event_id: u64,
sensor_id: u32,
charge: f32,
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
let filename = "hits.h5";
let file = Rc::new(hdf5::File::create(filename)?);
// Flush every 1024 rows. The dataset is created at /hits.
let writer = TableHdf5Writer::<Hit>::new(file, "/hits", 1024)?;
writer.write(Hit {
event_id: 0,
sensor_id: 12,
charge: 18.4,
})?;
writer.write(Hit {
event_id: 0,
sensor_id: 19,
charge: 4.7,
})?;
// Optional: data are also flushed when the writer is dropped.
writer.flush()?;
let hits = read_table::<Hit>(filename, "/hits")?;
println!("Read {} hits", hits.len());
Ok(())
}
```
The `#[h5type]` attribute expands the struct definition with the derives and
representation required for table storage:
```rust
#[derive(hdf5::H5Type, Clone, PartialEq, Debug)]
#[repr(C)]
```
## Writing fixed-shape arrays
`ArrayHdf5Writer` is useful for storing successive waveforms, images,
response maps, or any stream of arrays with the same shape.
```rust
use std::rc::Rc;
use h5rio::{read_array, ArrayHdf5Writer};
use hdf5_metno as hdf5;
use ndarray::arr2;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let filename = "waveforms.h5";
let file = Rc::new(hdf5::File::create(filename)?);
// Each appended entry has shape (2, 3); two entries are buffered per chunk.
let writer = ArrayHdf5Writer::<f32>::new(
file,
"/waveforms",
2,
vec![2, 3],
)?;
writer.write(arr2(&[
[0.0, 1.0, 0.5],
[0.2, 0.7, 0.1],
]))?;
writer.write(arr2(&[
[1.1, 1.4, 1.0],
[0.8, 0.3, 0.0],
]))?;
writer.flush()?;
let waveforms = read_array::<f32>(filename, "/waveforms")?;
assert_eq!(waveforms.shape(), &[2, 2, 3]);
Ok(())
}
```
## API overview
### Writers
| `TableHdf5Writer<T>` | Append scalar records of an HDF5-compatible type | `new`, `write`, `flush` |
| `ArrayHdf5Writer<T>` | Append fixed-shape `ndarray::Array` entries | `new`, `write`, `flush` |
The `chunk_size` constructor argument is the number of appended **entries**
buffered before a write to disk. For table datasets, one entry is one record.
For array datasets, one entry is one full array with the configured
`item_shape`.
### Readers
```rust
pub fn read_table<T>(filename: &str, dataset: &str) -> hdf5::Result<Vec<T>>
pub fn read_array<T>(filename: &str, dataset: &str) -> hdf5::Result<ArrayD<T>>
```
The current readers load the complete dataset into memory.
### Macro
```rust
#[h5type]
struct Row {
value: f64,
}
```
`#[h5type]` is intended for plain record structs used with
`TableHdf5Writer<T>` and `read_table<T>`.
## Development
### With Nix
The recommended development environment is provided by the flake. It includes
the pinned Rust toolchain, HDF5, `cargo-nextest`, `just`, `bacon`, and
Rust Analyzer support.
```bash
nix develop
just build
just test
```
### Without Nix
Install HDF5 and a compatible Rust toolchain, then run:
```bash
cargo build
cargo test
```
The repository pins Rust `1.95.0` in `rust-toolchain.toml`. The project's
`justfile` uses `cargo nextest` for its test recipes.
## Testing and continuous integration
Tests cover:
- construction of valid and invalid writers;
- table and array round trips;
- flushing incomplete chunks when writers are dropped;
- the derives and C-compatible memory layout generated by `#[h5type]`.
On every push and pull request, the GitHub Actions workflow builds and tests
the project inside the Nix development environment with warnings treated as
errors.
## License
This project is licensed under the terms of the [GNU General Public License
v3.0](LICENSE).