hdf5-pure

Pure-Rust HDF5 reader/writer. No C dependencies, no build scripts, WASM-compatible.

Features

• Write HDF5 files with datasets, groups, attributes, and nested hierarchies
• Read HDF5 files (v0/v1/v2/v3 superblocks, v1/v2 object headers, contiguous/chunked/compact storage)
• SWMR (single-writer / multiple-reader) append and refreshing read for 1-D unlimited datasets, interoperable with the reference C library and h5py
• No C dependencies — compiles to wasm32-unknown-unknown with --no-default-features
• MATLAB v7.3 compatible — userblock support, fixed-length ASCII attributes, variable-length string arrays, object references
• Deflate, shuffle, and scale-offset (lossless integer / lossy float) compression
• Compound types, enumerations, array types
• Complex number datasets (as compound {real, imag})

Quick start

Writing

use hdf5_pure::{FileBuilder, AttrValue};

let mut builder = FileBuilder::new();

// Datasets
builder.create_dataset("temperature")
    .with_f64_data(&[22.5, 23.1, 21.8])
    .with_shape(&[3]);

// Groups with nested datasets
let mut grp = builder.create_group("sensors");
grp.create_dataset("pressure").with_f32_data(&[101.3, 101.5]);
grp.set_attr("location", AttrValue::AsciiString("lab_a".into()));
builder.add_group(grp.finish());

// Attributes on the root group
builder.set_attr("version", AttrValue::I64(2));

builder.write("output.h5").unwrap();

Reading

use hdf5_pure::File;

let file = File::open("output.h5").unwrap();
let ds = file.dataset("temperature").unwrap();

println!("shape: {:?}", ds.shape().unwrap());    // [3]
println!("data:  {:?}", ds.read_f64().unwrap());  // [22.5, 23.1, 21.8]

let attrs = file.root().attrs().unwrap();
println!("version: {:?}", attrs.get("version"));  // Some(I64(2))

In-memory (WASM)

use hdf5_pure::FileBuilder;

let mut builder = FileBuilder::new();
builder.create_dataset("x").with_f64_data(&[1.0, 2.0]);

let bytes: Vec<u8> = builder.finish().unwrap(); // no filesystem needed

SWMR (single writer, multiple readers)

A single process can append to an unlimited dataset in place while other processes read it concurrently. The writer appends chunks and flushes in dependency order so readers only ever observe a consistent prefix; readers re-read to pick up new data. This interoperates with the reference HDF5 C library and h5py in both directions.

The dataset must have one unlimited dimension and be chunked (it is indexed by an Extensible Array, which the latest format selects automatically). Create it the usual way:

use hdf5_pure::FileBuilder;

let mut builder = FileBuilder::new();
builder.create_dataset("log")
    .with_i32_data(&[0, 1, 2])   // initial rows
    .with_shape(&[3])
    .with_maxshape(&[u64::MAX])  // one unlimited dimension
    .with_chunks(&[1]);
builder.write("stream.h5").unwrap();

Append in place (each call flushes durably; the file stays valid for concurrent readers throughout):

use hdf5_pure::SwmrWriter;

let mut writer = SwmrWriter::open("stream.h5").unwrap();
writer.append_i32("log", &[3, 4, 5]).unwrap();
writer.append_i32("log", &[6, 7]).unwrap();
writer.close().unwrap(); // clears the SWMR flag; or just drop the writer

Follow a growing file from another process (or the reference C library / h5py writing in SWMR mode):

use hdf5_pure::File;

let mut file = File::open_swmr("stream.h5").unwrap();
let n = file.dataset("log").unwrap().shape().unwrap()[0];
// ... later, after the writer appends ...
file.refresh().unwrap();                 // re-read appended data
let ds = file.dataset("log").unwrap();
println!("now {} rows", ds.shape().unwrap()[0]);

Supported subset: one unlimited dimension, chunked, unfiltered (no compression on the appended dataset), chunk-aligned appends, no userblock. Growth is unbounded. SWMR requires the std filesystem (not the in-memory/WASM path). If a writer process exits without close(), the file is left marked as having an active SWMR writer; recover it with SwmrWriter::clear_swmr_flag(path) (the equivalent of h5clear).

Supported data types

Datasets

Attributes

Compression

// Deflate (zlib)
builder.create_dataset("compressed")
    .with_f64_data(&data)
    .with_chunks(&[100])
    .with_deflate(6);

// Shuffle + deflate
builder.create_dataset("shuffled")
    .with_f64_data(&data)
    .with_chunks(&[100])
    .with_shuffle()
    .with_deflate(6);

Scale-offset (HDF5 filter id 6)

Scale-offset stores each chunk's values as offsets from the chunk minimum, packed into the fewest bits the chunk's range needs. It is a built-in HDF5 filter, so files we write are readable by the reference C library, h5py, and MATLAB, and files those tools produce are readable by us.

use hdf5_pure::ScaleOffset;

// Integer mode is lossless. `0` lets the encoder pick the bit width per chunk.
builder.create_dataset("counts")
    .with_i32_data(&counts)
    .with_chunks(&[1000])
    .with_scale_offset(ScaleOffset::Integer(0));

// Float D-scale is lossy: values are rounded to N decimal digits before packing.
builder.create_dataset("readings")
    .with_f64_data(&readings)
    .with_chunks(&[1000])
    .with_scale_offset(ScaleOffset::FloatDScale(3))  // keep 3 decimal digits
    .with_deflate(6);                                // may be followed by deflate

ZFP (optional, zfp feature)

Pure-Rust fixed-rate port of the LLNL/zfp codec, registered HDF5 filter ID 32013. Byte-for-byte interoperable with the reference H5Z-ZFP plugin: files we write are readable by h5py + hdf5plugin, and files those tools produce are readable by us. Supported slice:

• Scalar types: f32, f64, i32, i64
• Ranks: 1D, 2D, 3D, 4D (per-block sizes 4, 16, 64, 256)
• Mode: fixed-rate (rate bits per value)

// Compile with `--features zfp`
builder.create_dataset("temperature")
    .with_f32_data(&data)
    .with_shape(&[ny, nx])
    .with_chunks(&[ny, nx])
    .with_zfp(16.0);  // 16 bits per value

Interop is enforced by tests/zfp_crosscheck.rs, which compares against fixtures produced by h5py + hdf5plugin. See tests/fixtures/zfp/regen.py for the generator — run it after any codec change.

Userblock (MATLAB v7.3)

let mut builder = FileBuilder::new();
builder.with_userblock(512);
builder.create_dataset("data").with_f64_data(&[1.0]);

let mut bytes = builder.finish().unwrap();
// Write MATLAB header into userblock
bytes[126] = b'I';
bytes[127] = b'M';

MATLAB struct pattern

use hdf5_pure::{FileBuilder, AttrValue};

let mut builder = FileBuilder::new();
let mut grp = builder.create_group("my_struct");

let mut fields = Vec::new();
for (name, data) in [("x", vec![1.0, 2.0]), ("y", vec![3.0, 4.0])] {
    fields.push(name.to_string());
    grp.create_dataset(name).with_f64_data(&data)
        .set_attr("MATLAB_class", AttrValue::AsciiString("double".into()));
}

grp.set_attr("MATLAB_class", AttrValue::AsciiString("struct".into()));
grp.set_attr("MATLAB_fields", AttrValue::VarLenAsciiArray(fields));
builder.add_group(grp.finish());

MATLAB v7.3 .mat via serde

With the serde feature, Rust structs can be serialized directly to .mat v7.3 files and back:

use hdf5_pure::mat::{self, Complex64, Matrix};
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, PartialEq, Debug)]
struct Experiment {
    name: String,
    trial: u32,
    samples: Vec<f64>,
    data: Matrix<f64>,
    waveform: Vec<Complex64>,
    config: Config,
}

#[derive(Serialize, Deserialize, PartialEq, Debug)]
struct Config { threshold: f64, tag: String }

let e = Experiment {
    name: "run1".into(), trial: 3,
    samples: vec![1.0, 2.0, 3.0],
    data: Matrix::from_row_major(2, 3, vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0]),
    waveform: vec![Complex64::new(1.0, 0.0), Complex64::new(0.0, 1.0)],
    config: Config { threshold: 0.5, tag: "prod".into() },
};

mat::to_file(&e, "experiment.mat").unwrap();
let back: Experiment = mat::from_file("experiment.mat").unwrap();
assert_eq!(back, e);

The top-level value must be a struct (or HashMap<String, _>); each field becomes a MATLAB variable. Mapping:

Cell array pattern

Sequences that don't unify into a numeric matrix lower to a MATLAB cell array. Each element is interned under the conventional #refs# group and the parent dataset stores object references.

use hdf5_pure::mat;
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize)]
struct Point { x: f64, y: f64 }

#[derive(Serialize, Deserialize)]
struct Capture {
    /// 3x1 cell array of struct.
    path: Vec<Point>,
    /// 3x1 cell array; the `None` slot becomes `struct([])`.
    optionals: Vec<Option<Point>>,
    /// Outer 2x1 cell of cells; rows-of-variable-length-records shape.
    grid: Vec<Vec<Option<Point>>>,
    /// Ragged numerics also fall back to cell rather than erroring.
    ragged: Vec<Vec<f64>>,
}

In MATLAB this loads as iscell(path) == true, path{1}.x, etc. Empty None slots load as struct([]) (isempty(fieldnames(...))).

Reader compatibility. Cell arrays load correctly in MATLAB, libmatio (reference C library), Julia's MAT.jl, and Python via pymatreader / hdf5storage. GNU Octave 11's load does not yet follow object references for v7.3 cells (warns "unknown datatype"); load such files with one of the above instead.

Not supported in this release: non-unit enum variants, MATLAB objects (classdef), datetime / categorical types.

Cargo features

For WASM, disable default features:

[dependencies]
hdf5-pure = { version = "0.7", default-features = false, features = ["checksum"] }

Acknowledgements

The HDF5 format parsing and low-level I/O modules are derived from rustyhdf5 by the RustyStack project (MIT licensed).

License

MIT