hdf5-pure

Pure-Rust HDF5 reader, writer, and in-place editor. No C dependencies, no build scripts, WASM-compatible.

Features

• Write HDF5 files with datasets, groups, attributes, and nested hierarchies
• Read HDF5 files (v0/v1/v2/v3 superblocks, v1/v2 object headers, contiguous/chunked/compact storage)
• Edit in place — add, delete (H5Ldelete), and copy (H5Ocopy) datasets and groups in an existing file without reading it all in and rewriting it; the cost is proportional to what changes, not the file size
• SWMR (single-writer / multiple-reader) append and refreshing read for 1-D unlimited datasets, interoperable with the reference C library and h5py
• No C dependencies — compiles to wasm32-unknown-unknown with --no-default-features
• MATLAB v7.3 compatible — userblock support, fixed-length ASCII attributes, variable-length string arrays, object references
• Deflate, shuffle, and scale-offset (lossless integer / lossy float) compression
• Compound types, enumerations, array types
• Complex number datasets (as compound {real, imag})

Quick start

Writing

use hdf5_pure::{FileBuilder, AttrValue};

let mut builder = FileBuilder::new();

// Datasets
builder.create_dataset("temperature")
    .with_f64_data(&[22.5, 23.1, 21.8])
    .with_shape(&[3]);

// Groups with nested datasets
let mut grp = builder.create_group("sensors");
grp.create_dataset("pressure").with_f32_data(&[101.3, 101.5]);
grp.set_attr("location", AttrValue::AsciiString("lab_a".into()));
builder.add_group(grp.finish());

// Attributes on the root group
builder.set_attr("version", AttrValue::I64(2));

builder.write("output.h5").unwrap();

Reading

use hdf5_pure::File;

let file = File::open("output.h5").unwrap();
let ds = file.dataset("temperature").unwrap();

println!("shape: {:?}", ds.shape().unwrap());    // [3]
println!("data:  {:?}", ds.read_f64().unwrap());  // [22.5, 23.1, 21.8]

let attrs = file.root().attrs().unwrap();
println!("version: {:?}", attrs.get("version"));  // Some(I64(2))

Editing in place

EditSession opens an existing file and adds, deletes, or copies objects without reading it all in and rewriting it. New data and the rebuilt object headers are appended at the end of the file and the superblock is repointed last, so the cost is proportional to what changes and a failed commit leaves the file valid.

use hdf5_pure::EditSession;

let mut session = EditSession::open("output.h5").unwrap();

session.create_group("run2");
session.create_dataset("run2/signal").with_f64_data(&[1.0, 2.0, 3.0]);
session.copy("temperature", "temperature_backup");  // H5Ocopy
session.delete("sensors/pressure");                 // H5Ldelete

session.commit().unwrap();  // apply everything in place

Contiguous, unfiltered datasets and compact-link groups are supported, and the editor edits files across every on-disk format the reference C library and h5py produce — version 0/1/2/3 superblocks, single- and multi-chunk object headers (a multi-chunk header is collapsed into one chunk on rewrite, and a version 0/1 symbol-table group on the edited path is converted to the latest compact-link format). It refuses, rather than silently degrade the file, anything it cannot reproduce faithfully — a userblock (non-zero base address), chunked/compressed additions, dense-storage headers on the edited path, or copying an existing version-1 object. The space left by deleted or superseded objects is not yet reclaimed.

Streaming large files

File::open(path) reads the whole file into memory. To read a file too large to buffer (for example a multi-gigabyte file produced on a 32-bit host, where it exceeds the address space), open it with File::open_streaming(path) instead. It fetches metadata and dataset chunks from the file on demand rather than buffering it whole, so it never holds the entire file in memory at once: peak memory tracks the data you actually read (one dataset, decompressed, with its chunks fetched on demand) plus the metadata being parsed, not the whole file.

use hdf5_pure::File;

let file = File::open_streaming("huge.h5").unwrap();
let ds = file.dataset("signal").unwrap();
let values = ds.read_f64().unwrap();  // only this dataset's chunks are read

The reading API is identical to File::open; only the backing store differs. Dataset reads are fully supported: contiguous, compact, and every chunk-index layout (B-tree v1, fixed array, and extensible array). Two limits apply to the streaming backend that in-memory reading does not have: only latest-format (v2) groups resolve along a path (a v1 symbol-table group is rejected), and reading attributes is not yet supported. open_streaming requires the std filesystem.

In-memory (WASM)

use hdf5_pure::FileBuilder;

let mut builder = FileBuilder::new();
builder.create_dataset("x").with_f64_data(&[1.0, 2.0]);

let bytes: Vec<u8> = builder.finish().unwrap(); // no filesystem needed

N-dimensional arrays (ndarray feature)

Enable the ndarray feature to write and read datasets of any rank as ndarray arrays. Shape and datatype are taken from the array, and data is stored row-major (C order), matching HDF5:

use hdf5_pure::{File, FileBuilder};
use ndarray::{array, Array2};

let a: Array2<f64> = array![[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]];

let mut fb = FileBuilder::new();
fb.create_dataset("m").with_ndarray(&a);          // shape [2, 3], f64
let bytes = fb.finish().unwrap();

let file = File::from_bytes(bytes).unwrap();
let back: Array2<f64> = file.dataset("m").unwrap().read_array().unwrap();
assert_eq!(a, back);

// When the rank is only known at runtime:
let dynamic = file.dataset("m").unwrap().read_array_dyn::<f64>().unwrap(); // ArrayD<f64>

with_ndarray accepts owned arrays or views; non-standard layouts (transposed, Fortran-order, or strided) are repacked to row-major on write, and chunking and compression chain as usual (.with_ndarray(&a).with_chunks(&[64, 64]).with_deflate(6)).

SWMR (single writer, multiple readers)

A single process can append to an unlimited dataset in place while other processes read it concurrently. The writer appends chunks and flushes in dependency order so readers only ever observe a consistent prefix; readers re-read to pick up new data. This interoperates with the reference HDF5 C library and h5py in both directions.

The dataset must have one unlimited dimension and be chunked (it is indexed by an Extensible Array, which the latest format selects automatically). Create it the usual way:

use hdf5_pure::FileBuilder;

let mut builder = FileBuilder::new();
builder.create_dataset("log")
    .with_i32_data(&[0, 1, 2])   // initial rows
    .with_shape(&[3])
    .with_maxshape(&[u64::MAX])  // one unlimited dimension
    .with_chunks(&[1]);
builder.write("stream.h5").unwrap();

Append in place (each call flushes durably; the file stays valid for concurrent readers throughout):

use hdf5_pure::SwmrWriter;

let mut writer = SwmrWriter::open("stream.h5").unwrap();
writer.append_i32("log", &[3, 4, 5]).unwrap();
writer.append_i32("log", &[6, 7]).unwrap();
writer.close().unwrap(); // clears the SWMR flag; or just drop the writer

Follow a growing file from another process (or the reference C library / h5py writing in SWMR mode):

use hdf5_pure::File;

let mut file = File::open_swmr("stream.h5").unwrap();
let n = file.dataset("log").unwrap().shape().unwrap()[0];
// ... later, after the writer appends ...
file.refresh().unwrap();                 // re-read appended data
let ds = file.dataset("log").unwrap();
println!("now {} rows", ds.shape().unwrap()[0]);

Supported subset: one unlimited dimension, chunked, unfiltered (no compression on the appended dataset), chunk-aligned appends, no userblock. Growth is unbounded. SWMR requires the std filesystem (not the in-memory/WASM path). If a writer process exits without close(), the file is left marked as having an active SWMR writer; recover it with SwmrWriter::clear_swmr_flag(path) (the equivalent of h5clear).

Supported data types

Datasets

Attributes

Compression

// Deflate (zlib)
builder.create_dataset("compressed")
    .with_f64_data(&data)
    .with_chunks(&[100])
    .with_deflate(6);

// Shuffle + deflate
builder.create_dataset("shuffled")
    .with_f64_data(&data)
    .with_chunks(&[100])
    .with_shuffle()
    .with_deflate(6);

Scale-offset (HDF5 filter id 6)

Scale-offset stores each chunk's values as offsets from the chunk minimum, packed into the fewest bits the chunk's range needs. It is a built-in HDF5 filter, so files we write are readable by the reference C library, h5py, and MATLAB, and files those tools produce are readable by us.

use hdf5_pure::ScaleOffset;

// Integer mode is lossless. `0` lets the encoder pick the bit width per chunk.
builder.create_dataset("counts")
    .with_i32_data(&counts)
    .with_chunks(&[1000])
    .with_scale_offset(ScaleOffset::Integer(0));

// Float D-scale is lossy: values are rounded to N decimal digits before packing.
builder.create_dataset("readings")
    .with_f64_data(&readings)
    .with_chunks(&[1000])
    .with_scale_offset(ScaleOffset::FloatDScale(3))  // keep 3 decimal digits
    .with_deflate(6);                                // may be followed by deflate

ZFP (optional, zfp feature)

Pure-Rust fixed-rate port of the LLNL/zfp codec, registered HDF5 filter ID 32013. Byte-for-byte interoperable with the reference H5Z-ZFP plugin: files we write are readable by h5py + hdf5plugin, and files those tools produce are readable by us. Supported slice:

• Scalar types: f32, f64, i32, i64
• Ranks: 1D, 2D, 3D, 4D (per-block sizes 4, 16, 64, 256)
• Mode: fixed-rate (rate bits per value)

// Compile with `--features zfp`
builder.create_dataset("temperature")
    .with_f32_data(&data)
    .with_shape(&[ny, nx])
    .with_chunks(&[ny, nx])
    .with_zfp(16.0);  // 16 bits per value

Interop is enforced by tests/zfp_crosscheck.rs, which compares against fixtures produced by h5py + hdf5plugin. See tests/fixtures/zfp/regen.py for the generator — run it after any codec change.

Userblock (MATLAB v7.3)

let mut builder = FileBuilder::new();
builder.with_userblock(512);
builder.create_dataset("data").with_f64_data(&[1.0]);

let mut bytes = builder.finish().unwrap();
// Write MATLAB header into userblock
bytes[126] = b'I';
bytes[127] = b'M';

MATLAB struct pattern

use hdf5_pure::{FileBuilder, AttrValue};

let mut builder = FileBuilder::new();
let mut grp = builder.create_group("my_struct");

let mut fields = Vec::new();
for (name, data) in [("x", vec![1.0, 2.0]), ("y", vec![3.0, 4.0])] {
    fields.push(name.to_string());
    grp.create_dataset(name).with_f64_data(&data)
        .set_attr("MATLAB_class", AttrValue::AsciiString("double".into()));
}

grp.set_attr("MATLAB_class", AttrValue::AsciiString("struct".into()));
grp.set_attr("MATLAB_fields", AttrValue::VarLenAsciiArray(fields));
builder.add_group(grp.finish());

MATLAB v7.3 .mat via serde

With the serde feature, Rust structs can be serialized directly to .mat v7.3 files and back:

use hdf5_pure::mat::{self, Complex64, Matrix};
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize, PartialEq, Debug)]
struct Experiment {
    name: String,
    trial: u32,
    samples: Vec<f64>,
    data: Matrix<f64>,
    waveform: Vec<Complex64>,
    config: Config,
}

#[derive(Serialize, Deserialize, PartialEq, Debug)]
struct Config { threshold: f64, tag: String }

let e = Experiment {
    name: "run1".into(), trial: 3,
    samples: vec![1.0, 2.0, 3.0],
    data: Matrix::from_row_major(2, 3, vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0]),
    waveform: vec![Complex64::new(1.0, 0.0), Complex64::new(0.0, 1.0)],
    config: Config { threshold: 0.5, tag: "prod".into() },
};

mat::to_file(&e, "experiment.mat").unwrap();
let back: Experiment = mat::from_file("experiment.mat").unwrap();
assert_eq!(back, e);

The top-level value must be a struct (or HashMap<String, _>); each field becomes a MATLAB variable. Mapping:

Cell array pattern

Sequences that don't unify into a numeric matrix lower to a MATLAB cell array. Each element is interned under the conventional #refs# group and the parent dataset stores object references.

use hdf5_pure::mat;
use serde::{Serialize, Deserialize};

#[derive(Serialize, Deserialize)]
struct Point { x: f64, y: f64 }

#[derive(Serialize, Deserialize)]
struct Capture {
    /// 3x1 cell array of struct.
    path: Vec<Point>,
    /// 3x1 cell array; the `None` slot becomes `struct([])`.
    optionals: Vec<Option<Point>>,
    /// Outer 2x1 cell of cells; rows-of-variable-length-records shape.
    grid: Vec<Vec<Option<Point>>>,
    /// Ragged numerics also fall back to cell rather than erroring.
    ragged: Vec<Vec<f64>>,
}

In MATLAB this loads as iscell(path) == true, path{1}.x, etc. Empty None slots load as struct([]) (isempty(fieldnames(...))).

Reader compatibility. Cell arrays load correctly in MATLAB, libmatio (reference C library), Julia's MAT.jl, and Python via pymatreader / hdf5storage. GNU Octave 11's load does not yet follow object references for v7.3 cells (warns "unknown datatype"); load such files with one of the above instead.

Not supported in this release: non-unit enum variants, MATLAB objects (classdef), datetime / categorical types.

Cargo features

For WASM, disable default features:

[dependencies]
hdf5-pure = { version = "0.8", default-features = false, features = ["checksum"] }

Acknowledgements

The HDF5 format parsing and low-level I/O modules are derived from rustyhdf5 by the RustyStack project (MIT licensed).

License

MIT