[−][src]Crate hidefix
HIDEFIX
A fast and concurrent reader for HDF5 and NetCDF (v4) files.
This library allows a HDF5 file to be read in a multi-threaded and concurrent way. The chunks of a dataset need to be indexed in advance, this can be time-consuming, but efficient serialization and partially zero-copy deserialization through serde is implemented. In particular by storing the indexes in a fast database, like sled allows speedy access.
This allows multiple datasets (variables) to be read at the same time, or even different domains of the same dataset to be read at the same time.
The library is meant to be used in conjunction with the bindings to the official HDF5 library.
Usage
Create an index, then read the values:
use hidefix::prelude::*; let idx = Index::index("tests/data/coads_climatology.nc4").unwrap(); let mut r = idx.reader("SST").unwrap(); let values = r.values::<f32>(None, None).unwrap(); println!("SST: {:?}", values);
or convert a hdf5::File or hdf5::Dataset into an index by using
try_from
or the index
method.
It is also possible to stream the values. The streamer is currently optimized for streaming bytes.
Fast indexing
The indexing can be sped up considerably (about 200x) by adding a new interface to iterating
over chunks in HDF5. The fast-index
feature flag currently requires a patched version of
hdf5-rust and
hdf5. See this upstream
pull-request. You therefore have to use patch
to
point the hdf5
and hdf5-sys
dependencies to the patched versions for now.
Modules
filters | |
idx | |
prelude | |
reader |