1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
//! # HIDEFIX
//!
//! A fast and concurrent reader for HDF5 and NetCDF (v4) files.
//!
//! This library allows a HDF5 file to be read in a multi-threaded or concurrent (async) way. The
//! chunks of a dataset need to be indexed in advance. Fast in newer versions of HDF5 (see below).
//! The index can be efficiently deserialized with zero-copy through [serde](https://serde.rs/).
//!
//! This allows multiple [datasets](idx::Dataset) (variables) to be read at the same time, or even different
//! domains of the same dataset to be read at the same time.
//!
//! The library is meant to be used in conjunction with the [bindings to the official HDF5
//! library](https://docs.rs/hdf5/0.7.0/hdf5/index.html).
//!
//! ## Usage
//!
//! Create an [index](idx::Index), then read the values:
//!
//! ```
//! use hidefix::prelude::*;
//!
//! let idx = Index::index("tests/data/coads_climatology.nc4").unwrap();
//! let mut r = idx.reader("SST").unwrap();
//!
//! let values = r.values::<f32, _>(..).unwrap();
//!
//! println!("SST: {:?}", values);
//! ```
//!
//! or convert a [hdf5::File] or [hdf5::Dataset] into an index by using
//! [`try_from`](std::convert::TryFrom) or the [`index`](idx::IntoIndex) method.
//!
//! or use the [IntoIndex](idx::IntoIndex) trait:
//!
//! ```
//! use hidefix::prelude::*;
//!
//! let i = hdf5::File::open("tests/data/coads_climatology.nc4").unwrap().index().unwrap();
//! let iv = i.reader("SST").unwrap().values::<f32, _>(..).unwrap();
//! ```
//!
//! ## NetCDF4 files
//!
//! NetCDF4 uses HDF5 as their underlying data-format. Hidefix can be used to read the NetCDF
//! variables, though there might be extra decoding necessary. The hidefix-`xarray` does that for
//! you in the python bindings.
//!
//! ```
//! use std::convert::TryInto;
//! use hidefix::prelude::*;
//!
//! let f = netcdf::open("tests/data/coads_climatology.nc4").unwrap();
//! let nv = f.variable("SST").unwrap().get_values::<f32, _>(..).unwrap();
//!
//! let i: Index = (&f).try_into().unwrap();
//! let iv = i.reader("SST").unwrap().values::<f32, _>(..).unwrap();
//!
//! assert_eq!(iv, nv);
//! ```
//!
//! or use the [IntoIndex](idx::IntoIndex) trait:
//!
//! ```
//! use hidefix::prelude::*;
//!
//! let i = netcdf::open("tests/data/coads_climatology.nc4").unwrap().index().unwrap();
//! let iv = i.reader("SST").unwrap().values::<f32, _>(..).unwrap();
//! ```
//!
//! It is also possible to [stream](reader::stream::StreamReader) the values. The streamer is
//! currently optimized for streaming bytes.
//!
//! ## Fast indexing
//!
//! The indexing can be sped up considerably (_about 200x_) by using the new interface to
//! [iterating over chunks](https://github.com/HDFGroup/hdf5/pull/6) in HDF5.