Expand description
OxiGeo HDF5 Driver - Real Pure-Rust HDF5 (no libhdf5 FFI)
This crate provides HDF5 file format support for OxiGeo, following the
COOLJAPAN Pure Rust policy. Reading and writing are backed by the real
Pure-Rust oxih5 crate, so the driver
parses and produces genuine .h5 files (including NetCDF-4, which is HDF5
under the hood) — real superblocks, object headers, datatypes, dataspaces,
attributes, and dataset values.
§Pure Rust Policy Compliance
oxih5 is used with its default (Pure Rust, oxiarc-deflate) feature set —
no C/C++/Fortran (libhdf5) dependency is pulled in.
§Reading (real HDF5 via oxih5)
Hdf5Reader::open opens a genuine HDF5 file, walks its group/dataset tree,
and exposes real attributes, real datatypes, real shapes, and real dataset
values. When oxih5 genuinely cannot decode a construct (e.g. a v2/v3
superblock, or a datatype with no oxigeo representation) the reader fails
loud or omits the object — it never fabricates zero-filled data.
Supported datatypes: i8, u8, i16, u16, i32, u32, i64, u64, f32, f64, fixed- and variable-length strings, plus best-effort mapping of arrays, enums, compounds, references, and vlen types.
§Writing (real HDF5 via oxih5)
Hdf5Writer accumulates groups, datasets, and attributes and, on
Hdf5Writer::finalize, emits a real HDF5 file through oxih5’s writer.
Root-level chunked datasets (DatasetProperties::with_chunks) are written
as real chunked (unlimited-first-dimension) HDF5 layouts. Constructs beyond
oxih5’s writer surface (nested groups, sub-group attributes, chunked
datasets inside sub-groups, any compression filter — oxih5 has no
compression support at all, non-string root attributes) return a typed
error rather than silently degrading.
§Feature Flags
std(default): Standard library supportpure_rust(default): Pure Rust minimal HDF5 implementationhdf5_sys: Full HDF5 support via C bindings (NOT Pure Rust)compression: Compression support (GZIP in Pure Rust, all filters with hdf5_sys)async: Async I/O support
§HDF5 Format Overview
HDF5 (Hierarchical Data Format version 5) is a file format designed for storing and organizing large amounts of data. It’s widely used in scientific computing, particularly for:
- Climate and weather data (NetCDF-4 is built on HDF5)
- Satellite imagery (HDF-EOS)
- Astronomy datasets
- Medical imaging
- Machine learning model storage
§Key Concepts
- File: Container for all HDF5 data
- Group: Directory-like container for organizing objects
- Dataset: Multi-dimensional array of homogeneous data
- Attribute: Small metadata attached to groups or datasets
- Datatype: Description of data element type
- Dataspace: Description of dataset dimensions
§Example - Writing HDF5 File (Pure Rust)
use oxigeo_hdf5::{Hdf5Writer, Hdf5Version};
use oxigeo_hdf5::datatype::Datatype;
use oxigeo_hdf5::dataset::DatasetProperties;
use oxigeo_hdf5::attribute::Attribute;
// Create HDF5 file
let mut writer = Hdf5Writer::create("output.h5", Hdf5Version::V10)?;
// Create group
writer.create_group("/measurements")?;
// Add group attribute
writer.add_group_attribute(
"/measurements",
Attribute::string("description", "Temperature measurements")
)?;
// Create dataset
writer.create_dataset(
"/measurements/temperature",
Datatype::Float32,
vec![100, 200], // 100x200 array
DatasetProperties::new()
)?;
// Write data
let data: Vec<f32> = vec![20.5; 20000]; // 100 * 200 elements
writer.write_f32("/measurements/temperature", &data)?;
// Add dataset attribute
writer.add_dataset_attribute(
"/measurements/temperature",
Attribute::string("units", "celsius")
)?;
// Finalize file
writer.finalize()?;§Example - Reading HDF5 File (Pure Rust)
use oxigeo_hdf5::Hdf5Reader;
// Open HDF5 file
let mut reader = Hdf5Reader::open("output.h5")?;
// Get root group
let root = reader.root()?;
println!("Root group: {}", root.name());
// List groups
for group_path in reader.list_groups() {
println!("Group: {}", group_path);
}
// List datasets
for dataset_path in reader.list_datasets() {
let dataset = reader.dataset(dataset_path)?;
println!("Dataset: {} (shape: {:?}, type: {})",
dataset.name(),
dataset.dims(),
dataset.datatype()
);
}
// Read dataset
let temperature = reader.read_f32("/measurements/temperature")?;
println!("Temperature data: {} elements", temperature.len());§Example - Chunked Dataset (real HDF5 chunked layout, no compression)
oxih5’s writer has no compression support at all, so requesting
with_gzip/with_compression (anything but CompressionFilter::None)
makes Hdf5Writer::finalize fail loud with
Hdf5Error::FeatureNotAvailable rather than silently writing an
uncompressed file. Chunking alone (with_chunks, no compression) is
written as a real chunked HDF5 layout, but only as a single whole-array
chunk — chunk_dims must equal the dataset’s full dims (a smaller,
partitioned chunk shape is rejected rather than silently dropping data
outside the first chunk on read).
use oxigeo_hdf5::{Hdf5Writer, Hdf5Version};
use oxigeo_hdf5::datatype::Datatype;
use oxigeo_hdf5::dataset::DatasetProperties;
let mut writer = Hdf5Writer::create("chunked.h5", Hdf5Version::V10)?;
// Create a chunked (uncompressed) dataset — chunk_dims == dims.
let properties = DatasetProperties::new()
.with_chunks(vec![1000, 2000]);
writer.create_dataset(
"/data",
Datatype::Float64,
vec![1000, 2000],
properties
)?;
// Write data
let data: Vec<f64> = vec![0.0; 2_000_000];
writer.write_f64("/data", &data)?;
writer.finalize()?;§Integration with OxiGeo
This driver integrates with the OxiGeo core types and can be used to read and write geospatial data in HDF5 format, particularly for:
- HDF-EOS (Earth Observing System) satellite data
- NetCDF-4 files (which use HDF5 as backend)
- Climate model outputs
- Remote sensing imagery
§References
- HDF5 Specification
- HDF5 User Guide
- hdf5file - Pure Rust HDF5 implementation
- oxifive - Pure Rust HDF5 reader
- hdf5-rust - HDF5 C bindings for Rust
§See Also
oxigeo-netcdf- NetCDF driver (NetCDF-4 uses HDF5)oxigeo-geotiff- GeoTIFF driveroxigeo-zarr- Zarr driver (alternative to HDF5)
Re-exports§
pub use attribute::Attribute;pub use attribute::AttributeValue;pub use attribute::Attributes;pub use cache::CacheConfig;pub use cache::CacheStatistics;pub use cache::CachedChunk;pub use cache::ChunkCache;pub use cache::EvictionPolicy;pub use chunking::AccessPattern;pub use chunking::ChunkAllocStrategy;pub use chunking::ChunkGrid;pub use chunking::ChunkIndex;pub use chunking::ChunkIndexStructure;pub use chunking::ChunkMetadata;pub use chunking::calculate_optimal_chunk_size;pub use chunking::recommend_chunk_size;pub use dataset::CompressionFilter;pub use dataset::Dataset;pub use dataset::DatasetProperties;pub use dataset::LayoutType;pub use datatype::CompoundMember;pub use datatype::Datatype;pub use datatype::DatatypeClass;pub use datatype::EnumMember;pub use datatype::Hdf5ByteOrder;pub use datatype::StringPadding;pub use datatype::TypeConverter;pub use error::Hdf5Error;pub use error::Result;pub use external::ExternalDataset;pub use external::ExternalFile;pub use external::ExternalFileManager;pub use external::ExternalLink;pub use external::ExternalStorage;pub use filters::Filter;pub use filters::FilterId;pub use filters::FilterPipeline;pub use filters::parse_filter_pipeline_message;pub use group::Group;pub use group::ObjectRef;pub use group::ObjectType;pub use group::PathUtils;pub use reader::Hdf5Reader;pub use reader::Hdf5ReaderBuilder;pub use reader::Superblock;pub use reader::SuperblockVersion;pub use swmr::FileLock;pub use swmr::MetadataVersion;pub use swmr::SwmrConfig;pub use swmr::SwmrMode;pub use swmr::SwmrReader;pub use swmr::SwmrStatistics;pub use swmr::SwmrWriter;pub use vds::Hyperslab;pub use vds::SourceDataset;pub use vds::VdsBuilder;pub use vds::VdsMapping;pub use vds::VdsPattern;pub use vds::VirtualDataset;pub use vds::create_vds_from_pattern;pub use writer::Hdf5Version;pub use writer::Hdf5Writer;pub use writer::Hdf5WriterBuilder;
Modules§
- attribute
- HDF5 attribute handling for metadata storage and retrieval.
- cache
- Chunk cache management for HDF5 datasets.
- chunking
- Advanced chunking support for HDF5 datasets.
- dataset
- HDF5 dataset handling for multi-dimensional array storage.
- datatype
- HDF5 datatype definitions and conversions.
- error
- HDF5-specific error types for OxiGeo HDF5 driver.
- external
- External storage support for HDF5 datasets.
- filters
- HDF5 filter pipeline for data transformation and compression.
- group
- HDF5 group navigation and hierarchy management.
- reader
- HDF5 file reader backed by the real Pure-Rust
oxih5crate. - superblock_
v2 - HDF5 Superblock Version 2 and 3 parsing.
- swmr
- SWMR (Single Writer Multiple Reader) coordination primitive for HDF5.
- vds
- Virtual Dataset (VDS) in-memory mapping helper for HDF5 — not file-format-level VDS support.
- writer
- HDF5 file writer backed by the real Pure-Rust
oxih5crate.
Constants§
Functions§
- version_
info - Get library information