Skip to main content

Crate oxigeo_hdf5

Crate oxigeo_hdf5 

Source
Expand description

OxiGeo HDF5 Driver - Real Pure-Rust HDF5 (no libhdf5 FFI)

This crate provides HDF5 file format support for OxiGeo, following the COOLJAPAN Pure Rust policy. Reading and writing are backed by the real Pure-Rust oxih5 crate, so the driver parses and produces genuine .h5 files (including NetCDF-4, which is HDF5 under the hood) — real superblocks, object headers, datatypes, dataspaces, attributes, and dataset values.

§Pure Rust Policy Compliance

oxih5 is used with its default (Pure Rust, oxiarc-deflate) feature set — no C/C++/Fortran (libhdf5) dependency is pulled in.

§Reading (real HDF5 via oxih5)

Hdf5Reader::open opens a genuine HDF5 file, walks its group/dataset tree, and exposes real attributes, real datatypes, real shapes, and real dataset values. When oxih5 genuinely cannot decode a construct (e.g. a v2/v3 superblock, or a datatype with no oxigeo representation) the reader fails loud or omits the object — it never fabricates zero-filled data.

Supported datatypes: i8, u8, i16, u16, i32, u32, i64, u64, f32, f64, fixed- and variable-length strings, plus best-effort mapping of arrays, enums, compounds, references, and vlen types.

§Writing (real HDF5 via oxih5)

Hdf5Writer accumulates groups, datasets, and attributes and, on Hdf5Writer::finalize, emits a real HDF5 file through oxih5’s writer. Root-level chunked datasets (DatasetProperties::with_chunks) are written as real chunked (unlimited-first-dimension) HDF5 layouts. Constructs beyond oxih5’s writer surface (nested groups, sub-group attributes, chunked datasets inside sub-groups, any compression filter — oxih5 has no compression support at all, non-string root attributes) return a typed error rather than silently degrading.

§Feature Flags

  • std (default): Standard library support
  • pure_rust (default): Pure Rust minimal HDF5 implementation
  • hdf5_sys: Full HDF5 support via C bindings (NOT Pure Rust)
  • compression: Compression support (GZIP in Pure Rust, all filters with hdf5_sys)
  • async: Async I/O support

§HDF5 Format Overview

HDF5 (Hierarchical Data Format version 5) is a file format designed for storing and organizing large amounts of data. It’s widely used in scientific computing, particularly for:

  • Climate and weather data (NetCDF-4 is built on HDF5)
  • Satellite imagery (HDF-EOS)
  • Astronomy datasets
  • Medical imaging
  • Machine learning model storage

§Key Concepts

  • File: Container for all HDF5 data
  • Group: Directory-like container for organizing objects
  • Dataset: Multi-dimensional array of homogeneous data
  • Attribute: Small metadata attached to groups or datasets
  • Datatype: Description of data element type
  • Dataspace: Description of dataset dimensions

§Example - Writing HDF5 File (Pure Rust)

ⓘ
use oxigeo_hdf5::{Hdf5Writer, Hdf5Version};
use oxigeo_hdf5::datatype::Datatype;
use oxigeo_hdf5::dataset::DatasetProperties;
use oxigeo_hdf5::attribute::Attribute;

// Create HDF5 file
let mut writer = Hdf5Writer::create("output.h5", Hdf5Version::V10)?;

// Create group
writer.create_group("/measurements")?;

// Add group attribute
writer.add_group_attribute(
    "/measurements",
    Attribute::string("description", "Temperature measurements")
)?;

// Create dataset
writer.create_dataset(
    "/measurements/temperature",
    Datatype::Float32,
    vec![100, 200],  // 100x200 array
    DatasetProperties::new()
)?;

// Write data
let data: Vec<f32> = vec![20.5; 20000];  // 100 * 200 elements
writer.write_f32("/measurements/temperature", &data)?;

// Add dataset attribute
writer.add_dataset_attribute(
    "/measurements/temperature",
    Attribute::string("units", "celsius")
)?;

// Finalize file
writer.finalize()?;

§Example - Reading HDF5 File (Pure Rust)

ⓘ
use oxigeo_hdf5::Hdf5Reader;

// Open HDF5 file
let mut reader = Hdf5Reader::open("output.h5")?;

// Get root group
let root = reader.root()?;
println!("Root group: {}", root.name());

// List groups
for group_path in reader.list_groups() {
    println!("Group: {}", group_path);
}

// List datasets
for dataset_path in reader.list_datasets() {
    let dataset = reader.dataset(dataset_path)?;
    println!("Dataset: {} (shape: {:?}, type: {})",
        dataset.name(),
        dataset.dims(),
        dataset.datatype()
    );
}

// Read dataset
let temperature = reader.read_f32("/measurements/temperature")?;
println!("Temperature data: {} elements", temperature.len());

§Example - Chunked Dataset (real HDF5 chunked layout, no compression)

oxih5’s writer has no compression support at all, so requesting with_gzip/with_compression (anything but CompressionFilter::None) makes Hdf5Writer::finalize fail loud with Hdf5Error::FeatureNotAvailable rather than silently writing an uncompressed file. Chunking alone (with_chunks, no compression) is written as a real chunked HDF5 layout, but only as a single whole-array chunk — chunk_dims must equal the dataset’s full dims (a smaller, partitioned chunk shape is rejected rather than silently dropping data outside the first chunk on read).

ⓘ
use oxigeo_hdf5::{Hdf5Writer, Hdf5Version};
use oxigeo_hdf5::datatype::Datatype;
use oxigeo_hdf5::dataset::DatasetProperties;

let mut writer = Hdf5Writer::create("chunked.h5", Hdf5Version::V10)?;

// Create a chunked (uncompressed) dataset — chunk_dims == dims.
let properties = DatasetProperties::new()
    .with_chunks(vec![1000, 2000]);

writer.create_dataset(
    "/data",
    Datatype::Float64,
    vec![1000, 2000],
    properties
)?;

// Write data
let data: Vec<f64> = vec![0.0; 2_000_000];
writer.write_f64("/data", &data)?;

writer.finalize()?;

§Integration with OxiGeo

This driver integrates with the OxiGeo core types and can be used to read and write geospatial data in HDF5 format, particularly for:

  • HDF-EOS (Earth Observing System) satellite data
  • NetCDF-4 files (which use HDF5 as backend)
  • Climate model outputs
  • Remote sensing imagery

§References

§See Also

Re-exports§

pub use attribute::Attribute;
pub use attribute::AttributeValue;
pub use attribute::Attributes;
pub use cache::CacheConfig;
pub use cache::CacheStatistics;
pub use cache::CachedChunk;
pub use cache::ChunkCache;
pub use cache::EvictionPolicy;
pub use cache::SharedChunkCache;
pub use chunking::AccessPattern;
pub use chunking::ChunkAllocStrategy;
pub use chunking::ChunkGrid;
pub use chunking::ChunkIndex;
pub use chunking::ChunkIndexStructure;
pub use chunking::ChunkMetadata;
pub use chunking::calculate_optimal_chunk_size;
pub use chunking::recommend_chunk_size;
pub use dataset::CompressionFilter;
pub use dataset::Dataset;
pub use dataset::DatasetProperties;
pub use dataset::LayoutType;
pub use datatype::CompoundMember;
pub use datatype::Datatype;
pub use datatype::DatatypeClass;
pub use datatype::EnumMember;
pub use datatype::Hdf5ByteOrder;
pub use datatype::StringPadding;
pub use datatype::TypeConverter;
pub use error::Hdf5Error;
pub use error::Result;
pub use external::ExternalDataset;
pub use external::ExternalFile;
pub use external::ExternalFileManager;
pub use external::ExternalStorage;
pub use filters::Filter;
pub use filters::FilterId;
pub use filters::FilterPipeline;
pub use filters::parse_filter_pipeline_message;
pub use group::Group;
pub use group::ObjectRef;
pub use group::ObjectType;
pub use group::PathUtils;
pub use reader::Hdf5Reader;
pub use reader::Hdf5ReaderBuilder;
pub use reader::Superblock;
pub use reader::SuperblockVersion;
pub use swmr::FileLock;
pub use swmr::MetadataVersion;
pub use swmr::SwmrConfig;
pub use swmr::SwmrMode;
pub use swmr::SwmrReader;
pub use swmr::SwmrStatistics;
pub use swmr::SwmrWriter;
pub use vds::Hyperslab;
pub use vds::SourceDataset;
pub use vds::VdsBuilder;
pub use vds::VdsMapping;
pub use vds::VdsPattern;
pub use vds::VirtualDataset;
pub use vds::create_vds_from_pattern;
pub use writer::Hdf5Version;
pub use writer::Hdf5Writer;
pub use writer::Hdf5WriterBuilder;

Modules§

attribute
HDF5 attribute handling for metadata storage and retrieval.
cache
Chunk cache management for HDF5 datasets.
chunking
Advanced chunking support for HDF5 datasets.
dataset
HDF5 dataset handling for multi-dimensional array storage.
datatype
HDF5 datatype definitions and conversions.
error
HDF5-specific error types for OxiGeo HDF5 driver.
external
External storage support for HDF5 datasets.
filters
HDF5 filter pipeline for data transformation and compression.
group
HDF5 group navigation and hierarchy management.
reader
HDF5 file reader backed by the real Pure-Rust oxih5 crate.
superblock_v2
HDF5 Superblock Version 2 and 3 parsing.
swmr
SWMR (Single Writer Multiple Reader) coordination primitive for HDF5.
vds
Virtual Dataset (VDS) in-memory mapping helper for HDF5 — not file-format-level VDS support.
writer
HDF5 file writer backed by the real Pure-Rust oxih5 crate.

Constants§

NAME
Library name
VERSION
Library version

Functions§

version_info
Get library information