Struct File 

pub struct File { /* private fields */ }

An open HDF5 file.

A File is an owned, cheaply cloneable handle to an open file: cloning it (or deriving a Dataset/Group from it) shares one underlying open file rather than re-reading it. Object handles returned by dataset, group, and root are owned — they keep the file open for as long as they live and carry no borrow of the File, so they can be stored in a struct, cached, cloned, and moved across threads. They stay usable across a commit, which is what makes caching one worthwhile; see commit for the two cases that report instead.

Implementations

impl File

pub fn open<P: AsRef<Path>>(path: P) -> Result<Self, Error>

Open an HDF5 file from a filesystem path.

Reads the file into memory once. To follow a file that a concurrent single writer is appending to (SWMR), use File::open_swmr instead. To read a file larger than memory (e.g. on a 32-bit host) without buffering it, use File::open_streaming.

A file whose superblock marks it as held by a writer is refused with Error::FileMarkedInUse — the check H5Fopen makes of the same byte. That means a live writer or one that exited without closing the file; clear a stale flag with clear_swmr_flag, and follow a live SWMR writer with open_swmr. from_bytes does not check, since its caller already holds the bytes — which is also the way to read a flagged file on a read-only mount, where clearing the flag would need write access.

pub fn open_with_options<P: AsRef<Path>>( path: P, properties: FileAccessProperties, ) -> Result<Self, Error>

Open an HDF5 file from a filesystem path with explicit access properties.

pub fn open_streaming<P: AsRef<Path>>(path: P) -> Result<Self, Error>

Open an HDF5 file for streaming reads, fetching regions on demand from the file instead of buffering it whole.

This lets a host read a file larger than its address space. Metadata and dataset chunks are read through a ReadSeekSource, so peak memory stays close to one chunk plus the metadata being parsed; chunks adjacent on disk are fetched together, in reads of at most 256 KiB, and a chunk larger than that is read on its own. Attribute reading and v1 symbol-table groups on the resolved path are not yet supported on this backend.

Like open, this refuses a file whose superblock marks it as held by a writer.

pub fn open_streaming_with_options<P: AsRef<Path>>( path: P, properties: FileAccessProperties, ) -> Result<Self, Error>

Open an HDF5 file for streaming reads with explicit access properties.

pub fn from_source<S: Source + Send + Sync + 'static>( source: S, ) -> Result<Self, Error>

Open an HDF5 file from any Source, reading metadata and chunks on demand exactly as open_streaming does.

This is the streaming open for a caller whose bytes are not a path: an object store addressed by HTTP range request, a sandboxed guest that receives byte ranges from its host, a decrypting layer. A Source supplies a length and reads at an absolute offset, which is all the reader asks of a file, so peak memory stays at the metadata being parsed plus the chunks a read touches — not the file. Wrap a Read + Seek in ReadSeekSource rather than writing that impl again.

A file marked as held by a writer is refused here as it is by open_streaming; with no path to report, the error names the source instead. Recovering such a file in place needs a path, through File::clear_swmr_flag, so a caller that has none is left with File::from_bytes and the whole file in memory.

The metadata cache is off unless from_source_with_options turns it on, which matters more here than it does for a local file: without one, every read a parser makes is a round trip. See MetadataCacheConfig.

use hdf5_pure::{File, FormatError, Source};

// However the bytes actually arrive: a range request, a host call, a
// decrypting layer over a file.

struct Remote {
    len: u64,
}

impl Source for Remote {
    fn len(&self) -> u64 {
        self.len
    }

    fn read_at(&self, offset: u64, buf: &mut [u8]) -> Result<(), FormatError> {
        // Fill the whole request or fail: a short read is an error.
        let bytes = fetch(offset, buf.len()).map_err(FormatError::Source)?;
        buf.copy_from_slice(&bytes);
        Ok(())
    }
}

let file = File::from_source(Remote { len: 1 << 30 })?;
let rows = file.dataset("frames")?.read_f64_rows(0, 64)?;

A Read + Seek needs none of that: wrap it in ReadSeekSource. Reading a file that way is open_streaming, which does the wrapping for you.

pub fn from_source_with_options<S: Source + Send + Sync + 'static>( source: S, properties: FileAccessProperties, ) -> Result<Self, Error>

Open an HDF5 file from any Source with explicit access properties.

The metadata cache is what a remote source wants tuned: every parser read becomes a round trip without one. See MetadataCacheConfig.

pub fn open_swmr<P: AsRef<Path>>(path: P) -> Result<Self, Error>

Open an HDF5 file for SWMR (single-writer/multiple-reader) reading.

Like File::open, but retains a live handle to the file so that File::refresh can re-read data appended by a concurrent writer.

This is the open that follows a file marked as held by a SWMR writer, where open refuses one. Only a half-set mark is refused here, with Error::FileMarkedInUse: either bit without the other. Write access alone is what a plain (non-SWMR) writer leaves, and there is no protocol for following a writer that is not publishing consistent prefixes; the SWMR bit alone is a state no writer produces. Both bits is the live SWMR writer this exists to follow, and neither is a quiescent file.

pub fn open_swmr_with_options<P: AsRef<Path>>( path: P, properties: FileAccessProperties, ) -> Result<Self, Error>

Open an HDF5 file for SWMR reading with explicit access properties.

pub fn from_bytes(data: Vec<u8>) -> Result<Self, Error>

Open an HDF5 file from an in-memory byte vector.

pub fn from_bytes_with_options( data: Vec<u8>, properties: FileAccessProperties, ) -> Result<Self, Error>

Open an HDF5 file from an in-memory byte vector with explicit access properties.

pub fn open_rw<P: AsRef<Path>>(path: P) -> Result<Self, Error>

Open an existing HDF5 file for reading and in-place editing.

Unlike open (read-only, buffered), this takes an exclusive OS file lock held for the file’s life and lets owned handles modify the file — immediate Dataset::appends, plus Dataset::write/set_attr, Group::create_dataset/create_group/delete/set_attr, and copy/copy_from staged until commit. The file must use 8-byte offsets and lengths and keep its superblock at its base address (a canonical userblock, as in a MATLAB .mat file, is supported); anything else is refused with Error::EditUnsupported.

The fast immediate Dataset::append additionally requires a latest-format (version-2/3) file with no userblock and an Extensible-Array-indexed dataset; Dataset::append_staged covers the general case.

Two things can turn this open away because another writer holds the file: the exclusive OS lock, reported as Error::FileLocked, and the superblock’s status-flags byte, reported as Error::FileMarkedInUse. The second covers what the first cannot — a SWMR writer takes no lock, and a writer that exited without closing the file leaves the flag behind; recover a stale one with clear_swmr_flag.

Memory

This picks its backing from the file rather than making the caller pick a function (issue #198): a latest-format file with no userblock is edited bounded, holding only the metadata being parsed plus the configured caches plus what an edit is building, so resident memory does not scale with the file; anything else falls back to a whole-file in-memory mirror, which is what makes a pre-v2 or userblock file editable at all. The two backings are the same engine over different storage and offer the same edit surface, differing in one trade: the bounded one applies a large immediate append in whole-chunk batches, each crash-atomic on its own, so a crash mid-call leaves a valid shorter dataset rather than none of the append. Ask a file which it got with edit_backing, and demand one with FileAccessProperties::with_memory_strategy — MemoryStrategy::Mirrored restores the unconditional mirror this entry point used before it learned to dispatch.

pub fn open_rw_with_options<P: AsRef<Path>>( path: P, properties: FileAccessProperties, ) -> Result<Self, Error>

Open an existing file for reading and in-place editing with explicit access properties — see open_rw.

The properties carry the locking policy (the H5Pset_file_locking analogue, FileAccessProperties::with_locking), the memory strategy (FileAccessProperties::with_memory_strategy, which overrides the dispatch described on open_rw), the fsync cadence (FileAccessProperties::with_sync_policy), the metadata cache used by the bounded backing, and the file-wide chunk-cache default applied to datasets opened from this file. Because one FileAccessProperties value serves every open, the same configuration can be shared with a read path.

pub fn open_swmr_writer<P: AsRef<Path>>(path: P) -> Result<Self, Error>

Open an existing file for SWMR (single-writer/multiple-reader) appending: take no OS lock (so concurrent readers, and Windows’ mandatory locks, are never blocked) and raise the superblock’s SWMR-write flag so a reader may attach with File::open_swmr, the C library’s H5F_ACC_SWMR_READ, or h5py swmr=True.

Only immediate Dataset::append is permitted, and only over the SWMR subset — an unfiltered, chunk-aligned append, so a concurrent reader only ever observes a consistent prefix; a filtered or non-chunk-aligned append returns Error::SwmrAppendUnsupported. The staged edit surface (write/set_attr/create_*/delete/copy/ commit) returns Error::SwmrStagedUnsupported. close clears the SWMR-write flag; a writer that exits without a clean close leaves it set — recover with clear_swmr_flag. While the flag stands, this open is refused with Error::FileMarkedInUse, which is what keeps a second writer off a file SWMR gives only one (no OS lock is held to do it).

Requires a latest-format (version-3 superblock) file with no userblock and no persisted free-space; other files are refused with Error::SwmrAppendUnsupported. The version-3 requirement is the C library’s: neither library reads the SWMR-write flag back on an older superblock, so raising one there would announce the writer to nobody.

pub fn open_swmr_writer_with_options<P: AsRef<Path>>( path: P, properties: FileAccessProperties, ) -> Result<Self, Error>

Open for SWMR appending with explicit access properties — see open_swmr_writer.

The properties’ chunk cache is the file-wide default for datasets opened from this file. Its locking policy is ignored, which costs the caller nothing: SWMR takes no OS lock by design, which is stronger than any locking a caller could ask for. Its memory strategy is not ignored the same way — this writer always mirrors, so an explicit MemoryStrategy::Bounded is a guarantee it cannot meet and is refused with Error::EditUnsupported; MemoryStrategy::Auto and MemoryStrategy::Mirrored are both satisfied by the mirror.

Its SyncPolicy applies here as to any other read-write session, the SWMR-write flag included; a reader on this machine is unaffected either way, since the barriers carry the write order across power loss rather than across processes.

pub fn clear_swmr_flag<P: AsRef<Path>>(path: P) -> Result<(), Error>

Clear a stale status flag left in path by a writer that exited without a clean close — the h5clear -s equivalent, for recovering a file that both this crate and the reference C library otherwise refuse to open (Error::FileMarkedInUse). A no-op if the flag is already clear.

It takes the exclusive OS lock first, so it cannot clear the flag out from under a live open_rw writer. A live SWMR writer holds no lock, so make sure it is really gone: clearing the flag under one leaves its readers with no record that it is publishing.

It also clears the crash mark a page-buffered session raises (FileAccessProperties::with_page_buffer_size), and there the warning is sharper. That mark stands for pages that were still in memory, so a file still carrying it was left by a writer that did not finish: clearing it hands back a file whose datasets may read clean and return fill values or a deleted object’s bytes, with every checksum verifying. Clear it to salvage what is there, not to resume trusting it. h5clear makes the same trade for the same reason.

pub fn create<P: AsRef<Path>>(path: P) -> Result<Self, Error>

Create a new, empty HDF5 file at path and open it for reading and writing, so its contents can be built entirely through owned handles (Group::create_dataset/create_group, then commit).

Overwrites any existing file at path. For an all-at-once write, use FileBuilder instead.

pub fn create_with_options<P: AsRef<Path>>( path: P, create: FileCreateProperties, access: FileAccessProperties, ) -> Result<Self, Error>

Create a new, empty HDF5 file with explicit creation and access properties, then open it for reading and writing — see create.

Mirrors H5Fcreate(name, flags, fcpl_id, fapl_id): create carries the creation properties recorded in the new file (userblock, file-space strategy, library-version bounds), and access the properties governing the handle returned (locking policy, fsync cadence, chunk cache). Both are values, so a layout defined once can be reused across every file an application writes.

A creation property is validated as the file is written, so an invalid userblock or page size surfaces here rather than when the properties were built. A file created with FileSpaceStrategy::Page can be grown through either editor, by an immediate Dataset::append or a staged commit, provided it also persists its free space (issue #198).

pub fn commit(&self) -> Result<(), Error>

Apply all staged structural edits made through this file’s handles — Dataset::write/set_attr/remove_attr and Group::create_group/delete — as one transaction. Immediate Dataset::appends need no commit.

Requires a read-write file (File::open_rw); a read-only file returns Error::ReadOnly.

Outstanding Dataset and Group handles stay usable: a commit relocates object headers, and each handle looks its object up again by path on its first use afterwards, so a long-lived handle answers for the file the commit left rather than for the copy it moved away from. Two exceptions, both of which report rather than answer wrongly. A read through a handle onto an object the commit deleted — or replaced with one of a different kind, which is Error::NotADataset or Error::NotAGroup — fails the way opening it would; its write methods still address the file by path, so they stage and the commit refuses them. And a handle reached by object reference (Dataset::dereference) has no path to look up, so it returns Error::StaleHandle — not only after a commit but after anything staged, synced or torn down, since only an immediate Dataset::append is known to leave every header where it stands. Dereference again from a fresh read.

A handle onto an object this commit publishes — one Group::create_group, Group::create_group_with or Group::create_dataset handed back, or a lookup of a staged name found — starts reading its object here. Until then it answers Error::NotCommitted for anything needing bytes, and a refused commit leaves it doing so.

The commit is durable when it returns, under the default SyncPolicy::Always; under SyncPolicy::OnClose it has reached the operating system and waits for a sync.

A commit refused before it publishes leaves every dataset reading what it read before. Almost everything such a commit writes lands where nothing reaches it until the commit’s linearization point; the one edit that does not is a same-length Dataset::write, which overwrites the dataset’s existing block, and the refusal writes those bytes back on its way out. A refusal raised before the first write keeps the staged batch too, so it can be corrected and committed again (issue #316).

Errors

Two failures do not carry that promise, and both call for re-reading the datasets the batch named rather than for a retry:

• Error::CommitPartiallyApplied, where the restore itself failed, so a dataset may hold either value.
• An error from a step after the commit published — repointing the object references that named a moved object is the one that can raise it. The batch is in the file and stays there; what failed is work the commit owed afterwards. The file is valid either way.

pub fn copy(&self, src: &str, dst: &str) -> Result<(), Error>

Copy the object at src to dst within this file (the in-file H5Ocopy), staged until commit.

A dataset whose storage was never allocated is copied as the storage it has — none — rather than as the fill value reading it answers with, so a schema-only dataset stays one. A dataset whose elements live in external files (H5Pset_external) carries that same empty storage while holding data this crate does not read, and is refused with Error::EditUnsupported rather than copied without it.

Requires a read-write file (File::open_rw); a read-only file returns Error::ReadOnly.

pub fn copy_from( &self, source: &File, src: &str, dst: &str, ) -> Result<(), Error>

Copy the object at src in source — a separate, buffered read-only file — into this file at dst: the cross-file H5Ocopy, staged until commit.

source must be a buffered file (File::open or File::from_bytes, not File::open_streaming) that uses 8-byte offsets and has no userblock; anything else is refused with Error::EditUnsupported. The source subtree is read and validated eagerly, so source need not outlive this call — and so a source this cannot reproduce, external storage included, is refused by this call. Refusals that concern the destination — dst already exists, or its parent group does not — still come from commit. Requires a read-write destination (File::open_rw); a read-only one returns Error::ReadOnly.

pub fn has_staged_edits(&self) -> bool

Report whether this file has structural edits staged but not yet applied by commit — Dataset::write/set_attr/remove_attr, Dataset::append_staged, Group::create_group/create_dataset/ delete/set_attr/remove_attr, and copy/ copy_from. Immediate Dataset::appends are never staged and do not count. Always false for a read-only file.

A commit that refuses puts the staged set back untouched, so this still answers true afterwards and the same batch can be committed again — to the same refusal, until the session is dropped.

pub fn space_accounting(&self) -> Result<SpaceAccounting, Error>

Report this read-write file’s live space usage as a SpaceAccounting — the current logical size, total reusable free bytes, and reusable free regions. It reflects committed state plus immediate in-place appends, not edits still staged for commit.

Requires a read-write file (File::open_rw); a read-only file returns Error::ReadOnly.

pub fn sync(&self) -> Result<(), Error>

Force everything written to this file so far to durable storage — the fsync the application issues at its own cadence under SyncPolicy::OnClose, and a redundant one under the default SyncPolicy::Always. A SWMR-writer file syncs the same way.

This is a durability barrier, not a flush: it writes nothing itself. Staged edits are not applied (commit does that, and a sync before one makes only the previous state durable), and elements held by a live BufferedAppender have not reached the file at all — flush it first.

There is no need to call it before close: close — and dropping the last handle — issues its own barrier under every policy, because both write and both destroy the handle that would have ordered those writes. This is the mid-session checkpoint, not the closing one.

Requires a read-write file (File::open_rw); a read-only file returns Error::ReadOnly, and a sealed one Error::FileClosed — a closed file has already been synced.

pub fn close(self) -> Result<(), Error>

Commit any staged edits and seal this file. The exclusive OS lock is released once the last handle derived from this file is also dropped.

After close, a write through any surviving Dataset/Group handle or File clone returns Error::FileClosed; reads still work. close commits, so the one handle a commit ends ends here too: one reached by Dataset::dereference reports Error::StaleHandle afterwards, where a handle opened by path re-resolves and keeps reading.

pub fn root(&self) -> Group

Returns an owned handle to the root group.

pub fn dataset(&self, path: &str) -> Result<Dataset, Error>

Resolve a path and return an owned Dataset handle.

The dataset uses the file-wide chunk-cache default (configured with FileAccessProperties::with_chunk_cache). To override the cache for this one dataset, use dataset_with_options.

Returns Error::NotADataset if the path names something that is not a dataset, and Error::NotAGroup if a component along the path is not a group: resolving a/b/c opens a and then a/b to look inside them, so a dataset at a/b reports NotAGroup("a/b") (issue #365).

pub fn dataset_with_options( &self, path: &str, properties: DatasetAccessProperties, ) -> Result<Dataset, Error>

Resolve a path and return an owned Dataset handle, applying per-dataset DatasetAccessProperties that override file-wide access defaults.

This is the dataset-open-with-access-property-list path (HDF5’s dapl): the properties’ chunk cache corresponds to H5Pset_chunk_cache and takes precedence, for this dataset only, over the H5Pset_cache-style file-wide default.

pub fn group(&self, path: &str) -> Result<Group, Error>

Resolve a path and return an owned Group handle.

Returns Error::NotAGroup if the path names an object that is not a group, the way dataset returns Error::NotADataset for the mirror case, and FormatError::PathNotFound if it names nothing.

The same error reports a component along the path that is not a group, naming that component’s own path rather than the one asked for: a/b/c stopped by a dataset at a/b reports NotAGroup("a/b") (issue #365).

pub fn refresh(&mut self) -> Result<(), Error>

Re-read the file from disk to pick up data appended by a concurrent writer, then re-parse the superblock.

This is the SWMR reader’s refresh primitive. Returns Error::SwmrUnsupported if the file was not opened with File::open_swmr, and Error::HandlesOutstanding if any owned Dataset/Group handle (or a clone of this File) is still alive — drop them before refreshing, then re-fetch them afterward, since they observe the new bytes only when re-derived from the refreshed file.

pub fn as_bytes(&self) -> &[u8]

Returns the raw file bytes for an in-memory file, or an empty slice for a streaming file (which has no whole-file buffer).

pub fn access_properties(&self) -> FileAccessProperties

Return the access properties used when opening this file.

pub fn edit_backing(&self) -> Option<EditBacking>

Which backend this file’s read-write session resolved to: EditBacking::Bounded when it reads through a handle, or EditBacking::Mirrored when it holds a whole-file image.

This is how a caller who opened with MemoryStrategy::Auto finds out whether the fallback was taken, and so whether memory scales with the file. A file with no editing session — a read-only open, a streaming open — reports None.

The answer is an EditBacking rather than the MemoryStrategy that was asked for, because Auto is a preference between the two backends and not an outcome either can report; .into() converts back when a later reopen should be pinned to what this one got.

pub fn metadata_cache_stats(&self) -> Option<MetadataCacheStats>

What this file’s metadata cache has done, and what it is holding.

FileAccessProperties::with_metadata_cache sets a byte budget before any read has happened; this is how a caller finds out whether it was the right one. See MetadataCacheStats for which figure answers which question. Together the two are the hdf5-pure counterpart of HDF5’s H5Fget_mdc_hit_rate and H5Fget_mdc_size.

None where there is no metadata cache to report on: a buffered open or from_bytes, which already holds the whole file; a mirrored read-write session, for the same reason; or a streaming or bounded open left at the default disabled budget.

use hdf5_pure::{File, FileAccessProperties, MetadataCacheConfig};

let properties =
    FileAccessProperties::new().with_metadata_cache(MetadataCacheConfig::new(8 << 20));
let file = File::open_streaming_with_options("data.h5", properties)?;
for name in file.root().datasets()? {
    let _ = file.dataset(&name)?.read_raw()?;
}

let stats = file.metadata_cache_stats().expect("the budget enabled a cache");
println!("{:?} over {} reads, {} evicted", stats.hit_rate(), stats.reads(), stats.evictions());

pub fn reset_metadata_cache_stats(&self)

Zero this file’s metadata-cache counters, keeping every cached entry.

HDF5’s H5Freset_mdc_hit_rate_stats, for measuring one phase of a program rather than a whole run: the reads that populate a cache miss by definition, so a hit rate taken over the run charges the steady state for the warm-up. Reset after warming to measure the part that repeats.

It evicts nothing: occupancy, which metadata_cache_stats also reports, is a measurement of the cache rather than a tally of its history. A file with no metadata cache ignores the call.

pub fn superblock(&self) -> &Superblock

Returns a reference to the parsed superblock.

pub fn file_space_strategy(&self) -> Option<FileSpaceStrategy>

The file-space management strategy this file records in its superblock extension, or None if it records none.

pub fn file_space_info(&self) -> Option<&FileSpaceInfo>

The full FileSpaceInfo recorded in this file’s superblock extension, if present and readable.

pub fn persisted_free_space(&self) -> Vec<(u64, u64)>

The free regions a file persists on disk in its free-space managers, as (address, length) pairs sorted by address.

pub fn file_size(&self) -> u64

The size of the underlying file in bytes (the HDF5 H5Fget_filesize).

pub fn libver_bound(&self) -> LibVer

The minimum library version required to read this file, derived from its superblock version (the low bound of HDF5’s H5Fget_libver_bounds).

Trait Implementations

impl Clone for File

fn clone(&self) -> File

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for File

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.