hdf5_pure/

edit.rs

//! In-place editing of an existing HDF5 file (issue #32, Group C).
//!
//! The in-place edit engine opens an existing file and adds objects, overwrites dataset
//! values, or edits compact group attributes **in place**:
//! new data and object headers are written at the end of the file, and the
//! object headers of the touched groups (and their ancestors up to the root)
//! are rewritten — also appended — so the superblock ends up pointing at the
//! new root header. Nothing already in the file is moved, so the cost is
//! proportional to what you add, not to the file size — unlike the
//! read-everything-then-rebuild path through [`FileBuilder`](crate::FileBuilder).
//!
//! Both new datasets, new (sub)groups, and group attribute edits are supported,
//! at any existing group path. Adding into a nested group `/a/b` rewrites `b`'s
//! header (with the new link), then `a`'s header (repointing its link to `b`'s
//! new location), then the root's — "relocation up the tree". This is always
//! safe for *additions* because no surviving object is relocated except the
//! groups on the path being edited, and those are reachable only through links
//! this same commit rewrites (the root through the superblock); absolute
//! object-reference addresses to other objects stay valid.
//!
//! Deletion ([`Group::delete`](crate::Group::delete), the HDF5 `H5Ldelete`) is the mirror image:
//! the parent group's header is rebuilt without the removed link, relocated up
//! the tree the same way, and the unlinked object (and its subtree) is freed —
//! its blocks are returned to a session-local free list (see below). A deletion
//! and an addition at the *same* path in one commit is a **replacement** (issue
//! #305): the link is removed from the rebuilt parent before the new object's is
//! appended, and the one superblock write publishes both, so a rotating store
//! expresses a rotation as one commit and the path is never momentarily absent.
//! The new object's storage is still appended rather than laid over the old
//! one's — the original stays live until the superblock repoint, which is what
//! makes a crash during the rotation land on one side or the other — so the
//! space the deletion released is what a *later* commit draws on, by reuse or by
//! truncation.
//! Object copy ([`File::copy`](crate::File::copy), the HDF5 `H5Ocopy`) deep-copies
//! a source subtree — appending fresh copies of every object, repointing internal
//! links and the contiguous data address — and links the copy in like an
//! addition; the headers are reproduced from their verbatim message bytes, so
//! datatypes, dataspaces, and attributes stay byte-exact. A chunked (and filtered)
//! dataset is copied with its chunk payloads and filter pipeline preserved
//! byte-for-byte, its index rebuilt at the new location. The same machinery,
//! [`File::copy_from`](crate::File::copy_from), copies an object **across two open files** — the
//! source being a separate [`File`](crate::File) reader rather than the file being
//! edited. Because the copy is byte-for-byte, the cross-file path refuses anything
//! that embeds a source-file absolute address (variable-length or reference data,
//! a committed datatype), which an in-file copy keeps valid by sharing the source
//! file's heaps and objects.
//!
//! Value overwrite ([`Dataset::write`](crate::Dataset::write), the HDF5 `H5Dwrite`) replaces
//! an **existing** dataset's values. The replacement's datatype and shape must
//! match the on-disk dataset (an overwrite, not a reshape or retype); contiguous,
//! compact, and chunked (including filtered) datasets are all supported, the chunk
//! geometry and filter pipeline taken from the on-disk header. A same-length
//! contiguous overwrite is the cheapest edit there is — the new bytes go straight
//! into the existing data block, so no header is rewritten and the superblock root
//! is not flipped, and the synced data write is the commit's linearization point.
//! A chunked overwrite takes the same in-place path when every (re-encoded) chunk
//! still fits its slot — always for unfiltered storage (chunk sizes are fixed by
//! the unchanged shape), and for filtered storage when the re-encoded chunks match.
//! When a length differs (a resized contiguous block, a filtered chunk that no
//! longer fits, or a compact dataset) the dataset's storage is rebuilt and its
//! header relocated like an addition: the new data and a rewritten header are
//! appended, the data-layout message is repointed, the old storage is freed, and
//! the parent group's link is patched. A relocating overwrite of a dataset
//! reachable through more than one hard link is refused, since only the one named
//! link could be repointed at the moved header.
//!
//! # Scope
//!
//! It is deliberately strict: rather than silently produce a degraded file, it
//! refuses with [`Error::EditUnsupported`] any case it cannot reproduce
//! faithfully. Requirements:
//!
//! - The file uses 8-byte offsets/lengths. A **userblock** (non-zero base
//!   address, as every MATLAB v7.3 `.mat` file has) is supported: addresses are
//!   read and written relative to the base and the userblock bytes are preserved
//!   verbatim. Every edit works on a userblock file — value overwrites, additions
//!   of contiguous and chunked/filtered datasets, in-place and relocating
//!   overwrites of every layout (with the old storage reclaimed), object deletion
//!   (with base-aware subtree reclaim), in-file copy, cross-file copy into a
//!   userblock destination, group creation, compact attributes, and free-space
//!   reuse. The one userblock-specific limitation left is cross-file copy *from* a
//!   userblock source (the source must have base 0; see [`copy_from`](crate::File::copy_from)).
//!   Any superblock version (0–3) is accepted: a version 0/1
//!   (symbol-table) file is edited by converting each group on the edited path
//!   to the latest format and repointing the superblock's root symbol-table
//!   entry.
//! - A version 2/3 group on an edited path stores its links compactly (not in a
//!   dense fractal heap); headers split across continuation chunks (as the
//!   reference C library often writes) are collapsed into a single chunk when
//!   rewritten. A version 1 group is converted to a compact-link v2 header,
//!   carrying its links and attributes over (other group messages — symbol
//!   table, modification time — are dropped); an attribute it cannot reproduce
//!   is refused.
//! - An object that tracks **attribute creation order** (`track_order=True`,
//!   and everything netCDF-4 writes) is edited like any other: its 6-byte
//!   message records are walked and re-emitted, a new attribute takes the
//!   object's next creation index, an overwrite keeps the one it had, and a
//!   deletion leaves a gap rather than renumbering. A group that tracks **link**
//!   creation order works the same way: a link added to it takes the next index
//!   from the running maximum its Link Info message records, and a deletion
//!   leaves a gap there too. Only an addition that would send such a group's
//!   links *dense* is refused, since the creation-order B-tree that indexes them
//!   is not written here.
//! - Added datasets may be contiguous *or* chunked, with any filter the
//!   whole-file writer supports (deflate, shuffle, fletcher32, scale-offset,
//!   LZF, ZFP), and may declare extensible (maximum, optionally unlimited)
//!   dimensions. A chunked dataset's data and index — and any filtered chunks —
//!   are produced by the same builder the whole-file writer uses and appended at
//!   end-of-file, so its object header is byte-identical to a freshly written
//!   one. A dataset may be empty (zero-element) under either storage, which is
//!   how an extensible dataset is created before the first
//!   [`append_staged`](crate::Dataset::append_staged) fills it: a contiguous one
//!   gets the undefined data address, a chunked one an index over zero chunks.
//!   A provenance dataset (`with_provenance`) is
//!   supported, its attributes computed the same way the whole-file writer
//!   computes them. A contiguous dataset may carry a variable-length-string
//!   payload (`with_vlen_strings`) or per-element object-reference targets
//!   (`with_path_references`); chunking either is not supported. A
//!   path-resolved reference may target any object this commit is not itself
//!   still writing (an ancestor group, a same-depth sibling group ordered
//!   later in the same commit, a copy destination or its interior, a
//!   `write_dataset` target, or an object this commit deletes) — targeting
//!   one of those is refused, up front and before any byte of the commit is
//!   written, rather than resolved to a stale or wrong address; a path that
//!   resolves nowhere at all becomes an undefined reference, matching the
//!   whole-file writer. Every
//!   added dataset must have a fixed-size datatype. Group, root, and **dataset**
//!   attribute edits (`set_group_attr` / `set_dataset_attr`) may be fixed-size or
//!   variable-length. An attribute set larger than an object header holds
//!   compactly — more than [`MAX_COMPACT_ATTRS`] of them, or one whose message
//!   overflows the header's 2-byte size field — is stored in a fractal heap, on
//!   an object this engine adds and on one whose edit takes it past the
//!   threshold alike, and an object already storing its attributes densely is
//!   rebuilt the same way ([`plan_attr_ops`], issue #102). A dataset attribute
//!   edit relocates the dataset header and so requires a single hard link.
//! - A new group's parent must already exist or be created in the same session
//!   (each level created explicitly); intermediate groups are not auto-created.
//! - Rows can be appended to an existing chunked, unlimited, Extensible-Array
//!   dataset **immediately and in place** with an in-place append (amortized O(1),
//!   crash-atomic, no `commit`), interleaved with the staged edits above. A
//!   target the fast path cannot handle — a userblock or pre-v2 file, an
//!   unallocated index, a non-Extensible-Array or multi-hard-link dataset, a
//!   a filtered dataset sitting on a partial trailing chunk — is refused with
//!   [`Error::AppendInPlaceUnsupported`]; use the staged `append_dataset` instead.
//!
//! # Free-space reuse (issue #21)
//!
//! Each commit vacates space: the object headers it rewrites are superseded, and
//! a deletion abandons its target's blocks. Those regions are recorded in a
//! session-local free list and drawn on by later writes in the same session —
//! a new object is written into a fitting freed region instead of growing the
//! file, and when freed space forms a run reaching end-of-file the file is
//! physically truncated. The reuse is crash-safe: it only ever overwrites space
//! freed by an *earlier*, already-durable commit (never space the current commit
//! is mid-way through freeing), and truncation happens only after the superblock
//! recording the smaller end-of-file is itself durable. A commit that fails
//! before its repoint gives back what it drew, so a failed attempt costs the
//! session nothing.
//!
//! Both write paths draw on that list, through allocators of their own. The
//! immediate [`Dataset::append`](crate::Dataset::append) allocates its chunks and
//! index blocks through [`Store::alloc_raw`], which
//! [`WriteEngine::immediate_reuse_allowed`] gates: an append has no superblock
//! repoint to publish its allocations with, so it reuses only where the space it
//! overwrites is dead on the disk as it stands (issue #349). On a file that
//! **persists** its managers a hole in the list is one the disk still advertises,
//! so the append does not spend it directly: it takes a batch out of the
//! published managers first
//! ([`WriteEngine::reserve_for_immediate_append`], issue #387) and spends only
//! that, which makes the on-disk record true of every byte it writes at every
//! instant. What the session does not spend goes back before the managers are
//! next rewritten; a session that dies in between strands the remainder, which is
//! accounted to nobody and so can never be handed out twice. The SWMR writer
//! reuses nothing at all, for a reason publication does not reach: its readers
//! may still be inside the region. Everything
//! a *commit* places goes through the other allocator
//! ([`WriteEngine::reserve`] and [`WriteEngine::place`]), including a chunked
//! dataset's data region and a dense attribute heap. Those carry addresses of
//! their own, so they are sized before they are placed and then built for the
//! address they got — which is why they can land in a freed region at all
//! (issue #261). A paged file (`H5F_FSPACE_STRATEGY_PAGE`) draws only from free
//! space of the page type it is placing, so reuse cannot make metadata and raw
//! data share a page — except from pages that are *wholly* free, which hold
//! nothing of either type to be mixed with and so may be opened for whichever
//! type asks ([`PagedEdit::alloc_typed`]). The free-space rewrite a commit
//! performs is placed through the same allocator where anything fits, rather than
//! at end-of-file (which on a paged file costs a page of its own), and that is
//! what keeps a delete-and-recreate workload from growing by a tail per commit
//! (issue #286 for the paged tail, issue #358 for the flat one).
//!
//! Reclaim is best-effort and conservative. Contiguous and chunked datasets
//! (chunk index plus chunk data) and whole group subtrees are reclaimed; a
//! deleted object whose blocks cannot be enumerated exhaustively —
//! variable-length global-heap storage, dense attribute/link heaps, a
//! non–version-2 header, a version 2 B-tree chunk index — is left as dead bytes
//! rather than risk freeing a region that is still in use; under-reclaiming only
//! wastes space, while over-reclaiming would corrupt.
//!
//! On a paged file that conservatism extends to anything whose *page type* is not
//! established. A file another writer produced records free space this one cannot
//! place — the reference library's generic-large manager holds metadata and raw
//! alike, and it puts a chunk index among its metadata where this crate puts one
//! beside the chunk data. Such space is kept and written back where it was found,
//! but never handed to an allocation
//! ([`PagedEdit::unclassified`], [`WriteEngine::index_is_provably_raw`]), because
//! placing a byte of the wrong kind in a page is the one thing paging exists to
//! prevent and no reader would report it.
//!
//! Space this editor vacates and cannot place is *dead* rather than lost
//! ([`PagedEdit::dead`]): out of use, not yet reusable. A page whose every byte is
//! free or dead holds nothing of either type, so it is promoted whole into the
//! free lists ([`PagedEdit::promote_whole_free_pages`]) — the same rule that lets
//! one page type claim a wholly free page from the other. That is what bounds a
//! paged file under delete-and-recreate churn: reclaim lands at page granularity
//! where the page type cannot be shown, rather than not at all (issue #388).
//!
//! Whether the free list outlives the session depends on how the file was
//! created. For the default (non-persisting) file it is **not** persisted: it is
//! forgotten on close, so reuse and shrinkage apply to churn within a session,
//! and a single delete-then-close shrinks the file only when the freed bytes
//! reach end-of-file. A file created with
//! `H5Pset_file_space_strategy(persist = true)` instead **persists** its free
//! space: `open` seeds the list from the on-disk free-space managers (the
//! `FSHD`/`FSSE` blocks the superblock-extension File Space Info message points
//! at), and each commit rewrites those managers, so freed regions survive
//! close/reopen and are reused across sessions — by this crate and the reference
//! C library alike. Such a commit *records* the freed space it keeps rather than
//! forgetting it; the blocks holding the managers go into space an earlier commit
//! freed, or past all live data when none fits, and the superblock is repointed
//! last, so a crash before the repoint leaves the prior file wholly intact.
//!
//! Freed space that reaches the end of the file is given back there too
//! (issue #418): the commit lowers the end-of-allocation to where the run starts,
//! drops those sections from the managers it writes, and truncates the file once
//! the smaller superblock is durable — page-aligned on a paged file, so the end
//! of allocation stays a whole number of pages. Two things bound it. A few tails'
//! worth of the run stays ([`release_trailing_run`]), because the manager blocks
//! are rewritten by every commit and can never land in their own predecessor's
//! extent, so a file trimmed closer than that sends its next tail past the end of
//! the file instead of into the reserve; and a run that would give back less than
//! it keeps is left alone, since
//! trimming the top off a hole the next write was about to fill costs a whole
//! object to return a fraction of one. A commit whose own tail had to be appended
//! *above* the run it freed takes a second, tail-only rewrite to move the blocks
//! down into it ([`WriteEngine::shrink_to_the_trailing_free_run`]) — the delete
//! case, where the freed regions only become writable once the repoint that
//! freed them is durable. Whole-file compaction that reclaims every hole at once
//! is still the separate repack path.

use std::borrow::Cow;
use std::collections::{BTreeMap, HashMap, HashSet};
use std::fs;
use std::io::{Read, Seek, SeekFrom};
use std::path::Path;

use core::num::NonZeroUsize;

use crate::access_mode::AccessMode;
use crate::address::{BaseAddress, StoredAddress};
use crate::attribute_info::AttributeInfoMessage;
use crate::checksum::jenkins_lookup3;
use crate::chunk_index_inplace::{Located, Store, apply_ea_append, plan_ea_append};
use crate::chunked_read::{
    chunk_index_spans_from_source, enumerate_chunks_from_source, plan_dense_grid,
};
use crate::chunked_write::{
    ChunkMeta, ChunkOptions, ChunkProvider, StorageAllocation, WrittenChunk, assemble_chunked_at,
    build_extensible_array_at, chunked_data_len, compress_chunks, emit_chunked_data_verbatim,
    extensible_array_len, full_chunk_bytes, plan_chunked_data_verbatim,
    serialize_v4_extensible_array, split_into_chunks,
};
use crate::convert::Narrow;
use crate::data_layout::{ChunkIndexLayout, DataLayout};
use crate::dataspace::{Dataspace, DataspaceType, Extent, MaxExtent};
use crate::datatype::{
    Datatype, datatype_holds_file_address, datatype_holds_object_address, embedded_reference_slots,
    stored_object_references,
};
use crate::error::{Error, FormatError, OBJECT_HEADER_MESSAGE_MAX};
use crate::extensible_array::ExtensibleArrayHeader;
use crate::file_create_properties::FileCreateProperties;
use crate::file_lock::{self, FileLocking};
use crate::file_space_info::{FileSpaceInfo, FileSpaceStrategy, NUM_FILE_FSM_MANAGERS};
use crate::file_writer::{
    DenseAttrCreationOrder, LENGTH_SIZE, OFFSET_SIZE, build_chunked_dataset_oh, build_dataset_oh,
    make_link,
};
use crate::filter_pipeline::{
    FILTER_DEFLATE, FILTER_FLETCHER32, FILTER_LZF, FILTER_SCALEOFFSET, FILTER_SHUFFLE,
    FilterPipeline,
};
use crate::filters::{ChunkContext, FilterScratch, compress_chunk_with, decompress_chunk};
use crate::free_space::{FreeList, trailing_run_start};
use crate::free_space_manager::{
    self, FreeSection, FsmHeader, PageType, PagedManagerPlan, SECT_CLASS_SIMPLE, align_up,
    file_fsm_blocks_len, fshd_len, plan_paged_managers, serialize_file_fsm,
};
use crate::group_v2::resolve_group_entries_from_source;
use crate::image::{FileImage, HandleImage, MirrorImage, WriteBuffering};
use crate::libver::LibVer;
use crate::link_info::LinkInfoMessage;
use crate::link_message::{LinkMessage, LinkTarget};
use crate::message_flags::MessageFlags;
use crate::message_type::MessageType;
use crate::object_header::ObjectHeader;
use crate::object_path::{LinkNameBuf, ObjectPathBuf};
use crate::reader::FileAccessProperties;
use crate::shared_message::DatatypeLocation;
use crate::source::{BaseOffsetSource, BytesSource, MetadataCacheConfig, Source};
use crate::superblock::Superblock;
use crate::type_builders::{
    AttrValue, DatasetBuilder, ObjectRefPatch, ObjectRefTarget, VlStringStaging,
    build_attr_message, build_global_heap_collections, make_f32_type, make_f64_type, make_i8_type,
    make_i16_type, make_i32_type, make_i64_type, make_u8_type, make_u16_type, make_u32_type,
    make_u64_type, patch_vl_refs, patch_vl_refs_masked, write_reference_address,
};
use crate::width::UintWidth;
use crate::{DatatypeByteOrder, signature};

/// An undefined on-disk address (all bits set), HDF5's "no address" sentinel.
const UNDEF: u64 = u64::MAX;

/// The most free space an immediate in-place append takes out of a persisting
/// file's on-disk managers in one draw, when the allocation in front of it needs
/// less (issue #387).
///
/// Every draw costs a manager rewrite and a superblock repoint — a few kilobytes
/// and two ordering barriers — so drawing per allocation would put a commit
/// between every pair of chunks and make reuse cost more than the growth it
/// avoids. A draw therefore takes as much as this of whatever holes can each hold
/// the allocation, largest first, and a megabyte covers sixteen chunks at the
/// 64 KiB scale the C library's own chunk-cache default is sized for, which puts
/// the rewrite well outside the per-chunk path.
///
/// It is a cap, not a floor. A file whose holes are all smaller than this still
/// has them drawn, one rewrite for however many of them fit under the cap: a
/// floor here made every persisting file whose deleted objects were under a
/// megabyte grow without bound, since no hole it left was ever spent
/// (issue #413). What a draw refuses is only a hole too small for the allocation
/// in hand, which nothing could be placed in. Every hole exists because a commit
/// freed it, so the rewrites spent reusing them are bounded by the commits the
/// caller already paid for.
///
/// It bounds one draw, not a session's total. A fresh draw is made whenever the
/// largest reserved region is smaller than the allocation in hand, and the
/// remainders of earlier draws are kept and still handed out, so what an abrupt
/// end strands is the sum of every remainder still held: each one smaller than
/// the allocation that forced the next draw, and one more of them possible per
/// draw. Larger would amortize further and strand more; this is the point where
/// the rewrite is already rare enough that making it rarer buys little.
const APPEND_RESERVE_BYTES: u64 = 1 << 20;

/// The refusal both address-side reference screens report: an object reference
/// this commit writes names an object the same commit removes.
///
/// One constant because the screens differ only in where the address came from
/// — [`WriteEngine::resolve_reference_target`] takes it from a builder's
/// [`ObjectRefTarget::Raw`], [`screen_resolved_references`] reads it out of a
/// dataset's or an attribute's element bytes — and a caller must not be able to
/// tell them apart by the message. It is the address-side twin of the by-name refusal
/// `resolve_reference_target` reports for a path, and shares its second clause.
const REFERENCE_INTO_RECLAIMED_SPACE: &str = "a reference this commit writes holds the address of an object this commit deletes, or of \
     one under it; the reference would be left pointing at storage the delete can reclaim";

/// The refusal for the other half of [`InvalidatedAddresses`]: the object is
/// still there, but not at that address any more.
///
/// It can offer a way out, which the removal refusal cannot: the target still
/// exists, so a path names it. Whether the path *works* depends on which half of
/// [`InvalidatedAddresses::moved`] the target came from, which is why the message
/// names separate commits as well rather than promising the first.
/// [`WriteEngine::resolve_reference_target`] resolves a **dirty group** once this
/// commit has placed it, and reports "still writing" until then; a **relocating
/// dataset write** it refuses outright, by exact match against `write_targets`,
/// with no later point at which that changes.
const REFERENCE_TO_A_MOVED_OBJECT: &str = "a reference this commit writes holds the pre-commit address of an object this commit \
     rewrites elsewhere; name the target by path (`with_path_references`) so it resolves to \
     where the object lands, or use separate commits";

/// A shared (SOHM) attribute message is held in the file's shared-message table
/// rather than in the object whose attribute it is, so rewriting the object's
/// attributes would have to account for the table as well. Both attribute paths
/// refuse it by the same name: the compact one in [`parse_compact_attr_name`],
/// which walks the messages it is about to copy, and the dense one in
/// [`plan_attr_ops`] — where the set is read through
/// [`crate::attribute::extract_attributes_full_from_source`], which *resolves*
/// such a message and would otherwise re-emit it into the new heap as a private
/// one, leaving the table's reference count naming an attribute that no longer
/// exists.
///
/// Since issue #417 the reader follows such a reference, so these are live
/// refusals rather than backstops: what they hold back is the *table*, whose
/// reference count this engine cannot yet decrement, not the read. `repack`
/// rewrites the file with every shared message inline, which is the way through
/// today.
const SHARED_ATTRIBUTE_MESSAGE: &str =
    "a target object has a shared attribute message (not editable in place yet)";

/// A shared-message (SOHM) index record does not always name a heap entry: where
/// the reference C library shares a message written into an object header it
/// already holds open, the record names *that header* and the message stays
/// where it is. Such a record is a stored address like any other, and it is the
/// one no rewrite reaches — the index lives outside every object a commit
/// rebuilds, and this engine writes no shared-message indexes. A commit that
/// removed or moved the named header would leave the file's own index pointing
/// at bytes that are no longer there, which the next library to share a message
/// would follow.
const SHARED_MESSAGE_INDEX_NAMES_A_MOVED_OBJECT: &str = "this file's shared-message (SOHM) index names an object header this commit \
     removes or rewrites elsewhere, and the index cannot be updated in place; \
     rewrite the file with `repack` instead";

/// Refusal for an attribute whose references a commit repoints today and could
/// not repoint from a heap. See [`plan_attr_ops`].
const REFERENCE_ATTRIBUTE_WOULD_LEAVE_THE_HEADER: &str = "an attribute holding an object reference cannot be moved to dense \
     (fractal-heap) storage, where a later commit could no longer repoint it when \
     its target moves; keep this object within compact attribute storage";

/// Maximum number of attributes an object header keeps inline; past this, HDF5
/// switches the object to dense (fractal-heap) attribute storage, which this
/// engine emits through [`plan_attr_ops`] and [`WriteEngine::place_dense_attrs`].
///
/// Taken from the whole-file writer's threshold rather than restated beside it:
/// the two decide the same thing about the same attribute set, and a file whose
/// objects were written by one and edited by the other must not disagree about
/// where the eighth attribute lives.
const MAX_COMPACT_ATTRS: usize = crate::file_writer::DENSE_ATTR_THRESHOLD;

/// Recursion-depth cap for object copy, guarding against a stack overflow on a
/// pathological or cyclic hard-link graph (HDF5 hard links can form cycles).
/// Far deeper than any real group hierarchy.
const MAX_COPY_DEPTH: u32 = 1000;

/// Upper bound on the number of object headers walked when counting hard links
/// across the file (issue #77 / reclaim safety). Far beyond any real file; a
/// graph larger than this aborts the count, and the commit then leaves deleted
/// objects unreclaimed (a safe leak) rather than risk an unbounded walk.
const MAX_LINK_GRAPH_NODES: u32 = 1 << 24;

/// Maximum number of object-header chunks to follow when gathering a header that
/// spans continuation blocks, guarding against a cyclic continuation chain.
/// Matches the reader's continuation-depth cap.
const MAX_OH_CHUNKS: usize = 256;

/// Maximum length of a version 2 object header's fixed prefix: signature (4) +
/// version (1) + flags (1) + optional access/modification/change/birth times
/// (16) + optional attribute phase-change thresholds (4) + the chunk-0 size
/// field (up to 8). Reading this many bytes always covers the prefix, so
/// [`oh_region_at`] can be handed one bounded window instead of a whole-file
/// image.
const OH_PREFIX_MAX: usize = 34;

/// A live [`BufferedAppender`](crate::BufferedAppender)'s hold on a dataset.
///
/// The appender accepts elements into memory and is the only thing that can
/// write them, so any staged edit that would make its flush refuse turns
/// accepted data into lost data at drop time. The claim lets the engine refuse
/// that edit up front instead.
struct AppenderClaim {
    /// Identifies this claim for release; ids are never reused within a session.
    token: u64,
    /// The appender's dataset path, or `None` for a handle reached by object
    /// reference. Such a handle is named by object-header address, which *any*
    /// staged edit may move, so a path-less claim conflicts with everything —
    /// exactly the rule `append_prepare` already applies to that target.
    path: Option<ObjectPathBuf>,
}

impl AppenderClaim {
    /// Returns `true` if this claim covers the dataset `path` identifies, or if either side has
    /// no path.
    fn conflicts_with(&self, path: Option<&ObjectPathBuf>) -> bool {
        match (&self.path, path) {
            (Some(claimed), Some(candidate)) => claimed == candidate,
            _ => true,
        }
    }
}

/// A variable-length group/root attribute staged by [`apply_compact_attr_ops`]
/// and resolved in the apply loop.
#[derive(Clone)]
struct PendingVlAttr {
    /// The attribute message, its global-heap references still placeholders.
    msg: crate::attribute::AttributeMessage,
    /// The collections whose real addresses those references need.
    collections: Vec<Vec<u8>>,
    /// The creation index this attribute keeps, taken from the message it
    /// replaces on a header that tracks attribute creation order. `None` asks
    /// for the object's next unused index when the message is appended — which
    /// is every case on a header that does not track the order.
    creation_index: Option<u16>,
}

/// Variable-length group/root attributes staged by [`apply_compact_attr_ops`].
type PendingVlAttrs = Vec<PendingVlAttr>;

/// Accumulates elements to append to an existing chunked, unlimited dataset via
/// [`Dataset::append_staged`](crate::Dataset::append_staged), in call order along the dataset's first
/// (axis-0) dimension.
///
/// It mirrors [`DatasetBuilder`]'s typed/generic vocabulary. Repeated typed or
/// [`append_raw`](Self::append_raw) calls concatenate; each typed method also
/// records the element datatype it implies, which `commit` checks against the
/// dataset's on-disk datatype (a mismatch — including a mix of element types in
/// one builder — is refused with [`Error::AppendUnsupported`], never written as
/// garbage).
pub struct AppendBuilder {
    /// Accumulated little-endian element bytes to append, in call order.
    raw: Vec<u8>,
    /// The element datatype implied by the typed `append_*` calls, if any were
    /// used. `None` when only [`append_raw`](Self::append_raw) was called (a raw
    /// append is checked structurally — element-size alignment and little-endian
    /// on-disk order — rather than by datatype equality).
    elem_dt: Option<Datatype>,
    /// Set when two typed calls implied different element datatypes; `commit`
    /// refuses such a builder rather than write a mix of encodings.
    dt_conflict: bool,
}

impl AppendBuilder {
    pub(crate) fn new() -> Self {
        Self {
            raw: Vec::new(),
            elem_dt: None,
            dt_conflict: false,
        }
    }

    /// Accumulated little-endian element bytes (for the general append writer,
    /// which reuses this builder to gather typed/generic appends).
    pub(crate) fn raw(&self) -> &[u8] {
        &self.raw
    }

    /// The element datatype implied by typed appends, if any.
    pub(crate) fn elem_dt(&self) -> Option<&Datatype> {
        self.elem_dt.as_ref()
    }

    /// Whether two typed appends implied conflicting element datatypes.
    pub(crate) fn dt_conflict(&self) -> bool {
        self.dt_conflict
    }

    /// A builder holding a copy of this one's first `byte_len` bytes, carrying
    /// the same element datatype so the prefix is type-checked exactly as the
    /// whole would have been. Used by [`BufferedAppender`](crate::BufferedAppender)
    /// to write out the chunk-aligned prefix of its buffer; it copies rather
    /// than splits so a failed write leaves the buffer intact and the appender
    /// can report precisely which elements did not land.
    pub(crate) fn head(&self, byte_len: usize) -> Self {
        Self {
            raw: self.raw[..byte_len.min(self.raw.len())].to_vec(),
            elem_dt: self.elem_dt.clone(),
            dt_conflict: self.dt_conflict,
        }
    }

    /// Discard the first `byte_len` buffered bytes (a prefix that reached the
    /// file), keeping the element datatype.
    pub(crate) fn drop_front(&mut self, byte_len: usize) {
        self.raw.drain(..byte_len.min(self.raw.len()));
    }

    /// Consume the builder, yielding its accumulated element bytes. Used by
    /// [`BufferedAppender::discard`](crate::BufferedAppender::discard) to hand
    /// abandoned elements back to the caller.
    pub(crate) fn into_raw(self) -> Vec<u8> {
        self.raw
    }

    /// Cut the buffer back to `byte_len` bytes, keeping the element datatype.
    /// [`BufferedAppender`](crate::BufferedAppender) uses this to undo a call
    /// whose write was refused before it touched the file, so the refusal leaves
    /// nothing buffered and a retry cannot append the same elements twice.
    pub(crate) fn truncate(&mut self, byte_len: usize) {
        self.raw.truncate(byte_len);
    }

    /// Record the datatype a typed append implies, flagging a conflict if an
    /// earlier typed call implied a different one.
    fn set_dt(&mut self, dt: Datatype) {
        match &self.elem_dt {
            Some(prev) if *prev != dt => self.dt_conflict = true,
            Some(_) => {}
            None => self.elem_dt = Some(dt),
        }
    }

    /// Append already-little-endian element bytes verbatim. The concatenated
    /// length must be a whole multiple of the dataset's on-disk element size, and
    /// the dataset's element datatype must be little-endian; no datatype is
    /// otherwise inferred. Prefer the typed methods when the element type is known.
    pub fn append_raw(&mut self, bytes: &[u8]) -> &mut Self {
        self.raw.extend_from_slice(bytes);
        self
    }

    /// Generic append of a flat slice of any supported scalar type — the
    /// counterpart of [`DatasetBuilder::with_data`](crate::DatasetBuilder::with_data).
    pub fn append<T: crate::element::H5Element>(&mut self, data: &[T]) -> &mut Self {
        T::append_into(self, data);
        self
    }
}

/// Generate the typed `append_*` methods: serialize each value little-endian and
/// record the implied element datatype.
macro_rules! append_typed {
    ($($method:ident, $ty:ty, $make:ident;)*) => {
        impl AppendBuilder {
            $(
                #[doc = concat!("Append `", stringify!($ty), "` values to the dataset.")]
                pub fn $method(&mut self, data: &[$ty]) -> &mut Self {
                    self.set_dt($make());
                    // Reserved up front: serializing element by element into a
                    // growing `Vec` re-allocated it ten times per call and copied
                    // the batch twice over, which for an append loop is the whole
                    // per-call cost (issue #228).
                    self.raw.reserve(data.len() * core::mem::size_of::<$ty>());
                    for &v in data {
                        self.raw.extend_from_slice(&v.to_le_bytes());
                    }
                    self
                }
            )*
        }
    };
}

append_typed! {
    append_f64, f64, make_f64_type;
    append_f32, f32, make_f32_type;
    append_i8, i8, make_i8_type;
    append_i16, i16, make_i16_type;
    append_i32, i32, make_i32_type;
    append_i64, i64, make_i64_type;
    append_u8, u8, make_u8_type;
    append_u16, u16, make_u16_type;
    append_u32, u32, make_u32_type;
    append_u64, u64, make_u64_type;
}

/// Every edit a session has staged and not yet committed, as one value.
///
/// [`WriteEngine::commit`] takes the whole set out for the duration of an
/// attempt and puts it back if that attempt refuses, so a refused commit costs
/// the session no staged work — the same guarantee [`FreeSnapshot`] gives the
/// free lists, for the same reason (issue #316). Keeping the vectors together
/// rather than beside the engine's other fields is what makes that total: a
/// staged kind added later participates by construction, where a tenth field
/// would have to be remembered in a hand-written restore.
#[derive(Default)]
struct StagedEdits {
    /// Datasets staged by `create_dataset`, as (parent group path, dataset).
    ///
    /// Flattened at staging rather than at commit: [`flatten_dataset`] is the
    /// one step of the commit's preflight that *consumes* what it validates, so
    /// running it here is what lets the preflight read the staged set without
    /// destroying it (issue #316). It is a pure function of the builder, so the
    /// guards it raises — a missing shape, data that does not match it, a
    /// feature this engine cannot reproduce — are answered at the call that
    /// stages the dataset, where the caller still has the context to fix them.
    datasets: Vec<(ObjectPathBuf, FlatDataset)>,
    /// Value overwrites staged by `write_dataset`, as (full dataset path,
    /// dataset). Each replaces an existing dataset's values in place; the new
    /// datatype and shape must match the on-disk ones byte-exactly (this is a
    /// value overwrite, not a reshape/retype). Applied on the next `commit`.
    /// Flattened at staging, for the reason given on [`datasets`](Self::datasets);
    /// the match against the on-disk dataset is a commit-time check and stays
    /// one, since it reads the file.
    writes: Vec<(ObjectPathBuf, FlatDataset)>,
    /// Appends staged by `append_dataset`, as (full dataset path, builder). Each
    /// grows an existing chunked, unlimited, Extensible-Array-indexed dataset
    /// along axis 0 by keeping its existing chunk data in place and rebuilding the
    /// index over the kept plus newly-appended (and any rewritten trailing) chunks.
    /// Applied on the next `commit`.
    appends: Vec<(ObjectPathBuf, AppendBuilder)>,
    /// New groups staged by `create_group`, as full paths.
    groups: Vec<ObjectPathBuf>,
    /// Group attribute edits staged as (group path, operation). The path may be
    /// a group created in this same session.
    group_attrs: Vec<(ObjectPathBuf, AttrOp)>,
    /// Dataset attribute edits staged as (full dataset path, operation), applied
    /// on the next `commit`. Each relocates the dataset's object header (like a
    /// relocating overwrite): the header is rebuilt with the compact-attribute
    /// change, its single naming link is patched, and the old header freed — the
    /// dataset's data and chunk index stay in place. The target must be an existing,
    /// single-hard-link dataset using compact (not dense fractal-heap) attributes.
    dataset_attrs: Vec<(ObjectPathBuf, AttrOp)>,
    /// Links staged for removal by `delete`, as full paths.
    deletes: Vec<ObjectPathBuf>,
    /// Object copies staged by `copy`, as (source path, destination full path).
    copies: Vec<(ObjectPathBuf, ObjectPathBuf)>,
    /// Cross-file object copies staged by `copy_from`, as (destination full path,
    /// the source subtree already read out of the other file). The subtree is read
    /// — and foreign-address-screened — eagerly in `copy_from` (the source file is
    /// borrowed only for that call), then linked in at the next `commit`.
    cross_copies: Vec<(ObjectPathBuf, CopyTree)>,
    /// Where the creation at each full path sits in [`datasets`](Self::datasets)
    /// / [`groups`](Self::groups), so a by-name question about the staged set
    /// costs a hash rather than a scan of everything staged.
    ///
    /// The scan is what made a writer that stages one dataset per column and
    /// then binds a handle to each quadratic in the number of columns (issue
    /// #392). Both maps are derived from the vectors beside them and are kept in
    /// step at the three places those change: the push that stages a creation,
    /// the truncation [`rewind`](Self::rewind) makes, and the rebuild
    /// [`withdraw_at`](Self::withdraw_at) makes after removing from the middle.
    /// A path staged twice keeps its *first* position, which is the entry a scan
    /// would have found; the commit refuses such a pair anyway.
    dataset_at: HashMap<ObjectPathBuf, usize>,
    group_at: HashMap<ObjectPathBuf, usize>,
}

/// What a session has staged at a path, for a handle addressing an object the
/// commit has not written yet.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub(crate) enum StagedKind {
    Group,
    Dataset,
}

/// A creation this session has staged at a path, and whether it takes that path
/// over from the file.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub(crate) struct StagedObject {
    pub(crate) kind: StagedKind,
    /// Whether this same commit removes the link the file holds at that path.
    ///
    /// True makes the creation a *replacement* (issue #305). False means the
    /// file holds no link there at all — a creation colliding with one that
    /// survives is refused where it is staged, and would not be reported here
    /// either, since the file's own object is what the name means.
    pub(crate) replaces_link: bool,
}

/// One direct child a group gains from this session's staged creations.
pub(crate) struct StagedChild {
    pub(crate) name: String,
    pub(crate) kind: StagedKind,
    /// Whether a staged deletion removes the link this name has in the file, so
    /// the creation supersedes it rather than colliding with it. See
    /// [`StagedObject::replaces_link`].
    pub(crate) replaces_link: bool,
}

/// What a staged dataset can say about itself before it is written.
///
/// Everything here is settled when the dataset is staged — [`flatten_dataset`]
/// has already validated the builder — so a handle answers from it without
/// reading the file, and the answers are what the commit goes on to write.
pub(crate) struct StagedMeta {
    pub(crate) datatype: Datatype,
    pub(crate) dimensions: Vec<u64>,
    /// `None` for a fixed-shape dataset, matching `Dataset::maxshape`.
    pub(crate) maxshape: Option<Vec<MaxExtent>>,
    pub(crate) chunked: bool,
    /// Each staged filter's registered id and its `H5Z_FLAG_OPTIONAL` flag, in
    /// pipeline order.
    pub(crate) filters: Vec<(u16, bool)>,
}

/// Where a [`StagedEdits`] stood before a batch of staging calls, so a batch
/// that fails partway can be undone (see [`StagedEdits::rewind`]).
#[derive(Clone, Copy, Default, PartialEq, Eq)]
struct StagedMark {
    datasets: usize,
    writes: usize,
    appends: usize,
    groups: usize,
    group_attrs: usize,
    dataset_attrs: usize,
    deletes: usize,
    copies: usize,
    cross_copies: usize,
}

impl StagedEdits {
    /// The length of every staged vector right now, for
    /// [`rewind`](Self::rewind).
    ///
    /// Destructured rather than read field by field, here and in
    /// [`rewind`](Self::rewind), so that the "a staged kind added later
    /// participates by construction" claim on [`StagedEdits`] is enforced rather
    /// than hoped for: a struct pattern naming fewer fields than the struct has
    /// does not compile, and the unused binding a tenth kind would leave in
    /// `rewind` is an error under the crate's `-D warnings`.
    fn mark(&self) -> StagedMark {
        let Self {
            datasets,
            writes,
            appends,
            groups,
            group_attrs,
            dataset_attrs,
            deletes,
            copies,
            cross_copies,
            // Derived from `datasets` and `groups`, and restored with them.
            dataset_at: _,
            group_at: _,
        } = self;
        StagedMark {
            datasets: datasets.len(),
            writes: writes.len(),
            appends: appends.len(),
            groups: groups.len(),
            group_attrs: group_attrs.len(),
            dataset_attrs: dataset_attrs.len(),
            deletes: deletes.len(),
            copies: copies.len(),
            cross_copies: cross_copies.len(),
        }
    }

    /// Drop everything staged since `mark`.
    ///
    /// Staging inside a batch only ever appends — every `stage_*` entry point
    /// validates and then pushes — so truncating to the recorded lengths is an
    /// exact undo of the calls made in between, and leaves anything staged
    /// before them alone. The two operations that instead *change* what is
    /// already staged — the fold
    /// [`WriteEngine::stage_dataset_append_pending`] makes into a staged
    /// creation, and the withdrawal [`withdraw_at`](Self::withdraw_at) makes for
    /// [`WriteEngine::delete`] — would leave that untrue, so
    /// [`WriteEngine::refuse_mid_batch`] refuses them while a batch is open
    /// rather than leaving the claim to be hoped for.
    ///
    /// The by-path index is truncated with the vectors it describes, so a
    /// rewound creation stops answering as well as stops existing.
    fn rewind(&mut self, mark: StagedMark) {
        for i in mark.datasets..self.datasets.len() {
            let key = {
                let (parent, fd) = &self.datasets[i];
                parent.join(&fd.name)
            };
            // Only where it still points at the entry being dropped: a second
            // creation at one path never displaced the first.
            if self.dataset_at.get(&key) == Some(&i) {
                self.dataset_at.remove(&key);
            }
        }
        for i in mark.groups..self.groups.len() {
            let key = self.groups[i].clone();
            if self.group_at.get(&key) == Some(&i) {
                self.group_at.remove(&key);
            }
        }
        let StagedMark {
            datasets,
            writes,
            appends,
            groups,
            group_attrs,
            dataset_attrs,
            deletes,
            copies,
            cross_copies,
        } = mark;
        self.datasets.truncate(datasets);
        self.writes.truncate(writes);
        self.appends.truncate(appends);
        self.groups.truncate(groups);
        self.group_attrs.truncate(group_attrs);
        self.dataset_attrs.truncate(dataset_attrs);
        self.deletes.truncate(deletes);
        self.copies.truncate(copies);
        self.cross_copies.truncate(cross_copies);
    }

    /// Stage a dataset creation, indexing it by its full path.
    fn push_dataset(&mut self, parent: ObjectPathBuf, fd: FlatDataset) {
        self.dataset_at
            .entry(parent.join(&fd.name))
            .or_insert(self.datasets.len());
        self.datasets.push((parent, fd));
    }

    /// Stage a group creation, indexing it by its path.
    fn push_group(&mut self, path: ObjectPathBuf) {
        self.group_at
            .entry(path.clone())
            .or_insert(self.groups.len());
        self.groups.push(path);
    }

    /// Where the staged dataset at `path` sits, if there is one.
    ///
    /// The position the index gives is checked against the entry it names
    /// rather than trusted: an index that fell behind the vector then reads as a
    /// miss — the answer a scan would give — instead of as another dataset.
    fn dataset_position(&self, path: &ObjectPathBuf) -> Option<usize> {
        let i = *self.dataset_at.get(path)?;
        let (parent, fd) = self.datasets.get(i)?;
        path.covers_child(parent, &fd.name).then_some(i)
    }

    /// The staged dataset at `path`, if there is one.
    fn dataset_at(&self, path: &ObjectPathBuf) -> Option<&FlatDataset> {
        let i = self.dataset_position(path)?;
        self.datasets.get(i).map(|(_, fd)| fd)
    }

    /// The staged dataset at `path`, mutably.
    fn dataset_at_mut(&mut self, path: &ObjectPathBuf) -> Option<&mut FlatDataset> {
        let i = self.dataset_position(path)?;
        self.datasets.get_mut(i).map(|(_, fd)| fd)
    }

    /// Whether a group creation is staged at `path`. Checked against the entry
    /// the index names, for the reason [`dataset_position`](Self::dataset_position)
    /// gives.
    fn has_group_at(&self, path: &ObjectPathBuf) -> bool {
        match self.group_at.get(path) {
            Some(&i) => self.groups.get(i).is_some_and(|p| p == path),
            None => false,
        }
    }

    /// Whether a staged deletion removes the link at `path`, or the link to an
    /// ancestor that carries it away.
    fn deletes_cover(&self, path: &ObjectPathBuf) -> bool {
        self.deletes.iter().any(|d| path.starts_with(d))
    }

    /// Whether a staged deletion hands `path` over to a creation staged there —
    /// which is what makes the pair a *replacement* rather than a collision
    /// (issue #305).
    ///
    /// Stricter than [`deletes_cover`](Self::deletes_cover), and deliberately:
    /// a deletion of an *ancestor* only carries the name over when this same
    /// session builds that ancestor again, which is exactly what the commit
    /// requires of everything at or under a replaced path. Deleting `g` and
    /// creating `g/x` without recreating `g` is a batch the commit refuses, so
    /// the file's own `g/x` is still what that name means — reporting it as a
    /// replacement would make a live, readable dataset answer
    /// [`Error::NotCommitted`] and vanish from its group's listing until the
    /// refusal.
    ///
    /// The exact deletion is the empty case of the same rule: with `d == path`
    /// there is no ancestor between them to have been recreated.
    fn deletes_hand_over(&self, path: &ObjectPathBuf) -> bool {
        self.deletes.iter().any(|d| {
            path.starts_with(d) && (d.len()..path.len()).all(|n| self.has_group_at(&path.prefix(n)))
        })
    }

    /// Withdraw every edit staged at or under `path`, as though the calls that
    /// staged them had never been made, and report whether a *creation* was
    /// among them.
    ///
    /// This is what lets [`WriteEngine::delete`] cancel an object this session
    /// staged and has not written: there is nothing in the file to unlink, so a
    /// deletion of it is a withdrawal. Deletions are deliberately left alone —
    /// one under a withdrawn creation still names a link the file holds, and two
    /// overlapping deletions are the commit's refusal to make rather than this
    /// call's.
    ///
    /// Destructured for the reason [`mark`](Self::mark) is: a staged kind added
    /// later has to be classified here rather than silently survive a
    /// withdrawal.
    fn withdraw_at(&mut self, path: &ObjectPathBuf) -> bool {
        let before = {
            let Self {
                datasets,
                writes,
                appends,
                groups,
                group_attrs,
                dataset_attrs,
                deletes,
                copies,
                cross_copies,
                dataset_at: _,
                group_at: _,
            } = self;
            let before = datasets.len() + groups.len();
            datasets.retain(|(parent, fd)| !path.covers_child(parent, &fd.name));
            writes.retain(|(p, _)| !p.starts_with(path));
            appends.retain(|(p, _)| !p.starts_with(path));
            groups.retain(|p| !p.starts_with(path));
            group_attrs.retain(|(p, _)| !p.starts_with(path));
            dataset_attrs.retain(|(p, _)| !p.starts_with(path));
            copies.retain(|(_, dst)| !dst.starts_with(path));
            cross_copies.retain(|(dst, _)| !dst.starts_with(path));
            let _ = &deletes;
            before
        };
        let withdrew = self.datasets.len() + self.groups.len() < before;
        if withdrew {
            self.reindex();
        }
        withdrew
    }

    /// Rebuild the by-path index from the vectors, after a removal from the
    /// middle shifted every position after it.
    fn reindex(&mut self) {
        self.dataset_at.clear();
        self.group_at.clear();
        for (i, (parent, fd)) in self.datasets.iter().enumerate() {
            self.dataset_at.entry(parent.join(&fd.name)).or_insert(i);
        }
        for (i, path) in self.groups.iter().enumerate() {
            self.group_at.entry(path.clone()).or_insert(i);
        }
    }

    /// Whether nothing at all is staged.
    ///
    /// Asked as "is the mark the zero mark", which is exact — nothing is staged
    /// exactly when every vector has length zero — and leaves [`mark`](Self::mark)
    /// as the single place that names every vector. The two callers,
    /// [`WriteEngine::has_staged_edits`] and the commit's own no-op return, then
    /// cannot come to disagree, and a staged kind added later reaches both. (The
    /// commit's *fast path* asks a narrower question and spells its own subset
    /// out.)
    fn is_empty(&self) -> bool {
        self.mark() == StagedMark::default()
    }
}

/// The in-place write engine behind the owned read-write [`File`](crate::File)
/// (its `Backend::Edit`).
///
/// Reads and edits the file through a [`FileImage`], which owns the writable
/// handle and decides how much of the file is resident. It carries two commit
/// models: staged tree edits applied by [`commit`](Self::commit), and immediate
/// crash-atomic in-place appends ([`append_inplace_gathered`](Self::append_inplace_gathered)).
pub(crate) struct WriteEngine {
    /// The file bytes this session reads and edits, behind the [`FileImage`]
    /// abstraction: reads go through its [`Source`] impl, and the write side —
    /// the end-of-file cursor, `append`, `write_at`, `truncate`, and the
    /// durability barriers — through its own primitives.
    ///
    /// Nothing in the engine assumes the whole file is resident, so one engine
    /// serves both a whole-file mirror and a file-backed image that holds only
    /// what it is reading (issue #198). [`image_slice`](Self::image_slice)
    /// exposes the mirror's buffer where a caller can exploit it.
    image: Box<dyn FileImage>,
    /// Absolute offset of the superblock signature in the file.
    sb_sig_off: u64,
    /// Parsed superblock. On-disk addresses are stored relative to `base_address`;
    /// the in-memory `root_group_address` is normalized to an absolute file offset
    /// on open and converted back to a base-relative address when serialized on
    /// commit. `base_address` equals the superblock's file location (`sb_sig_off`):
    /// 0 for a plain file, the userblock size for one with a userblock.
    superblock: Superblock,
    /// Every edit this session has staged and not yet applied. Held as one
    /// value so that [`commit`](WriteEngine::commit) can take the whole set out
    /// for the duration of an attempt and put it back when that attempt
    /// refuses; see [`StagedEdits`].
    staged: StagedEdits,
    /// Datasets with a live [`BufferedAppender`](crate::BufferedAppender), which
    /// holds accepted elements only it can write. A staged edit that would stop
    /// that appender from flushing is refused while the claim stands, rather
    /// than left to fail in the appender's `Drop`, where there is no caller to
    /// report it to and the buffer is simply lost. See `refuse_if_claimed`.
    appender_claims: Vec<AppenderClaim>,
    /// Monotonic id for the next claim, so releasing one is exact even when two
    /// appenders on different datasets are live at once.
    next_appender_token: u64,
    /// Session-local free-space tracker (issue #21). Holds regions vacated by
    /// prior commits in this session — superseded object headers and the blocks
    /// of deleted objects — so later commits reuse them instead of growing the
    /// file, and so a freed run reaching end-of-file can be truncated away. It
    /// starts empty on `open` for a non-persisting file: holes already present
    /// from earlier sessions or other tools are not tracked. When the file
    /// persists its free space (`persist` is `Some`), `open` instead seeds it
    /// from the on-disk free-space managers, so reuse spans sessions.
    free: FreeList,
    /// Space this session has taken *out* of the on-disk free-space managers so
    /// that an immediate in-place append may spend it (issue #387).
    ///
    /// Only a file that persists its free space uses this. Such a file records
    /// its holes on disk, and an append has no superblock repoint of its own to
    /// publish an allocation with — so an append that simply drew from
    /// [`free`](Self::free) would leave a manager advertising bytes a live chunk
    /// now occupies, through a clean close as much as a crash. The reserve is
    /// the answer: [`reserve_for_immediate_append`](Self::reserve_for_immediate_append)
    /// moves a batch of bytes out of `free` (or, on a paged file, out of
    /// [`PagedEdit`]'s raw list) and rewrites the managers *without* them, under
    /// the same crash-atomic superblock repoint every persisting commit uses.
    /// Only then may an append write there, because by then no durable record
    /// calls those bytes free.
    ///
    /// What is left unspent goes back — into `free` or `PagedEdit` — before the
    /// managers are next rewritten, which is [`release_reserve`](Self::release_reserve)
    /// at the next commit and at close. A session that dies in between strands
    /// the unspent remainder: it is accounted to nobody, so it is never handed
    /// out twice, and a later `H5repack` or whole-file rewrite recovers it. That
    /// is the same trade the module header records for
    /// [`finalize_persist`](Self::finalize_persist), on a surface bounded per
    /// draw — see [`APPEND_RESERVE_BYTES`], which is the most one draw takes,
    /// and note that this list holds the unspent remainder of every draw the
    /// session has made, not just the last.
    ///
    /// Empty for a non-persisting session, which reuses out of `free` directly:
    /// its list is in memory alone, so nothing on the disk claims those bytes
    /// are free and nothing has to be published before they are spent.
    reserved: FreeList,
    /// Whether this file has been *proved* to hold no object reference at all —
    /// the one thing that lets a commit skip the walk that repoints them
    /// (issue #324).
    ///
    /// Set by a walk that reaches every object and finds no reference-holding
    /// datatype; every later commit in this session then skips the walk
    /// outright, which matters because the walk is linear in the file's object
    /// count and the file that needs it is the exception.
    ///
    /// One bit rather than three states, because "not yet walked" and "walked
    /// and found one" are the same instruction to the caller: walk. Only the
    /// proof is worth carrying.
    ///
    /// Cleared by any commit that could introduce a reference the walk has not
    /// seen. Exactly one of [`StagedEdits`]' nine collections can: `datasets`,
    /// which carries a datatype and attributes the caller chose. The check
    /// covers four, and the other three are belt-and-braces rather than
    /// load-bearing — recorded here because a reason that does not hold is worse
    /// than no reason:
    ///
    /// - `writes` is a *value* overwrite, whose datatype must match the on-disk
    ///   dataset's exactly. A reference-typed one therefore overwrites a dataset
    ///   that already held references, which a file proved free of them does not
    ///   have. Checked anyway, because that match is a commit-time check running
    ///   after this one, so `fd.dt` here is still whatever the caller supplied.
    /// - `copies` re-emits an object *from this same file* — one the walk that
    ///   granted the proof had already visited and found reference-free.
    /// - `cross_copies` reads from another file, where `reject_foreign_addresses`
    ///   refuses a reference datatype outright.
    ///
    /// The remaining five cannot, and these reasons do hold:
    ///
    /// - `groups` stages a bare path: no datatype, no data, no attributes.
    /// - `group_attrs` and `dataset_attrs` carry an [`AttrValue`], which has no
    ///   reference variant — the reason a reference-typed attribute can only
    ///   have come from the reference C library.
    /// - `appends` grows an *existing* dataset along its own datatype, so it can
    ///   only add references to a dataset that already held them.
    /// - `deletes` removes objects and adds nothing.
    ///
    /// The commit fast path returns before the check and needs no reason of its
    /// own: it is taken only when every staged edit is a same-length in-place
    /// overwrite, which is the `writes` case above.
    ///
    /// A tenth collection would have to be classified the same way; the check
    /// sits beside the reference screen in `commit`, which iterates the same
    /// datatype-carrying collections.
    proved_free_of_references: bool,
    /// Free-space persistence read from the file's superblock extension on
    /// `open` (the file-creation `H5Pset_file_space_strategy(persist = true)`
    /// setting). `None` for the default non-persisting file; when `Some`, every
    /// [`commit`](Self::commit) rewrites the on-disk free-space managers so the
    /// free list survives close/reopen.
    persist: Option<PersistState>,
    /// Per-dataset geometry cache for the immediate O(1) in-place append
    /// ([`append_inplace_gathered`](Self::append_inplace_gathered)), keyed by the dataset's resolved
    /// **object-header address** (not its path, so two hard links to one dataset
    /// share one entry). Populated on the first append to a dataset and maintained
    /// across appends; cleared wholesale at the entry of every non-trivial
    /// [`commit`](Self::commit), since a commit can relocate a cached header or
    /// free the region it points into (see `commit`).
    located: HashMap<u64, LocatedState>,
    /// Global heap collections **this session placed** for a variable-length
    /// value overwrite, by the overwritten dataset's path, as absolute
    /// `(address, length)` extents.
    ///
    /// This is what lets the *next* overwrite of the same dataset reclaim the
    /// strings the previous one left, instead of growing the file by its whole
    /// payload on every commit forever (issue #321). Nothing else in the editor
    /// reclaims a collection, and the reason is sound: a collection can be
    /// shared between objects, and **nothing in the format proves otherwise**.
    /// The per-object reference count does not: an in-file
    /// [`copy`](Self::copy) of a variable-length dataset re-emits its
    /// element references verbatim, so two datasets name one collection with
    /// every object's count still 1 — and files with that shape are already in
    /// the wild.
    ///
    /// Provenance is therefore the whole proof: these collections were placed by
    /// this session, for this dataset, and were named by nothing else at the
    /// moment they were written. `invalidate_heap_provenance` is what keeps that
    /// true afterwards — every entry is dropped as soon as this session does
    /// anything that could name one of them a second time.
    ///
    /// The collections a dataset held when the session *opened* are never in
    /// here and are never reclaimed: their provenance is whatever wrote the
    /// file. `repack` is what recovers those.
    vl_overwrite_heaps: HashMap<ObjectPathBuf, Vec<(u64, u64)>>,
    /// Heap collections superseded by a value overwrite this commit is applying,
    /// freed once the commit's superblock repoint has landed — never before, so
    /// a mid-commit crash leaves the prior root reaching bytes that are still
    /// there. Drained into the commit's `to_free` list; cleared at commit entry,
    /// so an attempt that fails partway frees nothing on the next one.
    superseded_heaps: Vec<(u64, u64)>,
    /// The bytes each same-length in-place overwrite of the commit in flight
    /// wrote over, so a commit that fails before its linearization point can put
    /// the dataset's values back (issue #344).
    ///
    /// Everything else a commit places is written where nothing reaches it until
    /// the superblock is repointed, so an attempt that stops short of the
    /// repoint is invisible. A same-length value overwrite is the exception:
    /// [`WritePlan::InPlace`] writes straight over the dataset's existing data
    /// block, which the *current* root already reaches, so there is no repoint
    /// to withhold and the new values are live the moment they land. This
    /// journal is how they are withheld anyway.
    ///
    /// Written only by
    /// [`write_inplace_journaled`](Self::write_inplace_journaled) and drained
    /// only by [`undo_inplace_writes`](Self::undo_inplace_writes). Every
    /// [`commit`](Self::commit) that returns leaves it empty, and one that
    /// unwinds does not, which is why the next commit clears it on entry rather
    /// than trusting it.
    ///
    /// **It costs a full copy of every value being replaced, read back off the
    /// file first.** Measured on the bounded backing, whose contract is not to
    /// hold the file in memory: a same-length overwrite of a 4 MB dataset made
    /// its commit read 4,000,197 bytes where the same commit without the
    /// journal read 197, and held the 4 MB until the commit ended. That is the
    /// price of the guarantee and it is charged on the success path too, since
    /// whether the bytes will be needed is not known until the attempt ends.
    /// `a_bounded_commit_reads_the_value_it_is_replacing` states the rule so
    /// the number is in the repository rather than a surprise.
    ///
    /// The alternative — planning a same-length overwrite as a relocating write,
    /// as a variable-length one already is — would cost nothing here and was
    /// rejected for a harder reason than cost: a relocating write is refused for
    /// a dataset with more than one hard link
    /// ([`count_incoming_hard_links`](Self::count_incoming_hard_links)), so it
    /// would start refusing overwrites this engine accepts today.
    inplace_undo: Vec<(u64, Vec<u8>)>,
    /// True when this engine was opened for SWMR writing
    /// ([`open_swmr_writer`](Self::open_swmr_writer)): the append engine then
    /// enforces the SWMR subset (unfiltered, chunk-aligned) so a concurrent
    /// reader never observes a torn view. `false` for an ordinary edit session.
    swmr_mode: bool,
    /// Paged-file state (`H5F_FSPACE_STRATEGY_PAGE`), read from the superblock
    /// extension at `open` regardless of the persist flag; `None` for the common
    /// non-paged file. When `Some`, [`commit`](Self::commit) takes a page-aware
    /// tail that keeps pages homogeneous and rewrites the per-page-type managers
    /// (issue #198). A paged file that does not *persist* its free space is still
    /// refused: see [`PagedEdit`].
    paged: Option<PagedEdit>,
    /// Set by the first [`commit`](Self::commit) that does any work. A commit can
    /// relocate an object header, and nothing on disk distinguishes a relocated
    /// header from the intact bytes it vacated — the old header still parses, and
    /// its data-layout message still points at the live chunk index. An
    /// [`AppendTarget::Header`] captured before that commit would therefore append
    /// successfully *into the dead header*, growing its dataspace while the live
    /// dataset stayed put, and report `Ok`. A path is re-resolved on every append
    /// and so survives a commit; a raw address does not, so it is refused once one
    /// has run.
    committed: bool,
    /// Object-header address for each path an in-place append has resolved in
    /// this session. A single `Dataset::append` asks for the target's geometry
    /// and then appends to it, and a loop of appends repeats that; without this
    /// the path would be walked from the root on every one of those steps.
    ///
    /// An in-place append never moves an object header — that is why
    /// [`located`](Self::located) can be keyed by address and survive appends —
    /// so only a commit can stale an entry, and it clears both together.
    resolved: HashMap<ObjectPathBuf, u64>,
    /// Whether this session splits a large in-place append into batches, trading
    /// whole-call crash atomicity for a peak memory that does not scale with the
    /// call. Set by [`open_rw_with_strategy`](Self::open_rw_with_strategy); see
    /// [`batch_elems`](Self::batch_elems).
    batched_appends: bool,
    /// Whether this session reads through a handle rather than a whole-file
    /// mirror. Set by [`open_rw_with_strategy`](Self::open_rw_with_strategy), and reported by
    /// [`File::edit_backing`](crate::File::edit_backing) so a caller who
    /// asked for [`MemoryStrategy::Auto`] can tell which one it got. Distinct
    /// from [`batched_appends`](Self::batched_appends), which is a crash-atomicity
    /// trade the bounded engine happens to make, not a statement about memory.
    bounded: bool,
    /// The on-disk format this session may write, resolved from the fapl's
    /// [`FileAccessProperties::with_libver_bounds`]. `None` — the default —
    /// means unconstrained: the session adds whatever the content needs, which
    /// is what lets a file the C library wrote under its own bounds be edited at
    /// all.
    ///
    /// Set below [`LibVer::V110`] it refuses content the older format cannot
    /// carry, the way the whole-file writer does. The file's *own* superblock
    /// version cannot stand in for this: a version 2 superblock says the file is
    /// 1.8-readable today, not that the caller wants it to stay that way, and
    /// deriving the ceiling from it would refuse the C-library-file edits of
    /// issue #101.
    ///
    /// [`FileAccessProperties::with_libver_bounds`]: crate::FileAccessProperties::with_libver_bounds
    libver_ceiling: Option<LibVer>,
    /// The file length when the on-disk free-space managers were last written,
    /// for a file that persists them. Every immediate in-place append grows the
    /// file past those managers and leaves them mid-file, so a session that ends
    /// with `image.len() != fsm_len` owes a rewrite; that is what
    /// [`finalize_persist`](Self::finalize_persist) settles at close. Meaningless
    /// (and untouched) when `persist` is `None`.
    fsm_len: u64,
    /// Whether the commit currently running has *asked* the disk to repoint the
    /// superblock at its new root. Set by whichever tail performs that write,
    /// immediately **before** issuing it, and read only by
    /// [`commit`](Self::commit) to decide whether a failure may roll anything
    /// back — the free lists (see [`FreeSnapshot`]), the staged set, and the
    /// values in [`inplace_undo`](Self::inplace_undo).
    ///
    /// Set before rather than after for the reason the whole rollback exists: a
    /// write that alters the file and then reports failure is a real device
    /// behaviour, so a write call returning an error does not mean its bytes
    /// are not on the disk. Once the publish has been issued, whether this
    /// commit is live is unknowable from here, and every rollback is unsound —
    /// undoing the values would tear a commit that did land, and re-offering
    /// the regions would hand out space that commit is using. Doing nothing is
    /// the only answer that is never actively wrong, and the error the caller
    /// gets is the write failure that says so (issue #344).
    ///
    /// It is a property of one publish, not of the session, so every publisher —
    /// `commit`, and the manager rewrite an append's draw makes — clears it on
    /// entry rather than trusting the previous run to have left it false.
    publish_attempted: bool,
    /// Whether a [`stage_atomically`](Self::stage_atomically) batch is open, so
    /// [`refuse_mid_batch`](Self::refuse_mid_batch) can keep
    /// [`StagedEdits::rewind`]'s truncation an exact undo.
    staging_batch: bool,
    /// How many times this session's staged set has been *consumed* by a commit
    /// rather than handed back to it.
    ///
    /// It is what tells a handle born onto a staged creation
    /// ([`staged_generation`](Self::staged_generation)) which of two things has
    /// happened since: the object it names was published, so it addresses the
    /// file now — or the staging was withdrawn under it, and it names nothing.
    /// Both leave the staged set without an entry at its path, and reading them
    /// the same way is what let such a handle silently retarget onto whatever
    /// the file held at that path.
    ///
    /// Advanced by [`commit`](Self::commit) alone, on exactly the path where the
    /// set does not go back: a commit that succeeds, and one that fails past its
    /// publish (whose objects are in the file). A commit refused before its
    /// first write restores the set and leaves this alone, so the handles it
    /// concerns stay pending and the batch can be committed again.
    staged_generation: u64,
    /// Superblock status flags this session raised on the file and holds for its
    /// lifetime — the SWMR pair, or the page buffer's crash mark — or `0` when it
    /// holds none.
    ///
    /// A clean commit publishes *this* rather than a literal zero. The zero was
    /// there to scrub a flag the file arrived carrying (one a crashed writer left
    /// behind), and it still does: this is `0` unless the session raised
    /// something itself. What it must not do is scrub a mark that is still
    /// standing for a reason, which a page-buffered session's is for every
    /// commit it makes (issue #308).
    ///
    /// Maintained only by [`set_consistency_flags`](Self::set_consistency_flags),
    /// which is the one place this crate writes that byte, so the field cannot
    /// drift from what is on the disk.
    held_status_flags: u32,
    /// Who owns this session's `fsync` cadence, from the fapl's
    /// [`FileAccessProperties::with_sync_policy`]. Consulted by
    /// [`barrier`](Self::barrier) and [`barrier_data`](Self::barrier_data) —
    /// every durability point the write paths define — and by nothing else, so
    /// [`force_sync`](Self::force_sync) can serve an explicit
    /// [`File::sync`](crate::File::sync) whatever it says.
    ///
    /// [`FileAccessProperties::with_sync_policy`]: crate::FileAccessProperties::with_sync_policy
    sync_policy: SyncPolicy,
}

/// The free lists as they stood before a commit's apply loop drew from them.
///
/// A commit allocates as it goes — a free region it hands to an object header or
/// a chunk blob leaves the list immediately, so no two objects in the same commit
/// can be handed the same region. If the commit then fails, those regions are as
/// dead as they were before it started: nothing the commit wrote is reachable,
/// because the superblock still names the old root — and the one write that does
/// not wait for the repoint, a same-length value overwrite, is one
/// [`WriteEngine::undo_inplace_writes`] has already tried to put back by the
/// time this runs (issue #344). Restoring this snapshot gives them back instead
/// of leaking them for the rest of the session.
///
/// The restore is sound only *before* the repoint. After it, the objects written
/// into those regions are live, and handing the same addresses out again would
/// overwrite the tree the commit just published — which is why
/// [`WriteEngine::publish_attempted`] gates it.
struct FreeSnapshot {
    free: FreeList,
    /// The append reserve, which the persisting commit tail drains back into the
    /// lists above before it rewrites the managers. A commit that then fails
    /// before publishing leaves those managers as they were — still not listing
    /// the reserve — so restoring it is what keeps the session's picture of the
    /// disk exact (issue #387).
    reserved: FreeList,
    paged: Option<(FreeList, FreeList)>,
    /// The heap-collection provenance, rolled back with the free lists.
    ///
    /// [`resolve_overwrite_bytes`](WriteEngine::resolve_overwrite_bytes) records
    /// a placement in the *apply* phase, so a commit that fails after that and
    /// before its repoint hands the space back to the free list while leaving a
    /// record naming it. The next overwrite of that path would then free a
    /// region already handed out — the one piece of engine state naming file
    /// addresses that this rollback used to miss (issue #321).
    vl_overwrite_heaps: HashMap<ObjectPathBuf, Vec<(u64, u64)>>,
}

/// How much memory a read-write open may use to hold the file being edited.
///
/// The two read-write backends differ in memory, not in what they can express: a
/// *bounded* session reads through a handle and holds only what a commit is
/// building, while a *mirrored* session materializes the whole file in memory.
/// Bounded is the better default when it applies, but it cannot yet edit every
/// file — a pre-v2 (non-latest-format) superblock or a userblock still needs the
/// mirror.
///
/// This is what a caller says about that trade-off, on
/// [`FileAccessProperties::with_memory_strategy`](crate::FileAccessProperties::with_memory_strategy).
/// Leaving it unset lets the entry point decide: [`File::open_rw`](crate::File::open_rw)
/// prefers the bounded engine and falls back to the mirror ([`Auto`](Self::Auto)).
/// Stating [`Bounded`](Self::Bounded) refuses instead of falling back.
///
/// This is a *request*, so it is deliberately not the type a file answers with:
/// [`File::edit_backing`](crate::File::edit_backing) returns an [`EditBacking`],
/// which cannot express [`Auto`](Self::Auto).
///
/// Sealed: unlike [`FileLocking`] or [`FileSpaceStrategy`], whose variant sets
/// mirror a closed C-library enum, this is a policy this crate invented, so a
/// fourth strategy must not be a breaking change.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[non_exhaustive]
pub enum MemoryStrategy {
    /// Never build a whole-file mirror. A file the bounded engine cannot edit is
    /// refused at open with [`Error::EditUnsupported`], before anything is
    /// staged.
    Bounded,
    /// Prefer the bounded engine, but fall back to the whole-file mirror for a
    /// file it cannot edit, rather than refusing. Memory then scales with the
    /// file, which is the cost of the file opening at all. What
    /// [`File::open_rw`](crate::File::open_rw) uses when nothing is asked for.
    ///
    /// Two files reach that fallback, and no others: one with a pre-v2
    /// (non-latest-format) superblock, and one with a userblock (a non-zero base
    /// address). Every other file a read-write open accepts is edited bounded,
    /// whatever its [`FileSpaceStrategy`], whether or
    /// not it persists its free space, and however large it is. A paged file with
    /// no persisted free space is refused by *both* backings rather than
    /// mirrored, so it is not a third case. The list being exhaustive is what
    /// makes "prefer bounded" a guarantee to budget against: wherever
    /// [`Bounded`](Self::Bounded) opens a file, `Auto` gives the same backing.
    Auto,
    /// Always build the whole-file mirror, whatever the file looks like. What
    /// [`File::open_rw`](crate::File::open_rw) did before it learned to dispatch.
    Mirrored,
}

/// Which of the two read-write backends a file's editing session is actually
/// using, from [`File::edit_backing`](crate::File::edit_backing).
///
/// Deliberately a different type from [`MemoryStrategy`]: that one is what a
/// caller *asks* for and includes [`Auto`](MemoryStrategy::Auto), which is a
/// preference between these two rather than a third thing a session can be. A
/// single shared type would make `file.backing() == Auto` a comparison that
/// compiles and is false forever.
///
/// The two also evolve at different rates. A future `MemoryStrategy` may name a
/// new *policy* — a byte budget, a size threshold — without the set of backends
/// changing at all. Sealed for the rarer case that a third backend does appear.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[non_exhaustive]
pub enum EditBacking {
    /// Reads through a file handle, holding only what a commit is building.
    /// Memory does not scale with the file.
    Bounded,
    /// Holds the whole file in memory for the life of the session.
    Mirrored,
}

impl From<EditBacking> for MemoryStrategy {
    /// Turns an outcome back into the request that pins it, so a caller can
    /// reopen a file onto the backing it got the first time:
    /// `with_memory_strategy(file.edit_backing().unwrap().into())`.
    fn from(backing: EditBacking) -> Self {
        match backing {
            EditBacking::Bounded => Self::Bounded,
            EditBacking::Mirrored => Self::Mirrored,
        }
    }
}

/// When a read-write session forces its writes to durable storage — who owns the
/// `fsync` cadence, this crate or the application.
///
/// This is *not* about whether a write reaches the file. Every commit and every
/// append has gone to the operating system by the time it returns, whatever this
/// policy says, so a committed edit is visible to any other process on the same
/// machine and survives this process crashing. What it governs is the `fsync` on
/// top of that, which is what makes those bytes survive the *machine* losing
/// power.
///
/// A session does gather the many small writes *inside* one such operation and
/// issue them a page at a time (issue #288), and an explicit
/// [page buffer](crate::FileAccessProperties::with_page_buffer_size) extends that
/// across operations — trading exactly the guarantee in the paragraph above,
/// which is why it is opt-in and this is not.
///
/// The reference C library does not `fsync` on its normal path either: the
/// default `sec2` driver installs no flush callback at all, so `H5Fflush` drains
/// libhdf5's own caches with `write` and stops there, leaving power-loss
/// durability to the application. [`OnClose`](Self::OnClose) is that behavior;
/// [`Always`](Self::Always), the default here, is the stronger one.
///
/// The whole-file writer ([`FileBuilder`](crate::FileBuilder)) and
/// [`repack`](crate::repack()) are outside this: they never `fsync` under either
/// policy, since each writes a file and hands it over rather than holding an
/// editing session.
///
/// Sealed: a future policy (syncing on a timer, or once per N bytes) must not be
/// a breaking change.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
#[non_exhaustive]
pub enum SyncPolicy {
    /// Force durability at every point the write paths define one: after each
    /// immediate [`Dataset::append`](crate::Dataset::append) batch, at each
    /// ordering barrier inside a [`File::commit`](crate::File::commit), and when
    /// the file is closed or dropped.
    ///
    /// This is what makes an append crash-atomic and a commit all-or-nothing
    /// against power loss: the barriers order the appended data before the
    /// superblock repoint that publishes it, so an interrupted write leaves the
    /// previous state rather than a torn one.
    #[default]
    Always,
    /// No `fsync` during the session; one when the file is finished. Nothing an
    /// append or a commit does is forced to durable storage — writes still reach
    /// the operating system immediately, so another process on the same machine
    /// sees them and a crash of *this* process loses nothing — and
    /// [`File::close`](crate::File::close), or dropping the last handle, issues
    /// a single barrier at the end. [`File::sync`](crate::File::sync) adds a
    /// checkpoint wherever the application wants one.
    ///
    /// One setting suspends the "reach the operating system immediately" half,
    /// and it is the one that requires this policy:
    /// [`with_page_buffer_size`](crate::FileAccessProperties::with_page_buffer_size)
    /// holds dirty pages in this process until the budget is spent, a
    /// [`File::sync`](crate::File::sync) is issued, or the session closes. Off by
    /// default, and it marks the file on disk for its lifetime so a session that
    /// dies holding pages leaves one every reader refuses.
    ///
    /// The terminal barrier is not an exception grafted onto "never sync": it is
    /// the point past which the application *cannot* act. `close` and `drop`
    /// both write — they apply staged edits, re-home the free-space managers of
    /// a file that persists them, and clear a SWMR writer's flag — and they
    /// destroy the handle that would have ordered those writes. A policy that
    /// skipped the barrier there would not hand the caller a cadence; it would
    /// take one away. The cost is one `fsync` per session against the five per
    /// append batch and two or three per commit that this policy removes.
    ///
    /// Three things are given up in exchange. **Power-loss ordering** during the
    /// session: with the barriers gone, a machine that loses power mid-commit
    /// can have the superblock repoint on disk without the data it points at.
    /// **Deferred write errors**: on a filesystem that allocates late, a write
    /// that will fail at writeback still returns success, and `fsync` is where
    /// the `ENOSPC`/`EIO` surfaces, so a commit the filesystem cannot complete
    /// returns `Ok` until the terminal barrier reports it. **Cross-host
    /// visibility**: "another process sees it" is the page cache, so under NFS's
    /// close-to-open semantics a reader on another host — a SWMR reader
    /// included — may not see writes a client is holding.
    OnClose,
}

/// Why the bounded engine cannot edit a file, when the whole-file mirror can.
///
/// Kept separate from the refusals that apply to *both* engines: a fallback is
/// only ever worth taking for a limitation the mirror does not share. A paged
/// file with no persisted free-space managers, for instance, is refused by the
/// staged commit as well, so mirroring it would trade a clear error at open for
/// the same error later with work already staged.
fn bounded_only_limitation(session: &WriteEngine) -> Option<&'static str> {
    if session.superblock.version < 2 {
        return Some(
            "bounded read-write access requires a latest-format file (v2/v3 superblock); \
             leave MemoryStrategy unset, or pass MemoryStrategy::Auto, to fall back to \
             the whole-file mirror here",
        );
    }
    if !session.superblock.base_address.is_zero() {
        return Some(
            "bounded read-write access does not support a file with a userblock \
             (non-zero base address); leave MemoryStrategy unset, or pass \
             MemoryStrategy::Auto, to fall back to the whole-file mirror here",
        );
    }
    None
}

/// Whether a file built from `create` could not then be opened read-write under
/// `access`, and why.
///
/// [`File::create_with_options`](crate::File::create_with_options) writes a file
/// and hands back an open read-write handle, so a creation/access pair that
/// cannot survive that second half must be caught *before* the write — otherwise
/// the call leaves a file on disk and returns `Err`, which reads like a failed
/// create but is not one.
///
/// This mirrors the open-time refusals above and must be kept in step with them:
/// [`bounded_only_limitation`] for the userblock, and the shared paged check in
/// [`open_rw_with_strategy`](WriteEngine::open_rw_with_strategy) for a paged file
/// with no persisted free space. Both are stated here in terms of the properties
/// that *cause* them, because the open-time wording tells the caller to recreate
/// the file — advice that is circular when the caller is creating it.
pub(crate) fn create_would_refuse_reopen(
    create: &FileCreateProperties,
    access: &FileAccessProperties,
) -> Option<&'static str> {
    if let Some((FileSpaceStrategy::Page, false, _)) = create.file_space_strategy() {
        return Some(
            "a paged file (FileSpaceStrategy::Page) with persist = false cannot be reopened \
             read-write, so creating one this way would write the file and then fail to open \
             it; pass persist = true to with_file_space_strategy, or build the file with \
             FileBuilder if it is only ever going to be read",
        );
    }
    if create.userblock() != 0 && access.memory_strategy() == Some(MemoryStrategy::Bounded) {
        return Some(
            "a userblock cannot be combined with MemoryStrategy::Bounded: the bounded engine \
             cannot edit a file with a non-zero base address, so creating one this way would \
             write the file and then refuse to open it; drop the userblock, or leave \
             MemoryStrategy unset to mirror this file",
        );
    }
    // A userblock on a *paged* file reaches the same open-time refusal as
    // `persist = false` above, by a different route: persisted free space is
    // declined for a non-zero base address, which leaves a paged file with no
    // seeded managers however its creation properties were written.
    if create.userblock() != 0
        && matches!(
            create.file_space_strategy(),
            Some((FileSpaceStrategy::Page, _, _))
        )
    {
        return Some(
            "a userblock cannot be combined with FileSpaceStrategy::Page: free space is not \
             persisted for a file with a non-zero base address, and a paged file without it \
             cannot be opened read-write — so creating one this way would write the file and \
             then fail to open it; drop the userblock, or choose another file-space strategy",
        );
    }
    // The page-buffer refusals `set_page_buffer_size` makes at open, restated as
    // the creation pair that would walk into them. They are phrased as something
    // to *add* to the properties in hand rather than as something to recreate the
    // file with, since the file does not exist yet (issues #288 and #308).
    if access.page_buffer_size() != 0 {
        if access.sync_policy() == SyncPolicy::Always {
            return Some(
                "a page buffer does nothing under the default SyncPolicy::Always, whose every \
                 barrier is an fsync that flushes it, so creating one this way would write the \
                 file and then fail to open it; add \
                 with_sync_policy(SyncPolicy::OnClose) to the access properties, or drop the \
                 page buffer",
            );
        }
        // The version-3 counterpart. It used to need none: a page buffer required
        // a paged file, and `FileSpaceStrategy::Page` below the 1.10 format is
        // refused by the builder itself before anything is written, so that pair
        // never reached an open. Issue #357 lifted the paged requirement, and an
        // *unpaged* file at `LibVer::V18` is an ordinary, buildable file with a
        // version-2 superblock — one whose status-flags byte no library reads
        // back, so the mark could not be raised on it. Without this the pair
        // would write the file and fail the open it promised.
        //
        // Unsatisfiable bounds are left to the builder, which reports which
        // format it can write; this is not the place to restate that.
        if matches!(
            crate::libver::LibVer::resolve_writable(create.libver_bounds()),
            Ok(v) if v < crate::libver::LibVer::V110
        ) {
            return Some(
                "a page buffer marks the file in a version-3 superblock, and these bounds \
                 write the 1.8 format, so creating one this way would write the file and then \
                 fail to open it; raise with_libver_bounds to admit LibVer::V110, or drop the \
                 page buffer",
            );
        }
        // An unpaged file has no page size of its own and merges within
        // `DEFAULT_GATHER_PAGE`, which is the same figure the format defaults
        // `H5Pset_file_space_page_size` to. This arm decides on its own now that
        // a budget below the session's own gather budget is honored rather than
        // refused (issue #391): a 2 KiB buffer on an unpaged create clears every
        // other check and is caught only here. It is written out rather than
        // folded into the paged arm's `unwrap_or` because the two constants are
        // free to move apart, and because a reader comparing this function
        // against `set_page_buffer_size` should find the same rule in both.
        let page_size = match create.file_space_strategy() {
            Some((FileSpaceStrategy::Page, _, _)) => create
                .file_space_page_size()
                .unwrap_or(crate::file_space_info::DEFAULT_PAGE_SIZE),
            _ => DEFAULT_GATHER_PAGE,
        };
        if (access.page_buffer_size() as u64) < page_size {
            return Some(
                "a page buffer smaller than the file's file-space page size would be refused \
                 at open — so creating one this way would write the file and then fail to \
                 open it; raise with_page_buffer_size",
            );
        }
    }
    None
}

/// What a commit knows about a region it is vacating, which is what decides
/// where the region may be recorded.
///
/// A flat file ignores the distinction — every vacated region is simply free —
/// and a paged one cannot, because a byte handed out for the wrong page type
/// mixes the page it lands in.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum FreeClass {
    /// The region sits in pages of this type, so it becomes free space of that
    /// type and may serve the next allocation of it.
    Page(PageType),
    /// The region provably holds nothing live, but which page type surrounds it
    /// is not provable — a chunk index this crate cannot place
    /// ([`WriteEngine::index_is_provably_raw`]) is the case that arises.
    ///
    /// It may not be handed out as either type, so it is recorded as *dead*
    /// rather than free: bytes that are no longer in use and are not yet
    /// reusable. Once every other byte of the page around it is free or dead
    /// too, the page holds nothing of either type and
    /// [`PagedEdit::promote_whole_free_pages`] turns the whole page into free
    /// space, which is the same rule that lets one page type claim a wholly free
    /// page from the other ([`PagedEdit::alloc_typed`]).
    Dead,
}

impl From<PageType> for FreeClass {
    fn from(ty: PageType) -> Self {
        FreeClass::Page(ty)
    }
}

/// Paged-file bookkeeping for the whole-file editor (issue #198, step 1).
///
/// A paged file never mixes metadata and raw data within one page, so this tracks
/// free space per page *type* — the one distinction that governs where a byte may
/// be placed — and keeps the commit's appends homogeneous by padding a tail page
/// whenever the page type changes.
///
/// One list per type, not one per on-disk manager. A paged file records its free
/// space in three managers (metadata-small, raw-small, and a generic-large one for
/// whole free pages), but for space this session can place, that split is a size
/// classification of the same raw or metadata space, and [`plan_paged_managers`]
/// recomputes it from scratch at every commit. Tracking it here as well would only
/// stop a freed chunk-data run and the freed index that abuts it from coalescing —
/// leaving two neighboring holes neither of which fits the dataset that just
/// vacated both (issue #261).
///
/// The free lists are seeded only when the file *persists* its free space. A paged
/// non-persisting file has no on-disk record of which pages hold metadata and
/// which hold raw data, so there is nothing to seed and no way to stay segregated;
/// [`commit`](WriteEngine::commit) rejects it outright, exactly as the bounded
/// backend does.
struct PagedEdit {
    page_size: u64,
    /// Free space inside metadata pages, plus whole free pages this session last
    /// saw as metadata. A page holding nothing belongs to no type, so
    /// [`alloc_typed`](Self::alloc_typed) lets raw data claim one from here.
    meta: FreeList,
    /// Free space inside raw-data pages, plus whole free pages this session last
    /// saw as raw — including every one seeded from the generic-large manager,
    /// which records no type. Metadata claims from here on the same terms.
    raw: FreeList,
    /// Free space this session may record but must never hand out, because the
    /// page it sits in is of unknown type. Seeded at open — nothing this engine
    /// frees is ever of unknown type, so it only ever shrinks, and only where a
    /// commit released the end of the file over it (issue #418) — and written
    /// back to the generic-large manager verbatim, so the space stays available
    /// to the writer that recorded it.
    ///
    /// It exists because the reference C library's generic-large manager is
    /// exactly that: `H5F_MEM_PAGE_GENERIC` is *"large-sized generic: meta and
    /// raw"*, and under paged aggregation on a contiguous-address driver every
    /// allocation of a page or more lands there whatever its type. Its
    /// page-alignment tail (`H5MF__alloc_pagefs`) is then recorded as a free
    /// section in the same manager — a sub-page fragment sitting in a page whose
    /// earlier bytes are live. Handing such a fragment to raw data would put raw
    /// bytes in a metadata page, so a section that is not a whole run of aligned
    /// pages is kept here rather than guessed at.
    unclassified: FreeList,
    /// Space this session has vacated whose page type it could not prove
    /// ([`FreeClass::Dead`]): no longer in use, and not reusable as either type.
    ///
    /// It is held rather than discarded so that the page around it can still be
    /// reclaimed: a page every one of whose bytes is free or dead holds nothing
    /// of either type, and [`promote_whole_free_pages`](Self::promote_whole_free_pages)
    /// moves it into the typed lists whole. Without this list those bytes would
    /// simply be lost, and the page they sit in could never be shown to be empty
    /// — which is what made a paged file leak a chunk index per deleted empty
    /// resizable dataset (issue #388).
    ///
    /// Unlike the free lists this one is not written to disk: a dead sub-page
    /// fragment is not free space by the file's account, and recording it in the
    /// only manager that takes untyped space would offer it to the reference
    /// library for either type. A fragment that never completes a page is
    /// therefore forgotten at close, which wastes its bytes and nothing more.
    dead: FreeList,
    /// Page type of the current tail page. `None` until this session's first
    /// typed append; the file is page-aligned at open, so the first append never
    /// needs to pad regardless of this.
    last: Option<PageType>,
    /// Free tails left by padding a metadata page before a raw append.
    meta_pad: Vec<(u64, u64)>,
    /// Free tails left by padding a raw page before a metadata append.
    raw_pad: Vec<(u64, u64)>,
}

impl PagedEdit {
    /// Ensure the next allocation on `image` begins in a page holding page type
    /// `ty`: when the tail page holds the *other* type and is only partially
    /// filled, pad it to a page boundary and record the padding as free space of
    /// the outgoing type.
    ///
    /// This is the whole of the paged-append rule, and it lives here so the two
    /// places that grow a paged file — the staged commit through
    /// [`WriteEngine::begin_page`](WriteEngine::begin_page), and the shared
    /// Extensible-Array append engine through [`EditStore`] — cannot drift. They
    /// used to keep separate copies of this state, one per engine, which is what
    /// made an in-place append to a paged file unsafe from the whole-file editor
    /// (issue #198).
    ///
    /// Call it **before** reading the image's end-of-file to compute an address
    /// that will be embedded in the bytes being built: several callers build
    /// content whose interior addresses assume it lands at the current
    /// end-of-file, and padding inserted after that read would shift the landing
    /// address out from under them.
    fn begin(&mut self, image: &mut dyn FileImage, ty: PageType) -> Result<(), Error> {
        let len = image.len();
        if len % self.page_size != 0 {
            let pad_len = self.page_size - len % self.page_size;
            // `prev` is the outgoing page type to record the padding under, or
            // `None` for a crash-recovery pad whose tail-page type is unknown.
            let pad = match self.last {
                // Normal case: the tail page holds a known type; pad only on a
                // type switch, recording the tail as free of the outgoing type.
                Some(prev) if prev != ty => Some(Some(prev)),
                Some(_) => None, // same type: keep packing the tail page
                // A previous session grew this paged file and was killed before
                // its tail was page-aligned, so the file opened non-page-aligned
                // with no known tail type. Pad it up (extending whatever the tail
                // page holds, so the page stays homogeneous) and leave the padding
                // untracked, since recording it under the wrong page type could
                // let a reader reuse it and mix the page.
                None => Some(None),
            };
            if let Some(prev) = pad {
                let pad_at = len;
                image.append(&vec![0u8; pad_len.to_usize()?])?;
                match prev {
                    Some(PageType::Meta) => self.meta_pad.push((pad_at, pad_len)),
                    Some(PageType::Raw) => self.raw_pad.push((pad_at, pad_len)),
                    None => {} // crash-recovery pad: untracked (tail type unknown)
                }
            }
        }
        self.last = Some(ty);
        Ok(())
    }

    fn new(page_size: u64) -> Self {
        PagedEdit {
            page_size,
            meta: FreeList::new(),
            raw: FreeList::new(),
            unclassified: FreeList::new(),
            dead: FreeList::new(),
            last: None,
            meta_pad: Vec::new(),
            raw_pad: Vec::new(),
        }
    }

    /// Which list a persisted section from File Space Info `slot` belongs in.
    ///
    /// Slot 0 (SUPER) is small metadata and slot 2 (DRAW) is small raw: both name
    /// a page type outright. Slot 6 is the generic-large manager, which holds
    /// space of *either* type (see [`unclassified`](Self::unclassified)), so a
    /// section from it is only safe to reuse when it covers whole aligned pages —
    /// and such a section belongs to no type at all, since nothing lives in those
    /// pages to be mixed with. It is filed under raw and reached from either side
    /// through [`alloc_typed`](Self::alloc_typed), rather than kept in a third
    /// list, so that a run of free pages still coalesces with the sub-page
    /// fragment beside it. Everything else, including the per-type large managers
    /// a multi/split driver would populate, is left unclassified rather than
    /// guessed at.
    fn slot_list(slot: usize, addr: u64, size: u64, page_size: u64) -> Option<PageType> {
        match slot {
            0 => Some(PageType::Meta),
            2 => Some(PageType::Raw),
            6 if addr % page_size == 0 && size % page_size == 0 => Some(PageType::Raw),
            _ => None,
        }
    }

    /// Record `(addr, size)` in the list its class names: the free space of one
    /// page type, or the dead list for space whose page type is unproven.
    /// [`plan_paged_managers`] splits and classes the free lists into the on-disk
    /// managers at serialization time, so this only has to route.
    ///
    /// Takes the lists rather than `&mut self` because a commit routes into
    /// *copies* of them — nothing is free until the superblock repoint — and the
    /// rule must not be restated at that call site.
    fn route_free(
        meta: &mut FreeList,
        raw: &mut FreeList,
        dead: &mut FreeList,
        addr: u64,
        size: u64,
        class: FreeClass,
    ) {
        match class {
            FreeClass::Page(PageType::Meta) => meta.free(addr, size),
            FreeClass::Page(PageType::Raw) => raw.free(addr, size),
            FreeClass::Dead => dead.free(addr, size),
        }
    }

    /// Move every page that is *wholly* free or dead out of the three lists and
    /// into the raw list as one whole free page.
    ///
    /// A paged file may only place a byte in a page of its own type, which is why
    /// free space is tracked per type at all. A page holding nothing belongs to no
    /// type, so it may be opened for whichever type asks — the rule
    /// [`alloc_typed`](Self::alloc_typed) already applies to a whole page inside
    /// the other type's list, and [`slot_list`](Self::slot_list) to a whole page
    /// read back from the untyped manager. This applies the same rule one level
    /// up, to a page whose emptiness only the three lists *together* establish:
    /// part free metadata, part free raw data, part space vacated by an object
    /// whose page type could not be proven ([`FreeClass::Dead`]).
    ///
    /// That last part is what makes it necessary rather than a tidy-up. Dead
    /// space is never handed out on its own, so without this step a deleted chunk
    /// index this crate cannot place is lost for good, and a paged file under
    /// delete-and-recreate churn spends a page every few cycles that it can never
    /// take back (issue #388). Promoting the page is not a guess about what once
    /// sat in it: nothing live is left there to be mixed with.
    ///
    /// Filed under raw, which is where a whole free page belongs by the same
    /// convention [`slot_list`](Self::slot_list) reads them back under, and
    /// reachable from either side through `alloc_typed`.
    fn promote_whole_free_pages(
        meta: &mut FreeList,
        raw: &mut FreeList,
        dead: &mut FreeList,
        page_size: u64,
    ) {
        // The lists are individually coalesced and pairwise disjoint (a byte is
        // vacated once), so their union is a plain merge: sort by address and join
        // runs that touch. A page is promotable exactly when it lies inside one of
        // those runs, which is what a page split across two of the lists needs.
        let mut all = meta.sections();
        all.extend(raw.sections());
        all.extend(dead.sections());
        all.sort_unstable_by_key(|&(addr, _)| addr);
        let mut runs: Vec<(u64, u64)> = Vec::with_capacity(all.len());
        for (addr, len) in all {
            match runs.last_mut() {
                Some(run) if run.0 + run.1 >= addr => {
                    let end = (run.0 + run.1).max(addr + len);
                    run.1 = end - run.0;
                }
                _ => runs.push((addr, len)),
            }
        }
        for (addr, len) in runs {
            // The whole pages inside the run: its aligned interior, exactly as
            // `FreeList::alloc_whole_units` takes one. The partial edges sit in
            // pages whose other bytes may be live, so they keep whatever they are.
            let first = addr.next_multiple_of(page_size);
            let last = (addr + len) / page_size * page_size;
            if last <= first {
                continue;
            }
            let span = last - first;
            meta.take_range(first, span);
            raw.take_range(first, span);
            dead.take_range(first, span);
            raw.free(first, span);
        }
    }

    /// Draw `len` bytes of page type `ty` from this file's free space, or `None`
    /// when nothing can serve it.
    ///
    /// Its own list first, which is the only place a *partly* free page can serve
    /// it. Failing that, whole pages inside the other type's free space: a page
    /// with nothing in it belongs to neither type, so opening it for `ty` cannot
    /// mix it. Enough whole pages to cover `len` are claimed, and the remainder
    /// joins `ty`'s list, since those pages now hold `ty`.
    ///
    /// The cross-type claim is what lets space survive a close. A whole free page
    /// is written to the generic-large manager, which records no page type, so
    /// every one of them comes back from disk as raw ([`slot_list`](Self::slot_list));
    /// without this, metadata could never reuse a page again after a reopen, and
    /// each session's commits would append past all of them (issue #286).
    fn alloc_typed(&mut self, len: u64, ty: PageType) -> Option<u64> {
        let (own, other) = match ty {
            PageType::Meta => (&mut self.meta, &mut self.raw),
            PageType::Raw => (&mut self.raw, &mut self.meta),
        };
        if let Some(addr) = own.alloc(len) {
            return Some(addr);
        }
        let span = align_up(len, self.page_size);
        let addr = other.alloc_whole_units(span, self.page_size)?;
        if span > len {
            own.free(addr + len, span - len);
        }
        Some(addr)
    }

    /// The longest contiguous run [`alloc_typed`](Self::alloc_typed) could serve
    /// `ty` from in one call, or `0` when nothing could: the larger of its own
    /// list's longest region and the longest run of whole free pages in the
    /// other's.
    ///
    /// The same two sources in the same terms, so a caller sizing a draw on this
    /// figure gets exactly the run `alloc_typed` then hands it.
    fn largest_typed(&self, ty: PageType) -> u64 {
        let (own, other) = match ty {
            PageType::Meta => (&self.meta, &self.raw),
            PageType::Raw => (&self.raw, &self.meta),
        };
        own.largest().max(other.largest_whole_units(self.page_size))
    }

    /// Every free region this session could still hand out, ascending by address.
    /// Used for space accounting, where the caller wants one total rather than a
    /// per-page-type breakdown.
    ///
    /// [`unclassified`](Self::unclassified) is excluded: it is free space, and
    /// [`File::persisted_free_space`](crate::File::persisted_free_space) reports
    /// it as such, but this session will never place anything in it, which is
    /// what the accounting field it feeds is about.
    fn reusable_sections(&self) -> Vec<(u64, u64)> {
        let mut out = self.meta.sections();
        out.extend(self.raw.sections());
        out.sort_unstable_by_key(|&(addr, _)| addr);
        debug_assert!(
            out.windows(2)
                .all(|w| w[0].0.saturating_add(w[0].1) <= w[1].0),
            "a region is free in both the metadata and the raw list, so the two \
             page-type lists have stopped being disjoint"
        );
        out
    }
}

/// The free space a paged commit is about to write to disk, as
/// [`WriteEngine::paged_post_free`] computes it: the session's lists with this
/// commit's frees folded in, held apart from the session until the repoint makes
/// them true.
struct PagedPostFree {
    meta: FreeList,
    raw: FreeList,
    /// The session's dead space plus this commit's, with every page the three
    /// lists together empty already promoted out of it
    /// ([`PagedEdit::promote_whole_free_pages`]).
    dead: FreeList,
    /// Nothing in a commit adds to or draws from this one.
    unclassified: FreeList,
}

impl PagedPostFree {
    /// Give the run of unreferenced space that reaches `eof` back to the
    /// filesystem: drop it from every list here and return the end-of-allocation
    /// the commit should publish, which the caller truncates the file to
    /// (issue #418). `eof` itself when nothing at the end of the file is free.
    ///
    /// The cut is rounded **up** to a page boundary, so a paged file's
    /// end-of-allocation stays a whole number of pages: a run that begins
    /// mid-page begins in a page whose earlier bytes are live, and only the whole
    /// pages above it can go. That is the same granularity the reference C
    /// library shrinks a paged file at — its `H5MF__sect_large_can_shrink` sees
    /// only the large (whole-page) sections, the small per-type managers having
    /// no shrink at all — so a partial page at the end is kept and recorded,
    /// exactly as it is there.
    ///
    /// Every list is cut, including the space this session may record but never
    /// place ([`PagedEdit::unclassified`]) and the space it has vacated but
    /// cannot type ([`PagedEdit::dead`]). Bytes past the end-of-allocation are
    /// not in the file at all, so a section describing them would point outside
    /// it — the one thing the persisted managers must never do.
    ///
    /// Whole pages holding [`TRAILING_RESERVE_TAILS`] tails stay in the file, and
    /// stay recorded, for the reason [`release_trailing_run`] gives on a flat
    /// file — with the same refusal to release a run that gives back less than it
    /// keeps.
    fn release_trailing(&mut self, eof: u64, page_size: u64, tail_len: u64) -> u64 {
        let start =
            trailing_run_start([&self.meta, &self.raw, &self.dead, &self.unclassified], eof);
        let cut = align_up(start, page_size);
        if cut >= eof {
            return eof;
        }
        // Whole pages are this file's unit, so the reserve is one too, and the
        // same "give back more than you keep" condition applies.
        let keep = align_up(TRAILING_RESERVE_TAILS * tail_len, page_size);
        if eof - cut < 2 * keep {
            return eof;
        }
        let eoa = cut + keep;
        let span = eof - eoa;
        // A section straddling the cut keeps the part that is still inside the
        // file, which is what `take_range` leaves of it.
        self.meta.take_range(eoa, span);
        self.raw.take_range(eoa, span);
        self.dead.take_range(eoa, span);
        self.unclassified.take_range(eoa, span);
        eoa
    }
}

/// How much free space a released trailing run leaves behind, as a multiple of
/// the manager tail a persisting commit writes.
///
/// A persisting file rewrites those blocks on *every* commit, and a rewrite can
/// never land in its own predecessor's extent — the on-disk superblock still
/// points at it until the repoint, so overwriting it would destroy the file a
/// crash there falls back on. The end of the file therefore has to keep room for
/// the tails that follow this one, and a tail is not a fixed size: every
/// abandoned one adds a section to the managers, which makes the *next* tail
/// longer again.
///
/// Measured rather than chosen, on the shipped rule. At one,
/// `a_released_file_holds_its_length_across_many_tail_rewrites` leaves the flat
/// file alternating between 5,992 and 6,303 bytes instead of holding one length,
/// because a tail a section longer than the one that sized the reserve does not
/// fit it. At two, `persisting_churn_reaches_a_steady_size` has not settled by
/// its sixteenth round — 15,095 bytes after a middle third that held 14,685.
/// Three is the smallest value both accept; four is one tail of margin, the tail
/// length being itself a function of a section count that varies with the file.
const TRAILING_RESERVE_TAILS: u64 = 4;

/// Release the run of free space reaching `eof` from a flat file's post-commit
/// free list, and return the end-of-allocation the commit should publish
/// (issue #418).
///
/// Not the whole run: [`TRAILING_RESERVE_TAILS`] tails' worth of it stays in the
/// file, and stays recorded, so the commits after this one still have somewhere
/// to write their own manager blocks.
///
/// And nothing at all unless the run gives back at least as much as it keeps.
/// Trimming the top off a hole that is *about* to be reused is the worst of both:
/// it returns a few hundred bytes and leaves the hole too small for the object
/// that would have filled it, which then goes past end-of-file instead, so the
/// file grows by a whole object to give back a fraction of one.
///
/// So `eof` — the file's length, unchanged — when it ends in live bytes and when
/// the run is not worth releasing; the list is then left exactly as it came in.
fn release_trailing_run(post: &mut FreeList, eof: u64, tail_len: u64) -> u64 {
    let run_start = trailing_run_start([&*post], eof);
    let keep = TRAILING_RESERVE_TAILS * tail_len;
    if eof - run_start < 2 * keep {
        return eof;
    }
    post.take_range(run_start + keep, eof - run_start - keep);
    run_start + keep
}

/// Whether a persisting file's tail rewrite may grow the file to place its
/// manager blocks.
///
/// The two callers want opposite things from a tail that will not fit in free
/// space, and the difference is not a preference: a commit publishing a tree
/// *must* record its free space somewhere, while the shrink pass exists only to
/// make the file smaller and would defeat itself by appending.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum TailPlacement {
    /// Place the tail in free space where it fits and past end-of-file where it
    /// does not. Every commit that publishes a tree.
    Anywhere,
    /// Place the tail in free space or not at all: when nothing fits, the
    /// rewrite is abandoned before it writes a byte and the file is left exactly
    /// as the commit before it published.
    ReuseOnly,
}

/// Where a commit has decided to put one allocation's bytes, handed out by
/// [`WriteEngine::reserve`] and consumed by [`WriteEngine::place`].
///
/// The two arms differ in how the bytes are written — over a dead interior region
/// or past the end of the file — and in nothing else the caller sees, so a caller
/// that needs the address before the bytes (a relocatable blob) never has to know
/// which it got.
#[derive(Clone, Copy, Debug)]
enum Placement {
    /// A region an earlier commit freed, already removed from its free list.
    Reused { addr: u64, len: u64 },
    /// The end-of-file, with the page for the requested type already open.
    Appended { addr: u64, len: u64 },
}

impl Placement {
    /// The absolute file address the bytes will occupy.
    fn address(self) -> u64 {
        match self {
            Placement::Reused { addr, .. } | Placement::Appended { addr, .. } => addr,
        }
    }

    /// The reserved byte count, which the placed bytes must match exactly.
    fn len(self) -> u64 {
        match self {
            Placement::Reused { len, .. } | Placement::Appended { len, .. } => len,
        }
    }
}

/// What an in-place append names its target by.
///
/// A handle with a resolvable path uses it, which lets the session compare the
/// target against its own staged edits. A handle reached by object reference has
/// no path, so it names the dataset by the object-header address it was reached
/// through — the same key the geometry cache uses.
#[derive(Clone, Copy)]
pub(crate) enum AppendTarget<'a> {
    Path(&'a ObjectPathBuf),
    Header(u64),
}

/// Byte budget for one append batch on a session that batches: a large append is
/// split into whole-chunk batches of at most this many raw bytes (always at least
/// one chunk), each applied as its own crash-atomic fsync-barriered sequence, so
/// peak append memory never scales with the caller's slice.
const APPEND_BATCH_BYTES: u64 = 1 << 20;

/// The data-only durability barrier: an ordering point always, and an `fsync`
/// only when this session's [`SyncPolicy`] keeps the cadence.
///
/// One function rather than one per caller because the commit path
/// ([`WriteEngine::barrier_data`]) and the append engine's phase boundaries
/// ([`EditStore::sync`]) are the same point reached from two owners of the
/// image, and a barrier that orders at one and not the other is a
/// crash-consistency defect no test on the default policy can see (issue #288).
///
/// The match is exhaustive on purpose, as at [`WriteEngine::barrier`]:
/// `SyncPolicy` is sealed to the outside but not to this crate, so a policy
/// added later fails to compile here until someone decides what it means.
fn barrier_data(image: &mut dyn FileImage, sync_policy: SyncPolicy) -> Result<(), Error> {
    match sync_policy {
        SyncPolicy::Always => image.sync_data(),
        SyncPolicy::OnClose => image.ordering_barrier(),
    }
}

/// Byte budget for the writes one operation may gather before they are issued
/// (see [`WriteBuffering::Operation`]): the default every exclusively locked
/// read-write session opens under.
///
/// A megabyte, the same figure as [`APPEND_BATCH_BYTES`] and for a related
/// reason: that is already what a bounded session spends holding one batch of
/// raw append data, so this is a familiar quantum for the write path rather than
/// a new one. The two are not derived from each other and either may be tuned
/// alone. An operation larger than this is flushed part-way, which costs it
/// little — its writes are long and contiguous by then.
///
/// # Choosing a page-buffer budget
///
/// An explicit [page buffer](crate::FileAccessProperties::with_page_buffer_size)
/// *replaces* this budget rather than adding to it, and holds what it gathers
/// across operations rather than releasing it at every ordering barrier. The two
/// figures are therefore not comparable as memory: this one is a per-operation
/// peak, a page buffer's is continuous residency. A budget below it asks for less
/// of that residency and pays in writes, wherever one long contiguous run has to
/// be flushed and restarted. Writes issued on a 4 KiB-paged file:
///
/// ```text
/// workload                          unset   4 KiB   64 KiB   1 MiB
/// 32 chunk appends into 8 + commit    188      25        4       4
/// one 4 MiB append                    131    1094       74      10
/// ```
///
/// Nothing here bypasses the gatherer the way `H5PB_write` bypasses the C page
/// buffer for any I/O of a page or more (issue #357), so the second row is what
/// a small budget actually costs; `a_smaller_budget_issues_more_writes` asserts
/// that ordering rather than these counts. Which side of it to want is the
/// caller's trade, not this crate's (issue #391).
const WRITE_GATHER_BYTES: usize = 1 << 20;

/// Page the write gatherer merges within on a file that is not paged.
///
/// A non-paged file has no page size of its own, and this is the same figure
/// HDF5 defaults `H5Pset_file_space_page_size` to, so the merge quantum matches
/// what the file *would* have used had it been paged.
const DEFAULT_GATHER_PAGE: u64 = crate::file_space_info::DEFAULT_PAGE_SIZE;

/// One dataset's append geometry, handed to the public append path so it can
/// slice a large call into aligned batches without materializing the whole
/// call's bytes first.
pub(crate) struct AppendGeometry {
    /// Elements per chunk along axis 0 (>= 1).
    pub(crate) chunk_elems: u64,
    /// Bytes per on-disk element, proven non-zero.
    pub(crate) element_size: NonZeroUsize,
    /// Current length along the unlimited dimension.
    pub(crate) current_dim: u64,
    /// Whether this dataset's filter pipeline is **lossy** — ZFP, or float
    /// D-scale scale-offset. Such a dataset cannot have a partial trailing chunk
    /// grown, because that decodes and re-encodes already-committed values (see
    /// [`pipeline_lossless`]), so an append onto one has to start chunk-aligned.
    pub(crate) lossy_filters: bool,
    /// Whole-chunk elements in one full batch (>= one chunk's worth), or
    /// [`u64::MAX`] when the session does not batch.
    pub(crate) full_batch_elems: u64,
}

/// Superblock consistency-flag bits raised while a SWMR writer is active: bit 0
/// (write access) | bit 2 (SWMR write access). Cleared on a clean close. Matches
/// the reference C library, h5py, and [`crate::File::open_swmr_writer`]. These
/// are the bits every open path checks — see [`file_lock::check_status_flags`].
const SWMR_WRITE_FLAGS: u32 = file_lock::WRITE_ACCESS | file_lock::SWMR_WRITE_ACCESS;

/// A dataset located once for [`Dataset::append`](crate::Dataset::append) (or the bounded
/// backend's immediate append), then maintained across appends. Mirrors the
/// append writer's per-dataset state.
pub(crate) struct LocatedState {
    pub(crate) loc: Located,
    /// The dataset's on-disk element datatype (for the append type check and the
    /// filter chunk context).
    pub(crate) datatype: Datatype,
    /// Spatial (rank-length) chunk dimensions in elements: `[chunk_elems]`.
    pub(crate) spatial: Vec<u64>,
    /// Bytes per element (datatype size), proven non-zero.
    pub(crate) element_size: NonZeroUsize,
    /// The re-encodable filter pipeline, when the dataset is filtered.
    pub(crate) pipeline: Option<FilterPipeline>,
    /// What the slots past the new dimension must hold when an append completes
    /// a partial chunk (issue #296).
    pub(crate) fill: crate::fill_value::PaddingFill,
}

/// State for a file that persists its free space on disk. Carries the file's
/// fixed file-space parameters and the extents of the free-space-manager blocks
/// (and superblock extension) the *current* on-disk file uses, so the next
/// persisting commit can reclaim them when it writes fresh ones.
struct PersistState {
    strategy: FileSpaceStrategy,
    threshold: u64,
    page_size: u64,
    /// `(addr, len)` of the on-disk superblock-extension header and every
    /// free-space-manager `FSHD`/`FSSE` block currently in use. Superseded — and
    /// therefore freed — by the next persisting commit.
    old_blocks: Vec<(u64, u64)>,
}

/// A snapshot of a writable file's live space usage (issue #150).
///
/// This is the mutating-session counterpart of the read-only accounting on
/// [`File`](crate::File) ([`file_size`](crate::File::file_size) and
/// [`persisted_free_space`](crate::File::persisted_free_space)): it describes the
/// file *as the session currently holds it*, taken atomically at the moment of
/// the [`space_accounting`](crate::File::space_accounting) call.
///
/// It reflects the committed file plus any immediate in-place appends
/// ([`append`](crate::Dataset::append)), but **not** edits still
/// staged for the next [`commit`](crate::File::commit) — `create_group`,
/// `create_dataset`, `write_dataset`, `append_dataset`, `delete`, `copy`,
/// `copy_from`, and attribute edits change these figures only when they are
/// applied at commit. Use [`has_staged_edits`](crate::File::has_staged_edits) to
/// tell whether such pending work exists.
#[derive(Debug, Clone, PartialEq, Eq)]
#[non_exhaustive]
pub struct SpaceAccounting {
    /// The session's current logical size in bytes: the byte length of the file
    /// as the session holds it. It equals what
    /// [`File::file_size`](crate::File::file_size) reports for the file on disk
    /// right now (the HDF5 `H5Fget_filesize` value), because the session keeps its
    /// in-memory mirror byte-for-byte identical to the file — every committed
    /// write and every immediate in-place append
    /// ([`append`](crate::Dataset::append)) updates both together.
    ///
    /// It is not monotonic: [`commit`](crate::File::commit) can reclaim trailing
    /// free space and *shrink* the file. It can also exceed the superblock's
    /// recorded end-of-file address when the file was opened carrying unaccounted
    /// trailing bytes (the same slack [`File::file_size`](crate::File::file_size)
    /// surfaces), since opening does not rewrite that address.
    pub logical_size: u64,
    /// Total reusable free bytes this session can draw from before the file has to
    /// grow — the summed length of
    /// [`reusable_free_space`](Self::reusable_free_space).
    ///
    /// Counts holes left inside [`logical_size`](Self::logical_size) by this
    /// session's earlier commits (superseded object headers, the blocks of
    /// deleted objects) and, for a file created with
    /// `H5Pset_file_space_strategy(persist = true)` and no userblock, the regions
    /// seeded from the on-disk free-space managers when the session was opened (so
    /// reuse spans sessions). A fresh non-persisting session reports `0` even if
    /// the file contains holes left by other tools — those are never tracked. It
    /// is neither a lower bound on the next write's growth nor a promise of
    /// shrinkage: a region counted here may be truncated away at commit — rather
    /// than reused — if adjacent space is later freed and the coalesced run
    /// reaches end-of-file.
    ///
    /// **Which write can spend it is not uniform on a persisting file.** Such a
    /// session may write into a hole only once it has taken that hole *out* of
    /// the on-disk managers, so an in-place [`append`](crate::Dataset::append)
    /// holds a reserve of up to a megabyte per draw that this figure counts and
    /// the next [`commit`](crate::File::commit) cannot place into: the commit
    /// gives the unspent part back in the same tail that rewrites the managers,
    /// which runs *after* it has placed everything. The commit after that one
    /// can. So on a
    /// persisting file read this as "space this session will reuse rather than
    /// grow for", not as "space the very next commit can fill".
    ///
    /// It is also not the on-disk view. That reserve is deliberately absent from
    /// the free-space managers for as long as the session holds it, so
    /// [`File::persisted_free_space`](crate::File::persisted_free_space) — which
    /// reads those managers — reports less than this by whatever is reserved,
    /// and is the figure to compare against what another tool would find.
    pub reusable_free_bytes: u64,
    /// The reusable free regions as `(offset, length)` pairs, sorted ascending by
    /// offset and fully coalesced (no two regions touch or overlap).
    ///
    /// The offsets are **absolute** file offsets (from byte 0, including any
    /// userblock prefix), matching [`logical_size`](Self::logical_size). This
    /// differs from [`File::persisted_free_space`](crate::File::persisted_free_space),
    /// whose pairs are relative to the superblock base address; the two coincide
    /// for a file with no userblock (base address 0), which is the only kind whose
    /// persisted free space a session seeds. Empty when nothing is reusable.
    pub reusable_free_space: Vec<(u64, u64)>,
}

impl WriteEngine {
    /// Open an existing HDF5 file for in-place editing under an explicit
    /// file-locking policy.
    ///
    /// Reads the file into memory and retains a read/write handle. Under
    /// [`FileLocking::Enabled`] it takes an exclusive OS advisory lock so the file
    /// cannot be opened concurrently by another writer or reader; the lock is
    /// released automatically when the session is dropped or the process exits
    /// (including on a crash). Fails with [`Error::FileLocked`] if the file is
    /// already locked, or [`Error::EditUnsupported`] if the file is not a
    /// supported target; its documentation enumerates the exact requirements.
    /// `HDF5_USE_FILE_LOCKING` overrides the requested policy, as in the C
    /// library.
    pub fn open_with_locking<P: AsRef<Path>>(path: P, locking: FileLocking) -> Result<Self, Error> {
        Self::open_inner(path.as_ref(), Some(locking))
    }

    /// Open exactly as [`open_with_locking`](Self::open_with_locking) does, but
    /// behind an image that withholds its whole-file slice, so every read takes
    /// the [`Source`] path rather than the slice fast path.
    ///
    /// Distinct from [`open_rw_with_strategy`](Self::open_rw_with_strategy), which withholds the
    /// slice *and* the residency: this one still mirrors the file, so a test can
    /// compare the two read forms on a file the bounded open would refuse.
    #[cfg(test)]
    pub(crate) fn open_source_only(path: &Path) -> Result<Self, Error> {
        Self::open_imaged(path, Some(FileLocking::Enabled), |handle, _len| {
            Ok(Box::new(crate::image::SourceOnlyImage::new(
                Self::read_mirror(handle)?,
            )))
        })
    }

    /// Open exactly as [`open_with_locking`](Self::open_with_locking) does, but
    /// behind an image whose writes into `fails` behave like a dying device:
    /// the first alters the file and then errors, the rest are refused
    /// outright. See [`crate::image::TornWriteImage`] for why it takes both to
    /// reach the state a refused commit cannot repair.
    #[cfg(test)]
    pub(crate) fn open_torn_writes(
        path: &Path,
        fails: core::ops::Range<u64>,
    ) -> Result<Self, Error> {
        Self::open_imaged(path, Some(FileLocking::Enabled), |handle, _len| {
            Ok(Box::new(crate::image::TornWriteImage::new(
                Self::read_mirror(handle)?,
                fails,
            )))
        })
    }

    /// Open a bounded session whose image counts the bytes read through it, so a
    /// test can assert that an operation touches only a small part of the file.
    /// The counter is shared with the caller.
    #[cfg(test)]
    pub(crate) fn open_bounded_counting(
        path: &Path,
        read_bytes: std::sync::Arc<std::sync::atomic::AtomicU64>,
    ) -> Result<Self, Error> {
        let mut session = Self::open_imaged(path, Some(FileLocking::Enabled), |handle, len| {
            Ok(Box::new(crate::image::CountingImage::new(
                Box::new(HandleImage::new(
                    handle,
                    len,
                    crate::source::MetadataCacheConfig::disabled(),
                )),
                read_bytes,
                std::sync::Arc::default(),
            )))
        })?;
        session.batched_appends = true;
        Ok(session)
    }

    /// Open a session under `policy` whose image counts the `fsync`s issued
    /// through it, so a test can assert what a write path actually costs. The
    /// counter is shared with the caller.
    ///
    /// It takes the bounded backing and its flags, so the session matches what
    /// [`File::open_rw`](crate::File::open_rw) builds for a latest-format file;
    /// the barrier sites live on the engine, above the choice of image, so one
    /// backing exercises them all.
    #[cfg(test)]
    pub(crate) fn open_sync_counting(
        path: &Path,
        policy: SyncPolicy,
        syncs: std::sync::Arc<std::sync::atomic::AtomicU64>,
    ) -> Result<Self, Error> {
        let mut session = Self::open_imaged(path, Some(FileLocking::Enabled), |handle, len| {
            Ok(Box::new(crate::image::CountingImage::new(
                Box::new(HandleImage::new(
                    handle,
                    len,
                    crate::source::MetadataCacheConfig::disabled(),
                )),
                std::sync::Arc::default(),
                syncs,
            )))
        })?;
        session.batched_appends = true;
        session.bounded = true;
        session.set_sync_policy(policy);
        Ok(session)
    }

    /// Open an existing file for read-write editing under `strategy`: the one
    /// place that picks between the bounded backing (a [`HandleImage`] keeping no
    /// whole-file mirror, so resident memory is the metadata-cache budget plus
    /// whatever is being parsed) and the whole-file mirror. Backs
    /// [`File::open_rw`](crate::File::open_rw), which selects between them by
    /// the strategy it is given.
    ///
    /// The eligibility rules are checked here rather than deferred, because a
    /// caller who asked for bounded memory cannot be silently given the mirror
    /// instead. Under [`MemoryStrategy::Bounded`] a file the bounded engine
    /// cannot edit is refused up front; [`MemoryStrategy::Auto`] opts in to
    /// falling back to the mirror instead (issue #198, steps 3 and 4).
    ///
    /// Only a *bounded-only* limitation is worth falling back for — see
    /// [`bounded_only_limitation`]. Non-8-byte offsets or lengths are refused by
    /// [`open_imaged`](Self::open_imaged) for both engines, and so is an
    /// unsupported superblock version; a paged file without persisted free space
    /// is refused below for both.
    pub(crate) fn open_rw_with_strategy(
        path: &Path,
        cache: MetadataCacheConfig,
        locking: FileLocking,
        strategy: MemoryStrategy,
    ) -> Result<Self, Error> {
        if strategy == MemoryStrategy::Mirrored {
            return Self::open_with_locking(path, locking);
        }
        let mut session = Self::open_imaged(path, Some(locking), |handle, len| {
            Ok(Box::new(HandleImage::new(handle, len, cache)))
        })?;
        session.batched_appends = true;
        session.bounded = true;
        // Refusals that apply to *both* backings come first, or falling back for a
        // bounded-only limitation would skip them and hand back a session that
        // cannot commit. A paged file with no persisted managers has no on-disk
        // record of which pages hold metadata and which hold raw data, so nothing
        // can keep the pages segregated; the staged commit refuses it too, so
        // deferring would only trade this error for the same one later, with work
        // already staged. A userblock is one way to reach this state without the
        // file saying `persist = false`: persisted free space is declined for a
        // non-zero base address, which leaves the managers unseeded all the same.
        if session.paged.is_some() && session.persist.is_none() {
            return Err(Error::EditUnsupported(
                "read-write access to a paged file (H5F_FSPACE_STRATEGY_PAGE) requires \
                 persisted free space; recreate the file with \
                 with_file_space_strategy(FileSpaceStrategy::Page, true, ..) to grow it in place",
            ));
        }
        if let Some(reason) = bounded_only_limitation(&session) {
            if strategy == MemoryStrategy::Bounded {
                return Err(Error::EditUnsupported(reason));
            }
            // Release the handle and its exclusive lock before reopening, or the
            // mirrored open would contend with the probe we are discarding —
            // fatally so on Windows, where the OS lock is mandatory. Dropping a
            // bare `WriteEngine` writes nothing: the free-space finalize that a
            // dropped writer owes lives on `FileInner`, which this is not yet.
            // Another writer can take the lock in that window; the reopen then
            // reports `Error::FileLocked`, which is the truthful answer.
            drop(session);
            return Self::open_with_locking(path, locking);
        }
        Ok(session)
    }

    /// Open an existing file for SWMR (single-writer/multiple-reader) writing:
    /// take **no** OS lock at all and raise the superblock's SWMR-write
    /// consistency flag. Backs [`File::open_swmr_writer`](crate::File::open_swmr_writer).
    ///
    /// The no-lock is unconditional — `lock = None` never reaches
    /// `acquire_exclusive`, so `HDF5_USE_FILE_LOCKING` cannot reintroduce a lock
    /// that would block the concurrent readers SWMR exists to permit (fatally so
    /// on Windows, where OS locks are mandatory). Requires a latest-format
    /// (version-3 superblock) file with no userblock and no persisted
    /// free-space, so the superblock can be rewritten in place.
    ///
    /// The version-3 requirement is the C library's (`H5F__super_read`: "superblock
    /// version for SWMR is less than 3"), and it is what keeps the SWMR-write flag
    /// meaningful: neither library reads the status-flags byte back on an older
    /// superblock, so a flag raised there would announce a live writer to nobody.
    /// This crate's writer emits version 3, so no file it produces is affected.
    pub(crate) fn open_swmr_writer<P: AsRef<Path>>(
        path: P,
        sync_policy: SyncPolicy,
    ) -> Result<Self, Error> {
        let mut session = Self::open_inner(path.as_ref(), None)?;
        // Before the flag write below, which is a durability point like any
        // other: a caller who asked for no `fsync` gets none, and the flag still
        // reaches every other process, which reads it from the operating system.
        session.set_sync_policy(sync_policy);
        if session.superblock.version < 3
            || !session.superblock.base_address.is_zero()
            || session.persist.is_some()
        {
            return Err(Error::SwmrAppendUnsupported(
                "SWMR writing requires a latest-format file (v3 superblock) with no userblock \
                 and no persisted free-space",
            ));
        }
        session.swmr_mode = true;
        session.set_consistency_flags(SWMR_WRITE_FLAGS)?;
        Ok(session)
    }

    /// Set the superblock's consistency flags in the mirror and on disk, then
    /// flush. The one place this crate writes that byte: the SWMR-write flag on
    /// open and off on close, and the page buffer's crash mark
    /// ([`raise_crash_mark`](Self::raise_crash_mark)).
    ///
    /// Requires a version-2/3 superblock, since [`Superblock::serialize`] emits
    /// that layout; `open_swmr_writer` and
    /// [`set_page_buffer_size`](Self::set_page_buffer_size) each check it.
    ///
    /// The root address is stored on disk *relative to the base address* while
    /// the session holds it absolute (see the normalization in `open_imaged`), so
    /// the serialized clone converts it back — as the commit tail does at the
    /// only other place this crate serializes a superblock. Without it, writing
    /// the absolute address here repoints the root past the end of the file while
    /// claiming only to have touched a flag, and the file stops reading.
    ///
    /// `open_swmr_writer` happens to hold a base of zero, since it turns a
    /// userblock away outright. Its sibling no longer does:
    /// [`set_page_buffer_size`](Self::set_page_buffer_size) once required
    /// persisted free space, which is declined for a non-zero base, and issue
    /// #357 scoped that refusal to paged files — so an unpaged userblock file
    /// now reaches here with a base to convert. It was never a property to build
    /// on, which is why `a_status_flag_write_leaves_a_userblock_files_root_alone`
    /// pins the conversion directly rather than through a caller.
    fn set_consistency_flags(&mut self, flags: u32) -> Result<(), Error> {
        self.superblock.consistency_flags = flags;
        self.held_status_flags = flags;
        let mut on_disk = self.superblock.clone();
        // `open_imaged` built the in-memory address by adding the base to the
        // stored one, so this cannot be below the base — but it is a subtraction
        // over two file-derived numbers, and `relative` reports rather than wraps.
        on_disk.root_group_address = on_disk
            .base_address
            .relative(on_disk.root_group_address)?
            .get();
        let bytes = on_disk.serialize();
        self.write_at(self.sb_sig_off, &bytes)?;
        self.barrier_data()?;
        Ok(())
    }

    /// Raise the page buffer's crash mark: superblock status-flag bit 0
    /// (`H5F_SUPER_WRITE_ACCESS`), the byte the C library raises for any writer
    /// and this crate otherwise raises only for a SWMR one.
    ///
    /// Why a page buffer is not offered without one is
    /// [`with_page_buffer_size`](crate::FileAccessProperties::with_page_buffer_size)'s
    /// to explain; what matters here is that two properties of *this* function
    /// are load-bearing, and neither is the [`SyncPolicy`]'s to skip:
    ///
    /// - It runs **before** the buffering mode changes, so no buffered write can
    ///   reach the disk ahead of it. Its caller states why nothing can fail in
    ///   between.
    /// - It is **fsynced**, unconditionally — the half no test here can see.
    ///   Reaching the operating system already covers a process that dies, and
    ///   this suite reproduces no power loss, so deleting the `sync_all` fails
    ///   nothing. It is there because power loss is exactly the case where the
    ///   buffer's held writes may be partly on the platter while a mark still in
    ///   the page cache is not. One `fsync` per session buys it.
    ///
    /// A failure rolls the byte back, best-effort. The alternative is a file
    /// marked in use by a session that never opened, which nothing can read until
    /// [`File::clear_swmr_flag`](crate::File::clear_swmr_flag) — a steep price
    /// for an open that failed. Best-effort because the rollback writes through
    /// the same handle that just failed; it is a chance, not a guarantee.
    fn raise_crash_mark(&mut self) -> Result<(), Error> {
        let raised = self
            .set_consistency_flags(file_lock::WRITE_ACCESS)
            .and_then(|()| self.image.sync_all());
        if raised.is_err() {
            let _ = self.set_consistency_flags(0);
            let _ = self.image.sync_all();
        }
        raised
    }

    /// Take down whatever status flags this session raised, once everything they
    /// stood for is durable — the SWMR pair, or a page buffer's crash mark. A
    /// no-op for a session that raised none, which is every ordinary
    /// [`File::open_rw`](crate::File::open_rw) session.
    ///
    /// The whole byte goes to zero, so this cannot leave the half-set state
    /// [`check_status_flags`](crate::file_lock::check_status_flags) reports as
    /// flags in disagreement; that is why the guard asks whether anything is held
    /// rather than which bits are.
    ///
    /// Called from `File::close` and `FileInner::drop`, **after** their
    /// `force_sync`. The order is the whole point: clearing first and flushing
    /// second would leave a window in which the file's writes are still in memory
    /// and nothing on the disk says so, which is the state the flags exist to
    /// make unreachable. Nothing in this suite can tell the two orders apart —
    /// it takes a crash inside the window — so the argument is the reason, not a
    /// test.
    pub(crate) fn release_status_flags(&mut self) -> Result<(), Error> {
        if self.held_status_flags == 0 {
            return Ok(());
        }
        self.set_consistency_flags(0)?;
        self.image.sync_all()
    }

    /// Shared open path. `lock = Some(policy)` acquires an exclusive OS lock under
    /// that policy (the ordinary read-write session); `lock = None` takes no lock
    /// at all (the SWMR writer — see [`open_swmr_writer`](Self::open_swmr_writer)).
    fn open_inner(path: &Path, lock: Option<FileLocking>) -> Result<Self, Error> {
        Self::open_imaged(path, lock, |handle, _len| {
            Ok(Box::new(Self::read_mirror(handle)?))
        })
    }

    /// Read `handle` whole into a [`MirrorImage`]. The one place the engine
    /// still assumes it can hold the file, kept behind a named constructor so
    /// the mirrorless opens visibly do not call it.
    ///
    /// `read_to_end` reads from the handle's *current* cursor, and the open path
    /// has already read the superblock through it, so this rewinds first. Reading
    /// from wherever the last read landed would mirror a truncated file — with no
    /// error to say so, since a short mirror is a valid `Vec<u8>`.
    fn read_mirror(mut handle: fs::File) -> Result<MirrorImage, Error> {
        handle.seek(SeekFrom::Start(0)).map_err(Error::Io)?;
        let mut data = Vec::new();
        handle.read_to_end(&mut data).map_err(Error::Io)?;
        Ok(MirrorImage::new(handle, data))
    }

    /// Shared open path over any backing: acquire the handle (and, when asked,
    /// the exclusive lock), parse and validate the superblock through a borrowed
    /// view of the handle, and only then let `build` decide how the bytes are
    /// held.
    ///
    /// Every refusal comes before `build`, because `build` may read the whole
    /// file: reaching a refusal after it would spend `O(file size)` on a file
    /// that is then rejected — a 20 GB flagged file read into memory and thrown
    /// away. The superblock reads themselves are a few bounded windows either
    /// way, so nothing is read twice.
    ///
    /// `build` receives the file's length as well as the handle because a
    /// mirrorless image has to be told its end-of-file — it has no buffer whose
    /// length implies it.
    ///
    /// Nothing below this point reads the file as a slice, which is what lets
    /// one engine open a whole-file mirror, a mirrorless handle, and (in tests)
    /// a mirror that withholds its slice.
    fn open_imaged(
        path: &Path,
        lock: Option<FileLocking>,
        build: impl FnOnce(fs::File, u64) -> Result<Box<dyn FileImage>, Error>,
    ) -> Result<Self, Error> {
        let handle = fs::OpenOptions::new()
            .read(true)
            .write(true)
            .open(path)
            .map_err(Error::Io)?;
        // Acquire the exclusive lock before reading or mutating; the retained
        // `handle` holds it for the session's life. A `None` policy (SWMR) never
        // reaches `acquire_exclusive`, so no lock is ever taken.
        if let Some(policy) = lock {
            file_lock::acquire_exclusive(&handle, policy, path)?;
        }
        let len = handle.metadata().map_err(Error::Io)?.len();
        // Read the superblock through the handle itself, before any image owns
        // it. `probe` borrows, so it is gone by the time `build` takes the
        // handle; it leaves the handle's cursor wherever its last read ended,
        // which is why the mirror positions the handle before reading it whole.
        let probe = crate::image::BorrowedHandle::new(&handle, len);
        let sb_sig_off = signature::find_signature_in(&probe)?;
        let mut superblock = Superblock::parse_from_source(&probe, sb_sig_off)?;

        if superblock.version > 3 {
            return Err(Error::EditUnsupported("unsupported superblock version"));
        }
        // Refuse a file a writer already holds, before anything is mutated. This
        // is the one exclusion the OS lock above cannot make: a SWMR writer
        // takes no lock, so its file is lock-free but flagged (issue #245).
        file_lock::check_status_flags(
            &superblock,
            file_lock::OpenIntent::Write,
            file_lock::OpenTarget::Path(path),
        )?;
        if superblock.offset_size != OFFSET_SIZE || superblock.length_size != LENGTH_SIZE {
            return Err(Error::EditUnsupported(
                "only 8-byte offsets and lengths are supported for in-place editing",
            ));
        }
        // A userblock shifts the whole HDF5 image forward by `base_address`: the
        // superblock sits at the base address and every stored address is relative
        // to it (the end-of-file address is the sole absolute field). The editor
        // supports this by reading at `stored + base` and writing back
        // `file_offset - base`. Only the canonical layout — superblock located
        // exactly at the base address (e.g. a MATLAB v7.3 `.mat` file's 512-byte
        // userblock) — is accepted; a base address that disagrees with the
        // superblock's location is a relocated or malformed file we will not rewrite.
        if superblock.base_address != BaseAddress::new(sb_sig_off) {
            return Err(Error::EditUnsupported(
                "a file whose superblock is not located at its base address is not editable in place",
            ));
        }
        // Normalize the root group address to an absolute file offset, exactly as
        // the reader does (`reader::parse_superblock`), so `resolve_path_any` and
        // the link-graph walk index the image correctly. It is converted back to a
        // stored (base-relative) address only when the superblock is serialized on
        // commit.
        superblock.root_group_address = superblock
            .base_address
            .absolute(StoredAddress::new(superblock.root_group_address))?;

        // Everything that can refuse this file has run; only now is it worth
        // holding the bytes.
        let image = build(handle, len)?;

        let mut session = Self {
            image,
            sb_sig_off,
            superblock,
            staged: StagedEdits::default(),
            appender_claims: Vec::new(),
            next_appender_token: 0,
            free: FreeList::new(),
            reserved: FreeList::new(),
            proved_free_of_references: false,
            persist: None,
            located: HashMap::new(),
            vl_overwrite_heaps: HashMap::new(),
            superseded_heaps: Vec::new(),
            inplace_undo: Vec::new(),
            swmr_mode: false,
            paged: None,
            committed: false,
            resolved: HashMap::new(),
            batched_appends: false,
            bounded: false,
            libver_ceiling: None,
            fsm_len: len,
            publish_attempted: false,
            staging_batch: false,
            staged_generation: 0,
            held_status_flags: 0,
            sync_policy: SyncPolicy::Always,
        };
        // If the file persists its free space, seed the free list from the
        // on-disk managers and arm persistence for future commits. Best-effort:
        // an unreadable or non-persisting extension simply leaves the session in
        // the default, non-persisting mode.
        session.load_persisted_free_space();
        // Gather this session's writes, now that the page size the file was laid
        // out on is known. Only an exclusively locked session: `lock = None` is
        // the SWMR writer, whose concurrent readers observe the order its ordered
        // phases become visible in, and so must see every write as it is made
        // (issue #288).
        //
        // `lock.is_some()` is a proxy for "not the SWMR writer", used because
        // `swmr_mode` is not set until after this. It is exact only while
        // `open_swmr_writer` is the sole lock-free entry point: a future one that
        // took a lock would silently *gain* gathering, which is the direction
        // that costs a reader. The other direction — a lock-free non-SWMR open
        // losing gathering — costs only writes. Whoever adds such an entry point
        // owns this condition.
        if lock.is_some() {
            let page_size = session.gather_page_size();
            session
                .image
                .set_write_buffering(WriteBuffering::Operation {
                    page_size,
                    max_bytes: WRITE_GATHER_BYTES,
                })?;
        }
        Ok(session)
    }

    /// The page this session's writes are merged within: the file's own
    /// file-space page size when it is paged, and the format's default otherwise.
    fn gather_page_size(&self) -> u64 {
        self.paged
            .as_ref()
            .map_or(DEFAULT_GATHER_PAGE, |pg| pg.page_size)
    }

    /// Let this session's writes span operations, up to `max_bytes` of them: the
    /// `H5Pset_page_buffer_size` analogue, requested through
    /// [`FileAccessProperties::with_page_buffer_size`](crate::FileAccessProperties::with_page_buffer_size).
    ///
    /// Refused for a budget below the page this session merges within — the
    /// file's own file-space page size when it is paged and
    /// [`DEFAULT_GATHER_PAGE`] when it is not, which is what
    /// [`gather_page_size`](Self::gather_page_size) answers. A buffer that
    /// cannot hold one page flushes on every page it touches. The C library
    /// reaches the same conclusion and acts on it *silently*, rounding the budget
    /// up at `H5Fopen`, which is the one part of its behavior not worth copying.
    ///
    /// Every larger budget is honored, including one below the
    /// [`WRITE_GATHER_BYTES`] this session was already gathering under. That was
    /// refused until issue #391; see the constant for what a small one costs.
    ///
    /// It does **not** require a paged file, where `H5Pset_page_buffer_size`
    /// does. The C page buffer is a page *cache*, and `H5PB_create` rejects an
    /// unpaged file because its `min_meta_perc`/`min_raw_perc` reservations are
    /// counted in pages that the paged allocator keeps segregated by kind. This
    /// is a write gatherer instead: it merges runs within a page-sized window and
    /// flushes whole, so the window is all it needs, and `gather_page_size`
    /// supplies one on an unpaged file already — it is what every read-write
    /// session's default [`Operation`](WriteBuffering::Operation) gathering runs
    /// under. Lifting the requirement leaves the file byte-identical, which is
    /// what the unpaged arm of `a_page_buffer_holds_dirty_pages_across_operations`
    /// asserts (issue #357).
    ///
    /// A zero budget leaves the session's default gathering alone, matching the
    /// property's own "unset" value.
    pub(crate) fn set_page_buffer_size(&mut self, max_bytes: usize) -> Result<(), Error> {
        if max_bytes == 0 {
            return Ok(());
        }
        // A lock-free session's readers observe the order its writes become
        // visible in. `File::open_swmr_writer` refuses this property before it
        // raises the on-disk flag, which is the refusal a caller sees; this one is
        // here because the guarantee belongs to the engine that would break it,
        // and a SWMR session can otherwise clear every check below — an unpaged
        // version-3 file with an ample budget is exactly what SWMR writing wants.
        if self.swmr_mode {
            return Err(Error::EditUnsupported(
                "a SWMR writer cannot buffer its writes: its readers observe the order they \
                 become visible in",
            ));
        }
        // One page is the whole of the size rule, and a buffer that cannot hold
        // one drains on every page it touches.
        //
        // A budget below `WRITE_GATHER_BYTES` was refused alongside it until
        // issue #391, on the grounds that a page buffer *replaces* the byte
        // budget the session was already gathering under rather than adding to
        // it, so a smaller one is also the point at which one long run is flushed
        // and restarted. That cost is real — nothing here bypasses the gatherer
        // the way `H5PB_write` bypasses the C page buffer for I/O of a page or
        // more (issue #357) — but it is not this crate's trade to make. The
        // budget it replaces is a per-operation peak released at every barrier,
        // while this one is held across operations, so a smaller number lowers
        // both the peak and the residency, and a writer inside a tight memory cap
        // is entitled to spend the writes for that. See `WRITE_GATHER_BYTES` for
        // the measurement.
        let page_size = self.gather_page_size();
        if (max_bytes as u64) < page_size {
            return Err(Error::EditUnsupported(
                "a page buffer must be at least the file's file-space page size",
            ));
        }
        // A page buffer holds dirty pages across ordering barriers, and under
        // `Always` every barrier is an `fsync` that flushes it — so it would hold
        // nothing, and the caller would have paid the mark below for nothing in
        // return. It lives here with its three siblings rather than at the fapl,
        // because a refusal kept apart from the others is one
        // `create_would_refuse_reopen` forgets to restate, which is how this one
        // came to write the file before refusing it.
        if self.sync_policy == SyncPolicy::Always {
            return Err(Error::EditUnsupported(
                "a page buffer does nothing under SyncPolicy::Always, whose every barrier is \
                 an fsync that flushes it; pair with_page_buffer_size with \
                 with_sync_policy(SyncPolicy::OnClose), or drop it",
            ));
        }
        // A paged file whose free space is not persisted can neither commit nor
        // append — both are refused — so a page buffer would have nothing to
        // hold, while its mark blocked every reader for the session's life.
        //
        // Scoped to a paged file: an *unpaged* one commits and appends whether it
        // persists its free space or not, so there is nothing here to refuse. The
        // base address the mark's superblock rewrite needs is not this check's to
        // settle — `set_consistency_flags` converts the root address itself, and
        // is tested for a userblock file directly rather than through a caller.
        if self.paged.is_some() && self.persist.is_none() {
            return Err(Error::EditUnsupported(
                "a page buffer on a paged file needs its free space persisted: without it \
                 this session can neither commit nor append, so the buffer would hold nothing \
                 while marking the file against every reader; recreate the file with \
                 with_file_space_strategy(FileSpaceStrategy::Page, true, ..), or drop the \
                 page buffer",
            ));
        }
        // The crash mark below is the whole reason this property is offered at
        // all, and only a version-3 superblock carries one that anybody reads:
        // `check_status_flags` is gated there, matching where the C library gates
        // it. A version-2 paged file is reachable — `LibVer::V18` writes a
        // version-2 superblock and nothing about paged file space rules it out —
        // so this is a refusal rather than an assertion (issue #308).
        if self.superblock.version < 3 {
            return Err(Error::EditUnsupported(
                "a page buffer marks the file for the life of the session, so that a session \
                 that crashes leaves one every reader refuses rather than one that reads \
                 clean; only a version-3 superblock carries a status-flags byte any library \
                 reads back, and this file's is older. Rewrite it at the 1.10 format \
                 (repack, or FileBuilder's default bounds), or drop the page buffer",
            ));
        }
        // Before the mode change, and durable: from here on a write may sit in
        // memory across a barrier, and the file has to say so on the disk first.
        //
        // Nothing can fail between the two and leave the file marked with no
        // session behind it. `raise_crash_mark` ends in a `sync_all`, which
        // flushes, so the gatherer is empty when `set_write_buffering` flushes it
        // again — the one fallible step it takes.
        self.raise_crash_mark()?;
        self.image.set_write_buffering(WriteBuffering::Session {
            page_size,
            max_bytes,
        })
    }

    /// Read the superblock-extension File Space Info message; if it requests
    /// persistence, seed [`self.free`](Self::free) from the on-disk free-space
    /// managers and record the manager/extension block extents for reclamation on
    /// the next commit. Silent on any malformed or absent metadata — persistence
    /// is then simply off for this session.
    fn load_persisted_free_space(&mut self) {
        if self.superblock.version < 2 {
            return; // no superblock extension exists before v2
        }
        let Some(ext_rel) = self.superblock.superblock_extension_address else {
            return;
        };
        let ext_rel = StoredAddress::new(ext_rel);
        if ext_rel.is_undefined(self.superblock.offset_size) {
            return;
        }
        // The header is read at the extension's absolute file offset. That is the
        // same offset on the base-0 file every path below the userblock check
        // sees, but the check itself needs the strategy of a *userblock* file.
        let Ok(ext_addr) = self.superblock.base_address.absolute(ext_rel) else {
            return;
        };
        let Some(info) = self.extension_fsinfo(ext_addr) else {
            return;
        };
        // Free-space reuse and persistence are not yet base-address aware: the
        // persisted section addresses (and the extension/manager block walk below)
        // are read as absolute, so on a userblock file they would seed `self.free`
        // with wrong regions that a later allocation could hand out into live
        // data. Leave persistence off for such a file — the on-disk managers stay
        // untouched and valid, this session simply appends rather than reusing.
        //
        // A *paged* userblock file is a different matter: appending without page
        // awareness would mix metadata and raw data in its pages and leave its end
        // of allocation unaligned, quietly producing a file that still claims the
        // paged strategy but no longer satisfies it. Install the paged marker
        // without persistence so the commit refusal below catches it, which is the
        // same rule a paged non-persisting file already takes.
        if !self.superblock.base_address.is_zero() {
            if info.strategy == FileSpaceStrategy::Page && info.page_size > 0 {
                self.paged = Some(PagedEdit::new(info.page_size));
            }
            return;
        }
        // Record the paged strategy regardless of the persist flag: a paged commit
        // needs page-aware bookkeeping, and a paged file that does not persist its
        // free space is refused outright (see `PagedEdit` and the commit refusal).
        //
        // A zero page size is refused rather than installed: every page calculation
        // divides by it, so a corrupt or hostile file declaring `Page` with a page
        // size of 0 would panic the editor. Leaving `paged` unset makes the file
        // take the ordinary flat path, which needs no page geometry.
        let paged = info.strategy == FileSpaceStrategy::Page && info.page_size > 0;
        if paged {
            self.paged = Some(PagedEdit::new(info.page_size));
        }
        if !info.persist {
            return;
        }
        let os = self.superblock.offset_size;
        let file_len = self.image.len();

        // Seed the free list(s) with every persisted section (addresses are stored
        // relative to the base address, which this editor requires to be 0).
        // Defensive against a malformed or corrupt manager: skip a section that is
        // empty, runs past end-of-file, or overlaps one already taken. A
        // well-formed file (this crate's or the C library's) has none of these;
        // tolerating them keeps a bad file from seeding a bogus or double-counted
        // free region that a later commit would hand out into live data.
        if paged {
            // A paged file's free space is segregated across per-page-type
            // managers, so read each slot on its own and keep the page type its
            // slot implies ([`PagedEdit::slot_list`]). Flattening them (as the
            // non-paged path below does) would lose exactly the distinction the
            // commit has to preserve. A section whose slot does not settle its page
            // type is recorded but never handed out.
            let page_size = info.page_size;
            let mut tagged: Vec<(FreeSection, Option<PageType>)> = Vec::new();
            for (slot, &m) in info.manager_addrs.iter().enumerate() {
                if StoredAddress::new(m).is_undefined(os) {
                    continue;
                }
                let Ok(sections) = free_space_manager::read_persisted_sections_source(
                    &self.image(),
                    &[m],
                    BaseAddress::ZERO,
                    os,
                )
                .map(|(sections, _)| sections) else {
                    continue;
                };
                for s in sections {
                    let ty = PagedEdit::slot_list(slot, s.addr.get(), s.size, page_size);
                    tagged.push((s, ty));
                }
            }
            // Distinct sections have distinct addresses in any well-formed file,
            // so the tie-break never arises; only a malformed manager can
            // advertise one address twice, and the overlap guard below already
            // discards the second of any such pair. No `debug_assert` here: this
            // parses untrusted bytes, which must not panic a debug build.
            tagged.sort_unstable_by_key(|(s, _)| s.addr);
            let mut prev_end = 0u64;
            for (s, ty) in tagged {
                let Some(end) = s.addr.get().checked_add(s.size) else {
                    continue;
                };
                if s.size == 0 || end > file_len || s.addr.get() < prev_end {
                    continue;
                }
                prev_end = end;
                let pg = self
                    .paged
                    .as_mut()
                    .expect("the paged state was just installed");
                match ty {
                    Some(ty) => PagedEdit::route_free(
                        &mut pg.meta,
                        &mut pg.raw,
                        &mut pg.dead,
                        s.addr.get(),
                        s.size,
                        ty.into(),
                    ),
                    None => pg.unclassified.free(s.addr.get(), s.size),
                }
            }
            // A page the seeded lists empty between them belongs to no type and
            // may serve either, so fold it in now rather than leaving it to the
            // first commit. Nothing this crate writes produces one — it files a
            // whole free page under a single manager — but a file another writer
            // laid out can, and the rule is the same wherever the page came from.
            let pg = self
                .paged
                .as_mut()
                .expect("the paged state was just installed");
            PagedEdit::promote_whole_free_pages(&mut pg.meta, &mut pg.raw, &mut pg.dead, page_size);
        } else if let Ok(mut sections) = free_space_manager::read_persisted_sections_source(
            &self.image(),
            &info.manager_addrs,
            BaseAddress::ZERO,
            os,
        )
        .map(|(sections, _)| sections)
        {
            // Unique addresses in any well-formed file; see the paged branch
            // above for why a duplicate is harmless and unasserted here.
            sections.sort_unstable_by_key(|s| s.addr);
            let mut prev_end = 0u64;
            for s in sections {
                let Some(end) = s.addr.get().checked_add(s.size) else {
                    continue;
                };
                if s.size == 0 || end > file_len || s.addr.get() < prev_end {
                    continue;
                }
                prev_end = end;
                self.free.free(s.addr.get(), s.size);
            }
        }

        // Record the byte extents of the blocks the live file uses so the next
        // persisting commit frees them when it writes replacements: the
        // extension header, and each defined manager's FSHD + FSSE.
        let mut old_blocks = Vec::new();
        if let Ok(spans) = self.oh_chunk_spans(ext_addr) {
            old_blocks.extend(spans);
        }
        for &m in &info.manager_addrs {
            if StoredAddress::new(m).is_undefined(os) {
                continue;
            }
            let Ok(hdr_len) = fshd_len(os).to_usize() else {
                continue;
            };
            let Ok(fshd) = self.image().read_metadata_at(m, hdr_len) else {
                continue;
            };
            if let Ok(h) = FsmHeader::parse(&fshd, os) {
                // `FsmHeader::parse` succeeding guarantees the header's own bytes
                // are present, so the FSHD extent is in-bounds; validate the
                // section-info extent before recording it, so a malformed
                // `fsse_used` can't later free a region running past end-of-file.
                old_blocks.push((m, fshd_len(os)));
                if !h.fsse_addr.is_undefined(os)
                    && h.fsse_addr
                        .get()
                        .checked_add(h.fsse_used)
                        .is_some_and(|end| end <= file_len)
                {
                    old_blocks.push((h.fsse_addr.get(), h.fsse_used));
                }
            }
        }

        self.persist = Some(PersistState {
            strategy: info.strategy,
            threshold: info.threshold,
            page_size: info.page_size,
            old_blocks,
        });
    }

    /// Parses the File Space Info message out of the superblock-extension object
    /// header at `ext_addr`, if present and readable.
    fn extension_fsinfo(&self, ext_addr: u64) -> Option<FileSpaceInfo> {
        let os = self.superblock.offset_size;
        let ls = self.superblock.length_size;
        let base = self.superblock.base_address;
        let oh = ObjectHeader::parse_from_source(
            &self.image(),
            AccessMode::ReadWrite,
            ext_addr,
            os,
            ls,
            base,
        )
        .ok()?;
        let msg = oh
            .messages
            .iter()
            .find(|m| m.msg_type == MessageType::FILE_SPACE_INFO)?;
        FileSpaceInfo::parse(&msg.data, os, ls).ok()
    }

    /// Stage a new dataset, added on the next [`commit`](Self::commit).
    ///
    /// `path` is the dataset's full path; everything before the last component
    /// names the parent group, which must exist (or be created in this session).
    /// `builder` is the same [`DatasetBuilder`] [`FileBuilder`](crate::FileBuilder)
    /// uses, configured by the caller; its name is taken from `path`, so the two
    /// cannot disagree.
    ///
    /// The builder is passed in finished rather than handed out as a `&mut` into
    /// this engine. That is deliberate: a borrow into the engine forces the
    /// caller to hold it — and, above this layer, its lock — for as long as the
    /// builder is being configured, which is what let a user closure deadlock
    /// against the same file it was reading (issue #200).
    ///
    /// The dataset may be contiguous or chunked, and chunked datasets may be
    /// filtered (`with_deflate`, `with_shuffle`, `with_fletcher32`,
    /// `with_scale_offset`, `with_zfp`) and/or extensible (`with_maxshape`). An
    /// empty (zero-element) dataset is supported under either storage, a
    /// provenance dataset (`with_provenance`) is supported, and a
    /// contiguous dataset may carry variable-length attributes, a
    /// variable-length-string payload (`with_vlen_strings`), or path-resolved
    /// object-reference elements (`with_path_references`; chunking any of
    /// these is not supported). An attribute set too large for the object header
    /// is written to a fractal heap, as the whole-file writer does.
    pub(crate) fn stage_created_dataset(
        &mut self,
        path: &ObjectPathBuf,
        builder: DatasetBuilder,
    ) -> Result<(), Error> {
        self.refuse_if_claimed(path)?;
        let Some((parent, leaf)) = path.split_leaf() else {
            return Err(Error::EditUnsupported("dataset path has an empty name"));
        };
        self.refuse_creation_collision(path, StagedKind::Dataset)?;
        self.staged
            .push_dataset(parent, flatten_dataset(builder, leaf)?);
        Ok(())
    }

    /// Stage an in-place overwrite of an **existing** dataset's values (the HDF5
    /// `H5Dwrite` whole-dataset write), applied on the next
    /// [`commit`](Self::commit).
    ///
    /// `path` is the full path of a dataset that must already exist; `builder`
    /// supplies the replacement data and is named from `path`, as in
    /// [`stage_created_dataset`](Self::stage_created_dataset).
    ///
    /// This is a *value* overwrite, not a reshape or retype: the new data's
    /// datatype and shape must match the on-disk dataset's exactly (byte-for-byte
    /// after serialization, so endianness and compound layout must agree), or
    /// `commit` reports [`Error::EditUnsupported`]. Contiguous, compact, and
    /// chunked (including filtered) datasets are all supported; the dataset's
    /// existing chunk geometry, filter pipeline, and chunk index are taken from the
    /// on-disk header. A chunk index this engine cannot enumerate (a version-2
    /// B-tree) is refused. Partial / sub-region writes are out of scope — the
    /// whole dataset is replaced.
    ///
    /// What the builder alone can rule out —
    /// [`refuse_unsupported_overwrite`](Self::refuse_unsupported_overwrite) —
    /// is refused *here* rather than at `commit`.
    ///
    /// When the new data is the same length as the existing contiguous data block
    /// (the common case), the bytes are written straight into that block: no
    /// object header is rewritten and the superblock root is not flipped, so the
    /// commit's linearization point is the synced data write itself. A chunked
    /// dataset is handled the same way when every (re-encoded) chunk is the same
    /// byte length as the slot it replaces — an unfiltered overwrite (chunk sizes
    /// are fixed by the unchanged shape) or a filtered one whose re-encoded chunks
    /// match — so it too writes straight into the existing chunk slots. When the
    /// length differs (a resized contiguous block, or a filtered chunk that no
    /// longer fits), the dataset's storage is rebuilt at end-of-file (or in
    /// reusable freed space), the old extent is freed, the data-layout message is
    /// repointed, the object header is rewritten, and the parent group's link is
    /// patched — exactly like an addition relocates the path up to the root. A
    /// relocating overwrite moves the object header, so it is refused unless the
    /// dataset has a single hard link.
    pub(crate) fn stage_dataset_write(
        &mut self,
        path: &ObjectPathBuf,
        builder: DatasetBuilder,
    ) -> Result<(), Error> {
        self.refuse_if_claimed(path)?;
        // The root group has no link to overwrite: the superblock holds its address.
        let Some((_, leaf)) = path.split_leaf() else {
            return Err(Error::EditUnsupported("cannot overwrite the root group"));
        };
        let fd = flatten_dataset(builder, leaf)?;
        Self::refuse_unsupported_overwrite(&fd)?;
        self.staged.writes.push((path.clone(), fd));
        Ok(())
    }

    /// Stage an append of new elements to an **existing** chunked, unlimited
    /// dataset, applied on the next [`commit`](Self::commit).
    ///
    /// `path` names a dataset that must already exist; `builder` supplies the
    /// elements to add via its typed / generic / raw `append_*` methods.
    ///
    /// Unlike [`stage_dataset_write`](Self::stage_dataset_write) (a value
    /// overwrite that forbids any shape change) this **grows** the dataset along
    /// its first (axis-0) dimension. It works on **filtered** datasets: the
    /// appended chunks are compressed through the dataset's own on-disk filter
    /// pipeline (deflate / shuffle / fletcher32 / scale-offset / LZF, and ZFP
    /// with the `zfp` feature), and the pipeline, datatype, fill value, and attributes are
    /// preserved verbatim. Appends of any length are supported — when the
    /// dataset's current length is not a whole multiple of the chunk length, the
    /// single trailing partial chunk is read, extended, and re-encoded; every
    /// other existing chunk is carried by metadata alone, so the existing data is
    /// not rewritten and the file does not grow by the whole dataset per append.
    ///
    /// This does **not** use SWMR and sets no consistency flag. Like every other
    /// staged edit it commits by appending the new chunks and a rebuilt
    /// index at end-of-file and repointing the superblock last (under the
    /// session's exclusive lock), so a crash leaves either the original dataset or
    /// the fully-grown one, never a torn state.
    ///
    /// The first release supports the Extensible-Array chunk index (the index the
    /// reference C library and h5py select for a single unlimited dimension under
    /// the latest format, and the one this crate writes for every unlimited
    /// dataset), rank-1 datasets, and datasets with a single hard link. A dataset
    /// that is not chunked, not unlimited along axis 0, not Extensible-Array
    /// indexed, higher than rank 1, uses a filter this engine cannot re-encode,
    /// has a sparse chunk grid, or (for [`append_raw`](AppendBuilder::append_raw))
    /// has a big-endian element datatype is refused with
    /// [`Error::AppendUnsupported`]. Use [`Dataset::is_chunked`](crate::Dataset::is_chunked),
    /// [`maxshape`](crate::Dataset::maxshape), and [`filters`](crate::Dataset::filters)
    /// to check eligibility up front.
    pub(crate) fn stage_dataset_append(
        &mut self,
        path: &ObjectPathBuf,
        builder: AppendBuilder,
    ) -> Result<(), Error> {
        self.refuse_if_claimed(path)?;
        if self.staged.dataset_at(path).is_some() {
            // This call came through a handle onto the object *in the file*: a
            // handle onto the staged creation goes to
            // [`stage_dataset_append_pending`] instead. So the elements could
            // only be meant for the object the same commit replaces, which is
            // never what the caller wants and is what `commit` refuses two calls
            // later. Said here, where the handle that asked is still in hand.
            return Err(Error::EditUnsupported(
                "a dataset is staged at this path in the same commit, so an append through a \
                 handle onto the object the file holds there could only grow the object being \
                 replaced; append through the handle that staged the creation, or commit first",
            ));
        }
        self.refuse_lossy_partial_tail(path, &builder)?;
        self.staged.appends.push((path.clone(), builder));
        Ok(())
    }

    /// Refuse, at the call that stages an append, a dataset whose lossy pipeline
    /// sits on a partial trailing chunk — the refusal
    /// [`prepare_append`](Self::prepare_append) raises during the commit's
    /// preflight, brought forward to where the caller can act on it (issue #407).
    ///
    /// Waiting for the commit makes that refusal permanent for the session and
    /// costs every edit staged beside it: a preflight failure restores the staged
    /// set, so the offending append is still there at the next
    /// [`commit`](crate::File::commit), and [`File::close`](crate::File::close)
    /// runs one and returns its `Err` — discarding the whole staged set on the way
    /// out. `BufferedAppender::new` refuses the same dataset eagerly for the same
    /// reason; this puts the staged path on the same footing.
    ///
    /// **Positive proof only.** Every reason the geometry cannot be read here — a
    /// path that does not resolve, a header this engine does not parse, a layout
    /// that is not chunked, a rank this release does not append to — leaves the
    /// answer to the preflight, which names each of them. Nothing about the bytes
    /// being appended enters into it: the on-disk length is what decides whether a
    /// trailing chunk has to be re-encoded, and staged appends never change that
    /// before the commit reads it. The commit-time check stays as the backstop,
    /// and is the only one that runs for an append staged through some other
    /// entry point.
    fn refuse_lossy_partial_tail(
        &self,
        path: &ObjectPathBuf,
        builder: &AppendBuilder,
    ) -> Result<(), Error> {
        // A zero-length append is dropped by the preflight before it ever reaches
        // `prepare_append`, so it stays a no-op here rather than becoming the one
        // append this refusal would newly reject.
        if builder.raw().is_empty() {
            return Ok(());
        }
        if self.appends_onto_a_lossy_partial_tail(path) {
            return Err(Error::AppendUnsupported(LOSSY_TAIL_REFUSAL));
        }
        Ok(())
    }

    /// The geometry question behind [`refuse_lossy_partial_tail`](Self::refuse_lossy_partial_tail):
    /// does the file hold, at `path`, a rank-1 chunked dataset under a lossy
    /// filter pipeline whose length is not a whole multiple of its chunk length?
    /// `false` whenever that cannot be established, for whatever reason.
    fn appends_onto_a_lossy_partial_tail(&self, path: &ObjectPathBuf) -> bool {
        let Ok(addr) = crate::group_v2::resolve_path_any_from_source(
            &self.image(),
            AccessMode::ReadWrite,
            &self.superblock,
            &path.as_path(),
        ) else {
            return false;
        };
        let Ok(region) =
            Self::gather_oh_messages(&self.image(), addr, self.superblock.base_address)
        else {
            return false;
        };
        let mut datatype: Option<(usize, usize)> = None;
        let mut dataspace: Option<(usize, usize)> = None;
        let mut layout: Option<(usize, usize)> = None;
        let mut filter: Option<(usize, usize)> = None;
        let mut p = 0;
        while let Ok(Some((msg_type, body, body_end))) = region.next_message(p) {
            match msg_type {
                MessageType::DATATYPE => datatype = Some((body, body_end)),
                MessageType::DATASPACE => dataspace = Some((body, body_end)),
                MessageType::DATA_LAYOUT => layout = Some((body, body_end)),
                MessageType::FILTER_PIPELINE => filter = Some((body, body_end)),
                _ => {}
            }
            p = body_end;
        }
        // No pipeline at all is nothing to re-encode, and an unparseable one is
        // refused by the preflight on its own terms.
        let (Some((fb, fe)), Some((dt_b, dt_e)), Some((ds_b, ds_e)), Some((lb, le))) =
            (filter, datatype, dataspace, layout)
        else {
            return false;
        };
        let Ok(pipeline) = FilterPipeline::parse(&region[fb..fe]) else {
            return false;
        };
        if pipeline_lossless(&pipeline) {
            return false;
        }
        let Ok((disk_dt, _)) = Datatype::parse(&region[dt_b..dt_e]) else {
            return false;
        };
        let Ok(disk_ds) = Dataspace::parse(&region[ds_b..ds_e], LENGTH_SIZE) else {
            return false;
        };
        let Ok(dl) = DataLayout::parse(&region[lb..le], OFFSET_SIZE, LENGTH_SIZE) else {
            return false;
        };
        // Rank 1 is the shape the preflight's `current_dim0 % chunk_elems`
        // arithmetic is about; anything else it refuses before reaching that test.
        if disk_ds.dimensions.len() != 1 {
            return false;
        }
        let Ok(geometry) = chunked_geometry(&disk_dt, &disk_ds, &dl) else {
            return false;
        };
        match (geometry.spatial.first(), disk_ds.dimensions.first()) {
            (Some(&chunk_elems), Some(&dim0)) if chunk_elems != 0 => dim0 % chunk_elems != 0,
            _ => false,
        }
    }

    /// Stage an append made through a handle onto a dataset **this session
    /// staged and has not committed**, folding the elements into the pending
    /// creation.
    ///
    /// The caller establishes that its handle is the pending one;
    /// [`stage_dataset_append`](Self::stage_dataset_append) is the entry point
    /// for a handle onto an object the file already holds, and refuses the same
    /// path. Between the caller's check and this lock another thread can commit,
    /// which leaves the dataset in the file and nothing staged at its path — so
    /// the append becomes an ordinary staged one rather than an error.
    pub(crate) fn stage_dataset_append_pending(
        &mut self,
        path: &ObjectPathBuf,
        builder: AppendBuilder,
    ) -> Result<(), Error> {
        self.refuse_if_claimed(path)?;
        if self.staged.dataset_at(path).is_none() {
            // Nothing staged here after all, so these elements grow the object the
            // file holds — including its trailing chunk, which the same eager
            // refusal applies to. The fold below reaches no on-disk chunk at all
            // and is untouched by it.
            self.refuse_lossy_partial_tail(path, &builder)?;
            self.staged.appends.push((path.clone(), builder));
            return Ok(());
        }
        self.refuse_mid_batch()?;
        self.extend_staged_dataset(path, &builder)
    }

    /// Refuse an edit that changes or withdraws something already staged while a
    /// [`stage_atomically`](Self::stage_atomically) batch is open.
    ///
    /// [`StagedEdits::rewind`] undoes a batch by truncating the staged vectors,
    /// which is exact only while staging appends to them. The two operations
    /// that do otherwise — the fold
    /// [`stage_dataset_append_pending`](Self::stage_dataset_append_pending)
    /// makes, and the withdrawal [`delete`](Self::delete) makes — would leave a
    /// refused batch holding half of one. No caller reaches this today: a batch
    /// replays [`StagedOp`](crate::StagedGroup)s, which only ever create. It is
    /// here so the claim `rewind` rests on is enforced rather than remembered.
    fn refuse_mid_batch(&self) -> Result<(), Error> {
        if self.staging_batch {
            return Err(Error::EditUnsupported(
                "an edit that changes or withdraws an already-staged object cannot be made \
                 inside an atomic staging batch, whose undo drops additions only",
            ));
        }
        Ok(())
    }

    /// Fold an append onto a dataset this same session staged and has not
    /// committed into the pending creation: the elements are added to the
    /// bytes the commit will write and the leading dimension grows to match.
    ///
    /// There is no dataset in the file to grow, so this is the same edit as
    /// having handed the elements to the builder in the first place — which is
    /// why it asks for neither an unlimited dimension nor a chunked layout,
    /// where an append onto a *committed* dataset needs both. Anything the
    /// staged creation must resolve at commit time per element — a
    /// variable-length string payload, an object reference — is refused: those
    /// carry a side table indexed by element that this cannot extend.
    ///
    /// A provenance dataset (`with_provenance`) is grown rather than refused:
    /// its attributes are derived from the bytes, so they are rebuilt from the
    /// grown ones ([`FlatDataset::rebuild_provenance`]) and the digest the
    /// commit writes is the digest of what it writes.
    fn extend_staged_dataset(
        &mut self,
        path: &ObjectPathBuf,
        builder: &AppendBuilder,
    ) -> Result<(), Error> {
        if builder.dt_conflict() {
            return Err(Error::AppendUnsupported(
                "this append mixes element datatypes; use one element type per append",
            ));
        }
        let fd = self
            .staged
            .dataset_at_mut(path)
            .expect("caller checked a dataset is staged at this path");
        if fd.vl_string_staging.is_some() || fd.reference_targets.is_some() {
            return Err(Error::AppendUnsupported(
                "a staged variable-length-string or object-reference dataset cannot be appended \
                 to before it is committed: its per-element side table is built with the \
                 dataset. Supply every element through the builder that creates it",
            ));
        }
        if let Some(dt) = builder.elem_dt() {
            if *dt != fd.dt {
                return Err(Error::AppendUnsupported(
                    "appended element datatype does not match the staged dataset's datatype",
                ));
            }
        }
        let Some((lead, inner)) = fd.ds.dimensions.split_first() else {
            return Err(Error::AppendUnsupported(
                "a scalar dataset has no dimension to append along",
            ));
        };
        // One row of the leading dimension, in bytes: the element size times
        // every trailing extent. A rank-1 dataset's row is one element.
        let mut row_bytes = u64::from(fd.dt.type_size());
        for &d in inner {
            row_bytes = row_bytes.checked_mul(d).ok_or(Error::AppendUnsupported(
                "staged dataset row size overflows",
            ))?;
        }
        if row_bytes == 0 {
            return Err(Error::AppendUnsupported(
                "a staged dataset whose rows hold no bytes cannot be appended to",
            ));
        }
        let bytes = builder.raw();
        if bytes.len() as u64 % row_bytes != 0 {
            return Err(Error::AppendUnsupported(
                "appended byte length is not a whole number of rows of the staged dataset",
            ));
        }
        let grown = lead
            .checked_add(bytes.len() as u64 / row_bytes)
            .ok_or(Error::AppendUnsupported("staged dataset length overflows"))?;
        // A finite maximum is a promise the commit would refuse to break.
        if let Some(max) = fd
            .ds
            .max_dimensions
            .as_ref()
            .and_then(|m| m.first().copied())
            && !max.admits(grown)
        {
            return Err(Error::AppendUnsupported(
                "appending would grow the staged dataset past its maximum shape",
            ));
        }
        fd.raw.extend_from_slice(bytes);
        fd.ds.dimensions[0] = grown;
        // The digest covers the bytes, so it is recomputed over the grown ones
        // rather than left describing the shorter dataset that was staged. Doing
        // it here, on the record the commit writes, is what keeps
        // `verify_provenance` passing on the file this produces.
        #[cfg(feature = "provenance")]
        fd.rebuild_provenance();
        Ok(())
    }

    /// Register a live [`BufferedAppender`](crate::BufferedAppender) on `path`
    /// (`None` for a handle reached by object reference), returning the token
    /// that releases it.
    ///
    /// Refused when the claim could not be honored: a second appender on the
    /// same dataset would interleave the two buffers a chunk at a time, and a
    /// staged edit already pending on that path is one the appender's own flush
    /// would later refuse.
    pub(crate) fn claim_for_appender(
        &mut self,
        path: Option<&ObjectPathBuf>,
    ) -> Result<u64, Error> {
        if self
            .appender_claims
            .iter()
            .any(|claim| claim.conflicts_with(path))
        {
            return Err(Error::EditUnsupported(
                "this dataset already has a live buffered appender; two of them would interleave \
                 their buffers a chunk at a time",
            ));
        }
        let blocked = match path {
            Some(p) => self.append_conflicts_with_pending(p),
            None => self.has_staged_edits() || self.committed,
        };
        if blocked {
            return Err(Error::EditUnsupported(
                "this session holds staged edits that would stop a buffered appender from \
                 flushing; commit or discard them before opening one",
            ));
        }
        let token = self.next_appender_token;
        self.next_appender_token += 1;
        self.appender_claims.push(AppenderClaim {
            token,
            path: path.cloned(),
        });
        Ok(token)
    }

    /// Drop a claim. Called from the appender's `Drop`, after its final flush.
    pub(crate) fn release_appender_claim(&mut self, token: u64) {
        self.appender_claims.retain(|c| c.token != token);
    }

    /// Refuse a staged edit at `path` that a live appender could not survive.
    ///
    /// An appender holds elements a caller has already been told were accepted,
    /// and only its own flush can write them; that flush goes through the
    /// immediate append path, which refuses a dataset with a staged edit on it or
    /// an ancestor. Left unchecked, staging such an edit turns accepted data into
    /// data lost silently in `Drop`. Refusing here moves the failure to the call
    /// that creates the conflict, where there is someone to report it to.
    fn refuse_if_claimed(&self, path: &ObjectPathBuf) -> Result<(), Error> {
        let conflicts = self.appender_claims.iter().any(|c| match &c.path {
            Some(p) => p.overlaps(path),
            None => true,
        });
        if conflicts {
            return Err(Error::EditUnsupported(
                "this dataset has a live buffered appender holding elements only its own flush \
                 can write, and this edit would stop that flush; finish or discard the appender \
                 first",
            ));
        }
        Ok(())
    }

    /// Whether this session is the SWMR writer, whose append rules are a strict
    /// subset of the ordinary ones (see `append_inplace_gathered`).
    pub(crate) fn is_swmr(&self) -> bool {
        self.swmr_mode
    }

    /// Whether any staged tree edit is still uncommitted. In-place appends
    /// ([`append_inplace_gathered`](Self::append_inplace_gathered)) are applied immediately and are
    /// never staged, so they never affect this; it reflects only edits awaiting
    /// [`commit`](Self::commit) — `create_group`, `create_dataset`,
    /// `write_dataset`, `append_dataset`, group and dataset attribute edits,
    /// `delete`, `copy`, and `copy_from`. Dropping the session silently discards
    /// any staged edits.
    ///
    /// A [`commit`](Self::commit) that refuses leaves this answering `true`: the
    /// staged set it declined to apply is put back exactly as it was.
    pub fn has_staged_edits(&self) -> bool {
        !self.staged.is_empty()
    }

    /// Run `f`, and if it fails, drop whatever it managed to stage.
    ///
    /// One caller-facing call can stage many edits — `create_group_with` stages
    /// a group, its attributes and its whole subtree — and each is validated as
    /// it is staged, so the fifth dataset can be refused after four have been
    /// recorded. Without this, such a call would return `Err` having changed the
    /// session anyway, which is the half of issue #316 that lives one level
    /// above `commit`: a refused operation must cost the session nothing.
    pub(crate) fn stage_atomically<R>(
        &mut self,
        f: impl FnOnce(&mut Self) -> Result<R, Error>,
    ) -> Result<R, Error> {
        let mark = self.staged.mark();
        // Saved and restored rather than simply cleared, so a nested batch does
        // not lift the outer one's guard on its way out.
        let outer = std::mem::replace(&mut self.staging_batch, true);
        let result = f(self);
        self.staging_batch = outer;
        if result.is_err() {
            self.staged.rewind(mark);
        }
        result
    }

    /// This session's file image as one slice, when its backing holds the whole
    /// file in memory; `None` for a file-backed image.
    ///
    /// The slice reflects committed state plus immediate in-place appends, not
    /// edits still staged for `commit`. The owned read-write
    /// [`File`](crate::File) uses it to serve reads by borrowing rather than
    /// copying, and falls back to [`image`](Self::image) when it is absent.
    pub(crate) fn image_slice(&self) -> Option<&[u8]> {
        self.image.as_slice()
    }

    /// A random-access [`Source`] view of this session's file image, for the
    /// parsers the edit engine drives.
    ///
    /// Every read the engine performs against the file goes through this, so the
    /// image needs to be no more than a source of bytes — whole-file mirror or
    /// not — without the parsers knowing which (issue #198).
    pub(crate) fn image(&self) -> &dyn Source {
        self.image.as_ref()
    }

    /// This session's parsed superblock, with `root_group_address` normalized to
    /// an absolute file offset (the open-time convention) and `base_address` the
    /// userblock size. A relocating commit updates it, so a caller holding a
    /// clone from an earlier moment may be reading a stale root.
    pub(crate) fn superblock(&self) -> &Superblock {
        &self.superblock
    }

    /// The file's shared-message (SOHM) master table, or `None` for a file that
    /// shares no messages — which is every file this crate writes and nearly
    /// every file it is handed.
    ///
    /// Costs one small object-header parse per commit on a file that has a
    /// superblock extension at all, and nothing on one that does not.
    fn shared_message_table(&self) -> Result<Option<crate::sohm::SohmTable>, Error> {
        let os = self.superblock.offset_size;
        let rel = match self
            .superblock
            .superblock_extension_address
            .map(StoredAddress::new)
        {
            Some(rel) if !rel.is_undefined(os) => rel,
            _ => return Ok(None),
        };
        let base = self.superblock.base_address;
        let ls = self.superblock.length_size;
        // Best-effort down to here, and strict past it. A superblock extension
        // this engine cannot parse is one nothing in this crate reads a shared
        // message through either, so it is treated as a file that shares none;
        // once the extension is readable and *says* the file shares messages,
        // failing to read the table is an error, because the screen below is
        // then the difference between a refusal and a corrupted index.
        let Ok(abs) = base.absolute(rel) else {
            return Ok(None);
        };
        let Ok(header) = ObjectHeader::parse_from_source(
            &self.image(),
            AccessMode::ReadWrite,
            abs,
            os,
            ls,
            base,
        ) else {
            return Ok(None);
        };
        let Some(msg) = header
            .messages
            .iter()
            .find(|m| m.msg_type == MessageType::SHARED_MESSAGE_TABLE)
        else {
            return Ok(None);
        };
        let message = crate::sohm::SharedMessageTableMessage::parse(&msg.data, os)?;
        let framed = BaseOffsetSource {
            inner: self.image(),
            base,
        };
        Ok(Some(crate::sohm::SohmTable::read_from_source(
            &framed, &message, os,
        )?))
    }

    /// Refuse a commit that would strand a shared-message index record naming an
    /// object header. See [`SHARED_MESSAGE_INDEX_NAMES_A_MOVED_OBJECT`].
    fn screen_shared_message_index(&self, invalidated: &InvalidatedAddresses) -> Result<(), Error> {
        if invalidated.is_empty() {
            return Ok(());
        }
        let Some(table) = self.shared_message_table()? else {
            return Ok(());
        };
        let base = self.superblock.base_address;
        let os = self.superblock.offset_size;
        let ls = self.superblock.length_size;
        let framed = BaseOffsetSource {
            inner: self.image(),
            base,
        };
        for index in &table.indexes {
            let records = crate::sohm::read_index_records_from_source(&framed, index, os, ls)?;
            screen_shared_message_records(&records, invalidated)?;
        }
        Ok(())
    }

    /// The on-disk format this session writes, read from the file it opened
    /// rather than chosen fresh, since a commit preserves the superblock version
    /// it found.
    ///
    /// This decides the *contiguous* data-layout message version, where the
    /// older number costs nothing: versions 3 and 4 have identical bodies there,
    /// so a dataset added to a 1.8 file stays readable by a 1.8 library.
    ///
    /// It does not gate chunked storage. A chunked dataset needs the version 4
    /// layout and a 1.10 chunk index whatever the superblock says, so adding one
    /// to an older file does make that file need 1.10 — deliberately, since the
    /// alternative is a version 1 B-tree index this crate does not write, and
    /// refusing instead would take away in-place editing of every file the C
    /// library wrote with its own default bounds. `crates/crosscheck/tests/main/edit.rs`
    /// covers exactly that case (issue #101).
    ///
    /// That is a default, not a verdict: a caller who needs the file to stay
    /// loadable by an old reader asks for it with
    /// [`FileAccessProperties::with_libver_bounds`], which sets
    /// [`libver_ceiling`](Self::libver_ceiling) and turns the addition into a
    /// refusal at commit rather than a silent format bump.
    ///
    /// [`FileAccessProperties::with_libver_bounds`]: crate::FileAccessProperties::with_libver_bounds
    pub(crate) fn libver(&self) -> LibVer {
        LibVer::from_superblock_version(self.superblock.version)
    }

    /// Constrain what this session may add, from the fapl's library-version
    /// bounds. See [`libver_ceiling`](Self::libver_ceiling).
    ///
    /// Resolved through [`LibVer::resolve_writable`], the same rule the
    /// whole-file writer applies, so bounds admitting no format this crate
    /// writes are refused here as they are there rather than silently ignored on
    /// the editing path.
    pub(crate) fn set_libver_bounds(
        &mut self,
        bounds: Option<(LibVer, LibVer)>,
    ) -> Result<(), Error> {
        self.libver_ceiling = match bounds {
            Some(_) => Some(LibVer::resolve_writable(bounds).map_err(Error::Format)?),
            None => None,
        };
        Ok(())
    }

    /// Refuse staged content the session's [`libver_ceiling`](Self::libver_ceiling)
    /// cannot express, before the commit writes anything.
    ///
    /// Only chunked storage is at stake: `build_chunked_dataset_oh` writes a
    /// version 4 data-layout message and a 1.10 chunk index unconditionally,
    /// while everything else this session adds is expressible in both formats
    /// (`libver` already picks the contiguous layout version). A filter or an
    /// unlimited dimension arrives as chunked storage, so they are covered here
    /// too.
    fn check_libver_admits<'a>(
        &self,
        datasets: impl IntoIterator<Item = &'a FlatDataset>,
    ) -> Result<(), Error> {
        let Some(ceiling) = self.libver_ceiling else {
            return Ok(());
        };
        if ceiling >= LibVer::V110 {
            return Ok(());
        }
        let chunked = datasets
            .into_iter()
            .any(|fd| fd.chunk_options.is_chunked() || fd.maxshape.is_some());
        if chunked {
            return Err(Error::Format(FormatError::LibverTooOldForContent {
                content: "a chunked, filtered, or resizable dataset",
                needs: LibVer::V110.name(),
                writing: ceiling.name(),
            }));
        }
        Ok(())
    }

    /// Which backend this session resolved to: [`Bounded`] when it reads through
    /// a handle, [`Mirrored`] when it holds a whole-file image.
    ///
    /// [`Bounded`]: EditBacking::Bounded
    /// [`Mirrored`]: EditBacking::Mirrored
    pub(crate) fn edit_backing(&self) -> EditBacking {
        if self.bounded {
            EditBacking::Bounded
        } else {
            EditBacking::Mirrored
        }
    }

    /// A snapshot of this session's live space usage — the current file size and
    /// the free space it can reuse — as a [`SpaceAccounting`].
    ///
    /// This is the mutating-session analogue of the read-only accounting on
    /// [`File`](crate::File): it answers "how big is the file right now, and how
    /// much space can be reused before it must grow?" from the session's own live
    /// state. The snapshot reflects the committed file plus any immediate in-place
    /// appends ([`append_inplace_gathered`](Self::append_inplace_gathered)) but excludes edits still
    /// staged for the next [`commit`](Self::commit); see [`SpaceAccounting`] for
    /// the field-by-field semantics and [`has_staged_edits`](Self::has_staged_edits)
    /// for detecting pending work.
    ///
    /// On a paged file (`H5F_FSPACE_STRATEGY_PAGE`) the reported regions are the
    /// union of the per-page-type managers, which a commit draws on under the rule
    /// paging exists for: an allocation takes a hole of the page type it is placing,
    /// or whole pages that are free of both, so reuse never re-mixes a page
    /// (issue #261). Space whose page type this file does not settle is recorded
    /// and handed back to the reference library but never spent, and is excluded
    /// from the reusable figure.
    ///
    /// ```no_run
    /// use hdf5_pure::File;
    ///
    /// let file = File::open_rw("existing.h5")?;
    /// let acct = file.space_accounting()?;
    /// println!(
    ///     "{} bytes on disk, {} reusable in {} free region(s)",
    ///     acct.logical_size,
    ///     acct.reusable_free_bytes,
    ///     acct.reusable_free_space.len(),
    /// );
    /// # Ok::<(), hdf5_pure::Error>(())
    /// ```
    #[expect(
        rustdoc::private_doc_tests,
        reason = "rustdoc's test mode collects a private item, so this example is compiled and run by `cargo test --doc`"
    )]
    #[must_use]
    pub fn space_accounting(&self) -> SpaceAccounting {
        // The append reserve is free space this session can still spend — it is
        // out of the on-disk managers, not out of the file — so leaving it out
        // would make an in-place append look like it had consumed bytes it has
        // not yet placed anything in (issue #387).
        //
        // It is folded back through `FreeList::free` rather than appended and
        // sorted, because a reserve span very often *abuts* what is left of the
        // hole it was drawn from — and the spent end of the reserve walks toward
        // that remainder until the two touch exactly. Reported as two regions
        // they would break the field's documented contract in the one way that
        // misleads: a caller sizing an allocation against the largest region
        // would be told nothing that big fits when it does.
        //
        // It goes back into the list it came from and no other. On a paged file
        // that is the raw list, which is what `take_raw_span` drew it out of;
        // the two page-type lists stay reported side by side as they always
        // were, because a caller cannot allocate across a page-type boundary and
        // merging them would claim it could. `FreeList::free`'s debug assertion
        // comes along and is welcome: the reserve is drawn out of exactly this
        // list, so an overlap here would mean that invariant had already broken.
        //
        // A paged file tracks its free space per page type; report the union of
        // the two, since the caller wants one total rather than a per-manager
        // breakdown.
        let reusable_free_space = match (&self.paged, self.reserved.is_empty()) {
            // The common case, and every non-persisting session: no reserve to
            // fold, so nothing is cloned.
            (Some(pg), true) => pg.reusable_sections(),
            (None, true) => self.free.sections(),
            (Some(pg), false) => {
                let mut raw = pg.raw.clone();
                for (addr, len) in self.reserved.sections() {
                    raw.free(addr, len);
                }
                let mut out = pg.meta.sections();
                out.extend(raw.sections());
                out.sort_unstable_by_key(|&(addr, _)| addr);
                debug_assert!(
                    out.windows(2).all(|w| w[0].0 < w[1].0),
                    "the same address is free in both the metadata and the raw \
                     list, so the two page-type lists have stopped being disjoint"
                );
                out
            }
            (None, false) => {
                let mut free = self.free.clone();
                for (addr, len) in self.reserved.sections() {
                    free.free(addr, len);
                }
                free.sections()
            }
        };
        let reusable_free_bytes = reusable_free_space.iter().map(|(_, len)| len).sum();
        SpaceAccounting {
            logical_size: self.image.len(),
            reusable_free_bytes,
            reusable_free_space,
        }
    }

    /// The shared Extensible-Array append engine's view of this session: the
    /// image, the superblock, and the paged-file state, paired as [`EditStore`].
    ///
    /// Takes `&mut self`, so a caller that also needs [`located`](Self::located)
    /// borrowed at the same time must destructure the fields itself rather than
    /// call this.
    fn store(&mut self) -> EditStore<'_> {
        EditStore {
            image: self.image.as_mut(),
            superblock: &mut self.superblock,
            sb_sig_off: self.sb_sig_off,
            paged: self.paged.as_mut(),
            // This adapter's one caller only reads; nothing is allocated here.
            free: None,
            sync_policy: self.sync_policy,
        }
    }

    /// Whether an immediate in-place append may allocate out of already-free
    /// space instead of growing the file (issue #349, issue #387).
    ///
    /// The staged [`commit`](Self::commit) always may: it publishes its
    /// allocations with the superblock repoint, so a crash before that leaves a
    /// file whose root never named them. An immediate append has no repoint, so
    /// what has to hold instead is that the bytes it overwrites are dead *on the
    /// disk as it stands*. Which list satisfies that depends on the session:
    ///
    /// - An ordinary [`File::open_rw`](crate::File::open_rw) session on a
    ///   default-strategy file holds its free space in memory alone. A region in
    ///   [`free`](Self::free) was freed by a commit this session already
    ///   published, so nothing on the disk points at it and nothing on the disk
    ///   claims it is free; a crash at any point of the append leaves it as dead
    ///   as it was. It is spent directly.
    /// - A file that **persists** its free-space managers records its holes on
    ///   disk, so a hole in `free` is one a durable manager still advertises and
    ///   spending it would leave that manager describing bytes a live chunk
    ///   occupies — through a clean close as much as a crash, since only a commit
    ///   or [`finalize_persist`](Self::finalize_persist) rewrites the record.
    ///   Such a session therefore spends [`reserved`](Self::reserved) instead:
    ///   space [`reserve_for_immediate_append`](Self::reserve_for_immediate_append)
    ///   has already taken *out* of the published managers.
    /// - A **paged** file keeps its free space per page type in [`PagedEdit`]
    ///   rather than in [`free`](Self::free), so drawing from that list would be
    ///   drawing from the wrong one and would put a byte of one kind in a page of
    ///   the other. It draws its reserve through [`PagedEdit::alloc_typed`] with
    ///   [`PageType::Raw`] instead, which is what keeps a page holding one kind.
    ///   Named as its own term at the call site rather than left to the argument
    ///   that the persist term already covers it (`append_prepare` refuses a
    ///   paged file that does not persist): a page-type invariant proved from
    ///   another function's refusal is exactly the shape that has gone wrong here
    ///   before (issue #261). It costs nothing to name, because `reserved` is
    ///   empty on a session that never reserved — the term declines reuse rather
    ///   than misdirecting it.
    ///
    /// The **SWMR** writer is the one absolute bar, and it is not about
    /// publication at all: its concurrent readers may still be inside a region
    /// this session freed, and the format forbids reusing one while they are,
    /// which is why the C library's own SWMR writer allocates append-only. No
    /// test can reach this term today, because a SWMR session refuses every
    /// staged edit ([`Error::SwmrStagedUnsupported`]) and so never frees
    /// anything. It is here so that lifting that refusal cannot silently enable
    /// reuse.
    ///
    /// One window is inherited rather than introduced. A commit whose superblock
    /// write itself fails leaves [`publish_attempted`](Self::publish_attempted)
    /// set, which withholds the free-list rollback, so the regions that commit
    /// freed stay in the list although whether it published is unknowable
    /// (issue #344). An append can now spend them where only a later commit could
    /// before. That is the same trade, on a larger surface: doing nothing is
    /// still the only answer that is never actively wrong.
    fn immediate_reuse_allowed(&self) -> bool {
        !self.swmr_mode
    }

    /// Take a batch of free space out of the on-disk free-space managers so an
    /// immediate in-place append may spend it, for a session that persists them
    /// (issue #387). Afterwards [`reserved`](Self::reserved) holds whatever could
    /// be drawn; the allocator that spends it is what says whether that serves
    /// the append, since the rewrite's own tail may have taken part of it.
    ///
    /// The draw and the publication are one step and in that order: the bytes
    /// leave [`free`](Self::free) (or [`PagedEdit`]'s raw list) first, and the
    /// managers are then rewritten through the ordinary persisting commit tail
    /// with nothing to free and the root unchanged — the same call
    /// [`finalize_persist`](Self::finalize_persist) makes, whose superblock
    /// repoint is the crash-atomic linearization point. Before that repoint the
    /// old managers stand and still list the batch, which is correct because
    /// nothing has been written into it; after it, no durable record calls those
    /// bytes free and the append may have them.
    ///
    /// Nothing here moves a live object: `to_free` is empty and the new root is
    /// the old one, so the tail places only itself, in a hole or past the live
    /// data. The [`located`](Self::located) append-geometry cache is therefore
    /// still valid across this call, which is what lets it run between an
    /// append's plan and its apply.
    ///
    /// Draws nothing — not an error — when the session persists nothing to draw
    /// from, or when no hole can hold the allocation in hand; the caller simply
    /// grows the file, as it always did. An error means the manager rewrite
    /// itself failed, and the lists are put back so they agree with whatever the
    /// disk now holds, on the rule [`commit`](Self::commit) applies: a rewrite that
    /// never reached its superblock write left the managers as they were, so the
    /// lists go back to exactly what they were, tail placement included; one that
    /// did may have published, so the draw is handed back — those bytes are
    /// genuinely free, nothing having been written into them, and the next commit
    /// records them again — while the span its tail took stays out, since that
    /// tail may now be the live one.
    fn reserve_for_immediate_append(&mut self, want: u64) -> Result<(), Error> {
        if want == 0 || self.persist.is_none() || self.swmr_mode || self.reserved.largest() >= want
        {
            return Ok(());
        }
        let snapshot = self.snapshot_free();
        if !self.take_raw_spans(want, APPEND_RESERVE_BYTES) {
            return Ok(());
        }
        self.publish_attempted = false;
        match self.commit_persisting(
            self.superblock.root_group_address,
            Vec::new(),
            TailPlacement::Anywhere,
        ) {
            Ok(()) => Ok(()),
            Err(e) => {
                if self.publish_attempted {
                    self.release_reserve();
                } else {
                    self.restore_free(snapshot);
                }
                Err(e)
            }
        }
    }

    /// Move raw-appendable free space into [`reserved`](Self::reserved) for one
    /// manager rewrite to publish: runs that can each hold `want` bytes, until
    /// `cap` bytes are held or no hole that large is left. Reports whether
    /// anything was drawn, which is exactly whether the first hole could hold
    /// `want`.
    ///
    /// The draw is sized on what the file has rather than on a fixed batch, and
    /// that is the whole of the fix for issue #413 — [`APPEND_RESERVE_BYTES`]
    /// says why the cap is a cap and not a floor. The rewrite that publishes a
    /// draw is the expensive part, so one draw gathers as many holes as the cap
    /// allows and the rewrite is paid once for all of them: a file fragmented
    /// into chunk-sized holes costs one rewrite per cap's worth of them, not one
    /// per chunk.
    ///
    /// Each run is sized on the largest hole and taken best fit, so it is a whole
    /// hole or a `cap`-bounded prefix of one, and the reserve never holds a
    /// fragment too small for the allocation that drew it. `want` above `cap`
    /// draws one run of exactly `want`: the cap bounds the amortization, not
    /// what a single allocation may need.
    fn take_raw_spans(&mut self, want: u64, cap: u64) -> bool {
        debug_assert!(want > 0, "a draw is sized on a real allocation");
        // The cap is the budget, except that a single allocation larger than it
        // still draws one run for itself.
        let budget = cap.max(want);
        let mut drawn = 0u64;
        while budget - drawn >= want {
            let available = self.largest_raw_run();
            if available < want {
                break;
            }
            let len = available.min(budget - drawn);
            if !self.take_raw_span(len) {
                // Sized on the allocator's own figure, so it serves; a refusal
                // here means the two disagree, and drawing nothing further is
                // the answer that leaves the file merely larger.
                debug_assert!(false, "a run of {len} bytes was reported and not served");
                break;
            }
            drawn += len;
        }
        drawn > 0
    }

    /// The longest contiguous run [`take_raw_span`](Self::take_raw_span) could
    /// move into the reserve right now, or `0` when nothing could.
    fn largest_raw_run(&self) -> u64 {
        match self.paged.as_ref() {
            Some(pg) => pg.largest_typed(PageType::Raw),
            None => self.free.largest(),
        }
    }

    /// Move `len` bytes of raw-appendable free space into
    /// [`reserved`](Self::reserved), reporting whether a contiguous run that
    /// large was available.
    ///
    /// A paged file draws through [`PagedEdit::alloc_typed`] with
    /// [`PageType::Raw`], the same call the staged commit makes for a dataset's
    /// data, so the reserve is raw-typed space and an append spending it cannot
    /// mix a page. A flat file has one list and draws from it.
    fn take_raw_span(&mut self, len: u64) -> bool {
        let addr = match self.paged.as_mut() {
            Some(pg) => pg.alloc_typed(len, PageType::Raw),
            None => self.free.alloc(len),
        };
        match addr {
            Some(addr) => {
                self.reserved.free(addr, len);
                true
            }
            None => false,
        }
    }

    /// Give the unspent append reserve back to the lists the managers are written
    /// from, so the next rewrite records it as the free space it is.
    ///
    /// Called immediately before every rewrite of the on-disk managers — the
    /// persisting commit tail and [`finalize_persist`](Self::finalize_persist) —
    /// and nowhere else, so a commit that does *not* rewrite them (the
    /// same-length-overwrite fast path) leaves the reserve intact for the appends
    /// that follow it.
    ///
    /// The reserve is raw-typed on a paged file, so it goes back to the raw list.
    /// A run that was claimed whole from the metadata list is raw space now, and
    /// a page of it that is wholly free can be claimed back by either type
    /// through [`PagedEdit::alloc_typed`], exactly as any other free page can.
    fn release_reserve(&mut self) {
        if self.reserved.is_empty() {
            return;
        }
        let spans = std::mem::replace(&mut self.reserved, FreeList::new()).sections();
        match self.paged.as_mut() {
            Some(pg) => {
                for (addr, len) in spans {
                    pg.raw.free(addr, len);
                }
            }
            None => {
                for (addr, len) in spans {
                    self.free.free(addr, len);
                }
            }
        }
    }

    /// Rewrite the on-disk free-space managers for a persisting commit, giving
    /// the unspent append reserve back first so the managers record it, and then
    /// give any trailing free space the rewrite exposed back to the filesystem.
    ///
    /// Every caller that publishes a *tree* takes this rather than
    /// [`commit_persisting`](Self::commit_persisting) directly. The one caller
    /// that does not is [`reserve_for_immediate_append`](Self::reserve_for_immediate_append),
    /// whose whole purpose is to publish managers that leave the reserve out.
    fn commit_persisting_releasing_reserve(
        &mut self,
        new_root: u64,
        to_free: Vec<(u64, u64, FreeClass)>,
    ) -> Result<(), Error> {
        self.release_reserve();
        self.commit_persisting(new_root, to_free, TailPlacement::Anywhere)?;
        self.shrink_to_the_trailing_free_run(new_root)
    }

    /// Move a commit's manager blocks down into the space that commit freed, and
    /// truncate the file to them (issue #418).
    ///
    /// A commit that frees a run reaching end-of-file cannot land its own tail in
    /// that run: the regions it is vacating are still referenced by the *on-disk*
    /// root until the superblock repoint, so writing over them would destroy the
    /// pre-commit file if the process died in between. The tail therefore goes
    /// past end-of-file, above the very space that should have been given back,
    /// and the file cannot shrink — the delete-then-commit case of issue #418.
    ///
    /// Once that repoint is durable those regions are genuinely free, so a second
    /// tail-only rewrite can place the blocks inside them. It publishes through
    /// the same crash-atomic tail as any other commit, with nothing to free and
    /// the root unchanged, and the truncation follows its superblock write.
    ///
    /// It runs under [`TailPlacement::ReuseOnly`], so it places the tail in free
    /// space or writes nothing at all. That is the safety property: a pass taken
    /// for the sake of a smaller file can never leave a larger one.
    ///
    /// Skipped unless the last commit's tail is the highest thing in the file and
    /// the free run reaching it is long enough for [`release_trailing_run`] to
    /// release any of — the shape a delete leaves, and the condition that bounds
    /// this to one extra rewrite on the commits that can actually use it. A
    /// non-persisting file never gets here (its commit truncates directly), and
    /// neither does the SWMR writer, which
    /// [`open_swmr_writer`](crate::File::open_swmr_writer) refuses to open on a
    /// file that persists its free space at all.
    fn shrink_to_the_trailing_free_run(&mut self, new_root: u64) -> Result<(), Error> {
        let Some(persist) = self.persist.as_ref() else {
            return Ok(());
        };
        let (Some(tail_start), Some(tail_end)) = (
            persist.old_blocks.iter().map(|&(a, _)| a).min(),
            persist.old_blocks.iter().map(|&(a, l)| a + l).max(),
        ) else {
            return Ok(());
        };
        // Only a tail at the very end of the file is standing in the way of one.
        if tail_end != self.image.len() {
            return Ok(());
        }
        // The free run reaching up to it, plus the tail the rewrite frees, is what
        // the rewrite would have to release from. Held to the same threshold
        // `release_trailing_run` applies, since below it that call returns the
        // file's length unchanged and the pass would cost a rewrite to publish
        // exactly what the commit before it did.
        let run_start = match self.paged.as_ref() {
            Some(pg) => {
                trailing_run_start([&pg.meta, &pg.raw, &pg.dead, &pg.unclassified], tail_start)
            }
            None => trailing_run_start([&self.free], tail_start),
        };
        if tail_end - run_start < 2 * TRAILING_RESERVE_TAILS * (tail_end - tail_start) {
            return Ok(());
        }
        self.commit_persisting(new_root, Vec::new(), TailPlacement::ReuseOnly)
    }

    /// Adopt the fapl's `fsync` cadence. Called once, on the funnel every
    /// read-write open passes through, before the session is handed out.
    pub(crate) fn set_sync_policy(&mut self, policy: SyncPolicy) {
        self.sync_policy = policy;
    }

    /// The cadence this session adopted. Nothing in the crate branches on this —
    /// [`barrier`](Self::barrier) does that — but the fapl reaching the engine is
    /// otherwise invisible from outside, and an entry point that forgot to pass
    /// it on would look exactly like one that did.
    #[cfg(test)]
    pub(crate) fn sync_policy(&self) -> SyncPolicy {
        self.sync_policy
    }

    /// Override how long this session's writes may sit in memory.
    ///
    /// The image is private to this module, and the only caller outside it is
    /// [`crate::crash_replay`], which sweeps the *same* workload under gathering
    /// and without it. That comparison is the answer to the obvious worry about
    /// making writes larger, so it is worth an accessor; nothing outside a test
    /// build has one.
    #[cfg(test)]
    pub(crate) fn set_write_buffering(&mut self, mode: WriteBuffering) -> Result<(), Error> {
        self.image.set_write_buffering(mode)
    }

    /// Every write this session issued, as `(offset, length)` in the order it
    /// went out. Same reasoning as [`set_write_buffering`](Self::set_write_buffering):
    /// the image is private, and a test that asks whether a publish left the
    /// engine as one write has nowhere else to look.
    #[cfg(test)]
    pub(crate) fn issued_write_order(&self) -> Vec<(u64, u64)> {
        self.image.issued_write_order()
    }

    /// The durability barrier a write path calls for, data and metadata both —
    /// issued unless this session's [`SyncPolicy`] leaves the `fsync` cadence to
    /// the application.
    ///
    /// Every such point routes through here rather than touching the image, so
    /// the policy is honored by construction at sites yet to be written.
    ///
    /// A barrier is an **ordering** point first and a durability point second,
    /// and only the second half is the policy's to skip. The writes gathered
    /// before it are issued here whatever the policy says, because the image
    /// issues gathered writes in address order: leave them gathered across a
    /// barrier and a commit's superblock — address 0 — goes to the disk *before*
    /// the content it names, which is the one order the whole tail is built to
    /// avoid. A write that then fails, or a process that dies mid-flush, would
    /// leave a superblock naming bytes that are not in the file, where before it
    /// left the previous file intact (issue #288).
    ///
    /// [`WriteBuffering::Session`] — an explicit page buffer — is the deliberate
    /// exception, and [`ordering_barrier`](crate::image::FileImage::ordering_barrier)
    /// is what encodes that: it does nothing under that mode, which is precisely
    /// the guarantee such a caller is trading away. What that caller gets back is
    /// a file that *says* so — see [`raise_crash_mark`](Self::raise_crash_mark).
    ///
    /// The teardown barrier is deliberately *not* one of these points:
    /// [`File::close`](crate::File::close) and `FileInner::drop` write after the
    /// last barrier a caller could have asked for, so they take
    /// [`force_sync`](Self::force_sync) instead.
    fn barrier(&mut self) -> Result<(), Error> {
        // Exhaustive on purpose, here and at the two sites below: `SyncPolicy` is
        // sealed to the outside but not to this crate, so a policy added later
        // fails to compile at every durability point until someone decides what
        // it means there.
        match self.sync_policy {
            // `sync_all` issues the gathered writes on its way to the disk.
            SyncPolicy::Always => self.image.sync_all(),
            SyncPolicy::OnClose => self.image.ordering_barrier(),
        }
    }

    /// The data-only counterpart to [`barrier`](Self::barrier), for a write that
    /// does not move end-of-file and so needs no metadata flush. It orders the
    /// gathered writes for the same reason.
    fn barrier_data(&mut self) -> Result<(), Error> {
        barrier_data(self.image.as_mut(), self.sync_policy)
    }

    /// Force this session's writes to durable storage *whatever* the policy
    /// says. The counterpart to [`barrier`](Self::barrier), which the policy may
    /// skip.
    ///
    /// Two callers: [`File::sync`](crate::File::sync), the application naming
    /// its own cadence, and the teardown path — `File::close` and
    /// `FileInner::drop` — whose own writes no caller can order, because both
    /// destroy the handle that would have done it.
    pub(crate) fn force_sync(&mut self) -> Result<(), Error> {
        self.image.sync_all()
    }

    /// Rewrite the on-disk free-space managers for a file that persists them, if
    /// this session left them stale.
    ///
    /// Immediate in-place appends grow the file past the managers, leaving them
    /// describing a length the file no longer has, or spend a reserve those
    /// managers no longer list. A staged
    /// [`commit`](Self::commit) rewrites them as part of its tail, but a session
    /// that only appends never runs one, so [`File::close`](crate::File::close)
    /// and `FileInner::drop` call this instead. It is the same tail the commit
    /// writes — placed in space an earlier commit freed, or appended past
    /// everything live when none fits, with the superblock repoint as the
    /// crash-atomic linearization point — with nothing to free and the root
    /// unchanged.
    ///
    /// Where those blocks end up is not something a reader has to care about, and
    /// after this change usually is not end-of-file: a file with free space to
    /// spend has its managers inside it, above and below live data alike. What has
    /// to hold is that they describe the file as it now stands, which is what makes
    /// this call's own writes the last the session issues.
    ///
    /// It is also where a session's last release of trailing free space happens,
    /// since it takes the same tail as a commit (issue #418).
    ///
    /// A no-op for a non-persisting file, and skipped when the file has not grown
    /// past the managers since they were last written *and* the session holds no
    /// unspent append reserve, so an unchanged session never grows the file. The
    /// reserve is the second condition because an append that spent a hole rather
    /// than the end of the file leaves the length alone, and what it did not
    /// spend is space the managers must be told about again (issue #387).
    ///
    /// If a session that grew the file ends without `close` or `drop` running (a
    /// true crash — `SIGKILL`, power loss), the managers are left describing the
    /// shorter file, and an unspent reserve is left out of them altogether. Every
    /// append was durable and crash-atomic, so no data is lost, and both this
    /// crate and the reference C library reopen the file and read it correctly;
    /// the managers simply under-report until a clean rewrite, which wastes those
    /// bytes and never hands them out twice.
    pub(crate) fn finalize_persist(&mut self) -> Result<(), Error> {
        if self.persist.is_none() || (self.image.len() == self.fsm_len && self.reserved.is_empty())
        {
            return Ok(());
        }
        self.commit_persisting_releasing_reserve(self.superblock.root_group_address, Vec::new())
    }

    /// Resolve and locate an in-place append target, applying every rule that
    /// does not depend on the bytes being appended: the file-level eligibility
    /// guards, the staged-edit conflict check, and the geometry lookup that
    /// populates [`located`](Self::located). Returns the dataset's object-header
    /// address, which is that cache's key.
    ///
    /// Split out from the append itself so [`append_geometry`](Self::append_geometry)
    /// can report a dataset's batching geometry under exactly the same rules the
    /// append will apply — a caller slicing a large append into batches must be
    /// refused before the first batch, not part-way through.
    fn append_prepare(&mut self, target: AppendTarget<'_>) -> Result<u64, Error> {
        // The fast in-place append is only sound on a base-0 latest-format file:
        // the slot math assumes absolute addresses and the superblock is patched in
        // place per call. A userblock or pre-v2 file falls back to the staged
        // `append_dataset`, which rebuilds the index and repoints the superblock
        // last.
        if !self.superblock.base_address.is_zero() {
            return Err(Error::AppendInPlaceUnsupported(
                "in-place append does not support a file with a userblock (non-zero base \
                 address); use Dataset::append_staged",
            ));
        }
        if self.superblock.version < 2 {
            return Err(Error::AppendInPlaceUnsupported(
                "in-place append requires a latest-format file (v2/v3 superblock); use \
                 Dataset::append_staged",
            ));
        }
        // A paged file (`H5F_FSPACE_STRATEGY_PAGE`) that does not persist its free
        // space has no on-disk record of which pages hold metadata and which hold
        // raw data, so neither this immediate append nor the staged commit can keep
        // the two segregated; refuse it outright. A paged *persisting* file appends
        // through the page-aware `EditStore`, which pads a tail page whenever the
        // page type changes, and has its managers rewritten at the next commit or
        // at close (issue #198).
        if self.paged.is_some() && self.persist.is_none() {
            return Err(Error::AppendInPlaceUnsupported(
                "in-place append is not supported on a paged file \
                 (H5F_FSPACE_STRATEGY_PAGE) without persisted free space; recreate the \
                 file with with_file_space_strategy(FileSpaceStrategy::Page, true, ..)",
            ));
        }

        // Refuse an append against a dataset (or a subtree) that a still-staged edit
        // in this same session will relocate, replace, or delete — which would
        // strand the durably-appended rows or plan against a header the commit
        // moves. The caller must commit those edits first.
        //
        // A target named by object-header address cannot be compared against the
        // staged paths, so any staged edit at all disqualifies it. That is a
        // superset of the path check, and the remedy is the same one.
        match target {
            AppendTarget::Path(dataset) => {
                if self.append_conflicts_with_pending(dataset) {
                    return Err(Error::AppendInPlaceUnsupported(
                        "the dataset or an ancestor has a staged edit pending in this session; \
                         commit the staged edits before appending in place, or use \
                         Dataset::append_staged",
                    ));
                }
            }
            AppendTarget::Header(_) if self.has_staged_edits() || self.committed => {
                return Err(Error::AppendInPlaceUnsupported(
                    "this append target was reached by object reference, so it names a dataset \
                     by object-header address, and this session has staged or committed edits \
                     that can move that header; re-open the dataset by path to append to it",
                ));
            }
            AppendTarget::Header(_) => {}
        }

        // Resolve the dataset's object-header address — the geometry cache key.
        // base == 0 here, so a resolved address is absolute; two hard links to
        // one dataset share the one entry.
        let oh_addr = match target {
            AppendTarget::Path(dataset) => match self.resolved.get(dataset) {
                Some(&addr) => addr,
                None => {
                    let addr = crate::group_v2::resolve_path_any_from_source(
                        &self.image(),
                        AccessMode::ReadWrite,
                        &self.superblock,
                        &dataset.as_path(),
                    )?;
                    self.resolved.insert(dataset.clone(), addr);
                    addr
                }
            },
            AppendTarget::Header(addr) => addr,
        };

        // Locate the dataset on the first append (cache miss) against the session's
        // own image — no second lock, no second view of the file, no re-read.
        if !self.located.contains_key(&oh_addr) {
            let store = self.store();
            let state = locate_dataset_state(&store, oh_addr)?;
            self.located.insert(oh_addr, state);
        }
        Ok(oh_addr)
    }

    /// The append geometry of the dataset `target` names, so a caller can slice a
    /// large append into aligned batches *before* materializing each batch's
    /// bytes — which is what keeps a bounded session's peak memory at one batch
    /// rather than the whole call.
    pub(crate) fn append_geometry(
        &mut self,
        target: AppendTarget<'_>,
    ) -> Result<AppendGeometry, Error> {
        let oh_addr = self.append_prepare(target)?;
        let st = &self.located[&oh_addr];
        let chunk_elems = st.loc.chunk_elems.max(1);
        Ok(AppendGeometry {
            chunk_elems,
            element_size: st.element_size,
            current_dim: st.loc.current_dim,
            lossy_filters: st.pipeline.as_ref().is_some_and(|p| !pipeline_lossless(p)),
            full_batch_elems: self.batch_elems(st.loc.chunk_bytes, chunk_elems),
        })
    }

    /// Whole-chunk elements in one append batch.
    ///
    /// A bounded session caps a batch at [`APPEND_BATCH_BYTES`] of raw data (at
    /// least one chunk) so peak memory is independent of the call size, at the
    /// cost of splitting one crash-atomic append into several: a crash between
    /// batches leaves a valid shorter dataset, exactly as if the caller had
    /// looped. A mirror session already holds the whole file, so bounding the
    /// call buys nothing there and would trade that atomicity away for free;
    /// it takes the whole append as one batch.
    fn batch_elems(&self, chunk_bytes: usize, chunk_elems: u64) -> u64 {
        if !self.batched_appends {
            return u64::MAX;
        }
        (APPEND_BATCH_BYTES / (chunk_bytes.max(1) as u64)).max(1) * chunk_elems
    }

    /// Apply a gathered in-place append (typed / generic / raw bytes) to the
    /// dataset `target` names, immediately and crash-atomically, driving the
    /// shared Extensible-Array engine against the session's own image through an
    /// [`EditStore`] adapter. Runs only the first `max_phase` durability phases;
    /// production callers pass 4, the crash-consistency tests stop at a boundary
    /// to simulate a crash.
    ///
    /// A bounded session splits the call into whole-chunk batches (see
    /// [`batch_elems`](Self::batch_elems)), each its own crash-atomic apply.
    /// Every predictable refusal is raised before the first batch, so a rejected
    /// append leaves the file untouched rather than partly grown.
    ///
    /// `Dataset::append` slices its own call the same way before it reaches here,
    /// so through the public API this loop runs once per call; it batches for the
    /// benefit of a caller that hands the engine one large builder directly, whose
    /// bytes are already materialized but whose plan need not be.
    pub(crate) fn append_inplace_gathered(
        &mut self,
        target: AppendTarget<'_>,
        b: &AppendBuilder,
        max_phase: u8,
    ) -> Result<(), Error> {
        if b.dt_conflict() {
            return Err(Error::AppendInPlaceUnsupported(
                "append mixes element types in one call; use one element type per append",
            ));
        }
        let oh_addr = self.append_prepare(target)?;

        // Validate the appended bytes against the on-disk datatype.
        let raw = b.raw();
        let new_elems = validate_gathered_append(&self.located[&oh_addr], b)?;
        if new_elems == 0 {
            return Ok(());
        }

        // In SWMR mode, hold to the subset a concurrent reader can follow safely:
        // unfiltered and chunk-aligned, so an append only ever inserts new,
        // not-yet-visible elements and never rewrites a visible trailing chunk
        // out from under a reader. Outside SWMR the session's exclusive lock is
        // what makes that rewrite safe; see `plan_ea_append`.
        if self.swmr_mode {
            let st = &self.located[&oh_addr];
            if st.pipeline.is_some() {
                return Err(Error::SwmrAppendUnsupported(
                    "filtered datasets are not supported for SWMR append",
                ));
            }
            let chunk_elems = st.loc.chunk_elems;
            if chunk_elems == 0
                || st.loc.current_dim % chunk_elems != 0
                || new_elems % chunk_elems != 0
            {
                return Err(Error::SwmrAppendUnsupported(
                    "SWMR append must be chunk-aligned: the current length and the appended \
                     length must both be whole multiples of the chunk length",
                ));
            }
        }

        let (chunk_elems, elem_bytes, full_batch_elems) = {
            let st = &self.located[&oh_addr];
            (
                st.loc.chunk_elems.max(1),
                st.element_size.get() as u64,
                self.batch_elems(st.loc.chunk_bytes, st.loc.chunk_elems.max(1)),
            )
        };
        // Outside SWMR the session holds an exclusive lock, so re-encoding a
        // filtered partial trailing chunk into a fresh allocation and repointing
        // its index element is a change no other reader is crossing; see
        // `plan_ea_append`, which owns that argument and keeps the refusal for
        // the SWMR writer.
        let grow_visible_tail = !self.swmr_mode;

        let mut done = 0u64;
        while done < new_elems {
            // Fill the trailing partial chunk first (so later batches start
            // chunk-aligned and never rewrite it again), then whole-chunk batches.
            // That first batch is the only one that can re-encode a filtered
            // trailing chunk; every batch after it starts on a boundary.
            let current_dim = self.located[&oh_addr].loc.current_dim;
            let to_boundary = (chunk_elems - current_dim % chunk_elems) % chunk_elems;
            let take = (new_elems - done).min(to_boundary.saturating_add(full_batch_elems));
            let batch =
                &raw[(done * elem_bytes).to_usize()?..((done + take) * elem_bytes).to_usize()?];

            // Read/plan phase (immutable borrows only, nothing published yet), then
            // the ordered, fsync-barriered write phase — both shared with
            // `Dataset::append` through the chunk-index engine. `EditStore` borrows
            // only the image-carrying fields, so `self.located` stays independently
            // borrowable.
            let plan_result = {
                let Self {
                    image,
                    superblock,
                    sb_sig_off,
                    paged,
                    located,
                    sync_policy,
                    ..
                } = self;
                let st = &located[&oh_addr];
                let store = EditStore {
                    image: image.as_mut(),
                    superblock,
                    sb_sig_off: *sb_sig_off,
                    paged: paged.as_mut(),
                    // Planning only reads; nothing is allocated here.
                    free: None,
                    sync_policy: *sync_policy,
                };
                plan_ea_append(
                    &store,
                    &st.loc,
                    &st.datatype,
                    &st.spatial,
                    st.element_size,
                    st.pipeline.as_ref(),
                    grow_visible_tail,
                    batch,
                    take,
                    st.fill.pattern(st.element_size),
                )
            };
            let plan = plan_result.map_err(as_inplace_error)?;
            // A persisting file's holes are advertised on disk, so this batch's
            // bytes have to be taken out of the published managers before any of
            // them is written into. Sized on the largest single blob, since the
            // allocator serves each one from a contiguous run: a reserve that
            // cannot hold the biggest is a reserve the biggest goes past. A
            // refusal here (nothing left to draw, or nothing to draw from) is not
            // an error — the append grows the file, as it always did. Placed
            // between the plan and the apply because the manager rewrite it makes
            // publishes a superblock, and nothing may be half-applied across that.
            let largest_blob = plan
                .new_chunk_bytes
                .iter()
                .map(|b| b.len() as u64)
                .max()
                .unwrap_or(0);
            self.reserve_for_immediate_append(largest_blob)?;
            {
                let reuse = self.immediate_reuse_allowed();
                // Two terms, not one. A **paged** session must never be handed
                // `free`: it keeps its space per page type in `PagedEdit`, so
                // that list is the wrong one to draw from and placing a byte out
                // of it would mix a page. Every paged session is a persisting one
                // (`append_prepare` refuses a paged file without persistence), so
                // the first term alone would pick `reserved` for it anyway — and
                // that is exactly the shape issue #261 was: a page-type invariant
                // proved from a refusal in another function. Named here so the
                // safety is by construction, and so deleting that refusal cannot
                // silently turn a paged append loose on the flat list. `reserved`
                // is empty on any session that never reserved, which makes the
                // extra term cost nothing but the reuse it correctly declines.
                let from_reserve = self.persist.is_some() || self.paged.is_some();
                let Self {
                    image,
                    superblock,
                    sb_sig_off,
                    paged,
                    located,
                    free,
                    reserved,
                    sync_policy,
                    ..
                } = self;
                let st = located.get_mut(&oh_addr).expect("dataset located above");
                let mut store = EditStore {
                    image: image.as_mut(),
                    superblock,
                    sb_sig_off: *sb_sig_off,
                    paged: paged.as_mut(),
                    // A persisting or paged session spends only what the reserve
                    // took out of the managers; every other session spends its
                    // in-memory list directly. See `immediate_reuse_allowed`.
                    free: reuse.then_some(if from_reserve { reserved } else { free }),
                    sync_policy: *sync_policy,
                };
                apply_ea_append(&mut store, &mut st.loc, &plan, max_phase)
                    .map_err(as_inplace_error)?;
            }
            if max_phase < 4 {
                // Crash-consistency hook: the caller asked to stop inside the
                // first batch's durability sequence, so there is no next batch.
                // Each phase's own barrier has already issued the writes up to it
                // under either policy, which is what leaves a phase boundary a
                // real one for the tests that stop at it.
                return Ok(());
            }
            done += take;
        }
        Ok(())
    }

    /// Test-only phased in-place append (stops after `max_phase` durability phases)
    /// used by the crash-consistency tests, mirroring `Dataset::append`'s harness.
    #[cfg(test)]
    fn append_inplace_i32_phased(
        &mut self,
        dataset: &str,
        values: &[i32],
        max_phase: u8,
    ) -> Result<(), Error> {
        let mut b = AppendBuilder::new();
        b.append_i32(values);
        self.append_inplace_gathered(
            AppendTarget::Path(&ObjectPathBuf::parse(dataset)),
            &b,
            max_phase,
        )
    }

    /// Whether `target` (an [`append_inplace_gathered`](Self::append_inplace_gathered) dataset path)
    /// or any of its ancestors is named by a staged edit that a later
    /// [`commit`](Self::commit) would relocate, replace, or delete. `create_group`
    /// and group-attribute edits are excluded: they rewrite a group header without
    /// moving a descendant dataset's header or freeing its storage, so they cannot
    /// stale the append geometry cache.
    fn append_conflicts_with_pending(&self, target: &ObjectPathBuf) -> bool {
        let hits = |p: &ObjectPathBuf| target.overlaps(p);
        self.staged.writes.iter().any(|(p, _)| hits(p))
            || self.staged.appends.iter().any(|(p, _)| hits(p))
            || self.staged.deletes.iter().any(&hits)
            || self.staged.copies.iter().any(|(_, dst)| hits(dst))
            || self.staged.cross_copies.iter().any(|(dst, _)| hits(dst))
            || self.staged.dataset_attrs.iter().any(|(p, _)| hits(p))
            || self
                .staged
                .datasets
                .iter()
                .any(|(parent, fd)| target.overlaps(&parent.join(&fd.name)))
    }

    /// What this session has staged at `path`, for a handle that wants to
    /// address a not-yet-committed object by name.
    ///
    /// Only a *creation* answers, and only when it owns the path — when the file
    /// holds no link there, or when this same commit removes the one it does.
    /// Both halves matter:
    ///
    /// - A staged **deletion** on its own hides nothing. The object is in the
    ///   file and still readable, and a lookup that pretended otherwise would
    ///   refuse a read the file can serve.
    /// - A creation **colliding** with a link the commit does not remove is one
    ///   [`refuse_creation_collision`](Self::refuse_creation_collision) turns
    ///   away where it is staged, so this arm is the same rule asked a second
    ///   time rather than a state a caller can reach. Reporting such a creation
    ///   would shadow a live, readable object with a handle that never becomes
    ///   valid, so the file's own object stays the meaning of the name.
    /// - The two together are a **replacement** (issue #305), and the creation
    ///   is what the path names from the moment it is staged, before the commit
    ///   as after. "The two together" is
    ///   [`StagedEdits::deletes_hand_over`]'s rule: the deletion names this path,
    ///   or an ancestor this session builds again.
    ///
    /// Only a path this session staged is reported, and every group above a
    /// staged object must be one the file already holds or one this session
    /// stages at its own path. [`commit`](Self::commit) rejects the whole batch
    /// with [`Error::EditUnsupported`] when a group above a creation is neither,
    /// the state `a/b` is in when [`stage_created_dataset`](Self::stage_created_dataset) stages
    /// `a/b/col` alone.
    pub(crate) fn staged_object(&self, path: &ObjectPathBuf) -> Option<StagedObject> {
        if self.stages_no_creations() {
            return None;
        }
        if path.is_empty() {
            // The root always exists; nothing can stage it.
            return None;
        }
        let kind = if self.staged.dataset_at(path).is_some() {
            StagedKind::Dataset
        } else if self.staged.has_group_at(path) {
            StagedKind::Group
        } else {
            return None;
        };
        let replaces_link = self.staged.deletes_hand_over(path);
        // The file is asked only about a path a creation actually names, so an
        // ordinary open of an object nothing is staged at never pays for this.
        if !replaces_link && self.path_in_file(path) {
            return None;
        }
        Some(StagedObject {
            kind,
            replaces_link,
        })
    }

    /// Refuse a creation at a path something already owns — a link the file
    /// holds and this session does not remove, or a creation this session
    /// already staged — at the call that stages it.
    ///
    /// The commit refuses the same batch (`a link with this name already
    /// exists`), and the file arm raises that refusal's own error so the two
    /// cannot disagree about what a collision is. Refusing here is what lets
    /// [`Group::create_dataset`](crate::Group::create_dataset) and its two
    /// siblings hand back a handle onto the object they stage: a creation that
    /// merely collides is not what its path names — the file's object still is
    /// — so a handle onto it would have addressed the *old* object while
    /// claiming to address the new one.
    ///
    /// A creation this session already stages at the path is refused for the
    /// same reason, one the commit's own refusal cannot stand in for. The staged
    /// set is indexed by path and the index keeps the *first* record at each one
    /// (`or_insert`), so a second creation is staged behind the first and every
    /// handle onto that path — the one this call would hand back included —
    /// answers `shape`, `dtype`, `filters` and `append_staged` from the record
    /// the caller did not just describe. No committed object is misaddressed,
    /// since the commit refuses the batch, but the window between the two is one
    /// where a handle reports another object's geometry as its own.
    ///
    /// Two *group* creations at one path are the exception and stay allowed:
    /// they name one node, the commit builds one group from them, and a handle
    /// onto either addresses that group. Re-staging a group is how a caller adds
    /// attributes or children to one it already staged.
    ///
    /// A [`copy`](Self::copy) destination is still the commit's collision to
    /// refuse: it hands back no handle, so nothing can be misaddressed by it.
    fn refuse_creation_collision(
        &self,
        path: &ObjectPathBuf,
        kind: StagedKind,
    ) -> Result<(), Error> {
        // An empty path identifies the root group, which no link reaches and
        // nothing creates. Both callers reject it with a message of their own
        // before this is reached, so this returns `Ok` and leaves the collision
        // below to a path that identifies a link.
        if path.is_empty() {
            return Ok(());
        }
        // Asked before the file, so that staging twice over a link this session
        // deletes — a replacement, which the file arm below lets through — is
        // caught by the same rule as staging twice over nothing.
        if self.staged.dataset_at(path).is_some()
            || (kind == StagedKind::Dataset && self.staged.has_group_at(path))
        {
            return Err(Error::EditUnsupported(
                "this session already stages an object with this name in the target group; \
                 delete it to withdraw that staging before staging another in its place",
            ));
        }
        if self.staged.deletes_hand_over(path) || !self.path_in_file(path) {
            return Ok(());
        }
        Err(Error::EditUnsupported(
            "a link with this name already exists in the target group",
        ))
    }

    /// Whether the file this session is editing holds a link at `path`, as of
    /// its committed state.
    ///
    /// Resolved against a borrowed slice where the backing can lend one, as the
    /// reader's own path resolution does: this is asked once per by-name lookup
    /// of a path a creation is staged at, so the copy a `Source` read would make
    /// per link block is worth avoiding.
    fn path_in_file(&self, path: &ObjectPathBuf) -> bool {
        match self.image.as_slice() {
            Some(data) => crate::group_v2::resolve_path_any(
                data,
                AccessMode::ReadWrite,
                &self.superblock,
                &path.as_path(),
            )
            .is_ok(),
            None => crate::group_v2::resolve_path_any_from_source(
                &self.image(),
                AccessMode::ReadWrite,
                &self.superblock,
                &path.as_path(),
            )
            .is_ok(),
        }
    }

    /// Whether this session has staged no creations at all, which is the common
    /// case on the by-name lookup path. Checked before the path is owned, so a
    /// lookup of an object already in the file returns on this flag alone.
    fn stages_no_creations(&self) -> bool {
        self.staged.groups.is_empty() && self.staged.datasets.is_empty()
    }

    /// How many times a commit has consumed this session's staged set, rather
    /// than handing it back to it. See the
    /// [`staged_generation`](Self::staged_generation) field for the rule, and
    /// [`crate::Error::StagingWithdrawn`] for what a handle does with it.
    pub(crate) fn staged_generation(&self) -> u64 {
        self.staged_generation
    }

    /// What a staged dataset at `path` says about itself, for the introspection
    /// a handle onto it can answer before the commit writes any bytes.
    ///
    /// Cloned out of the staged [`FlatDataset`], which is the whole record of
    /// what the commit will write, so these answers cannot drift from the
    /// dataset that lands. Answers only for a creation that owns its path, on
    /// the terms [`staged_object`](Self::staged_object) sets out.
    pub(crate) fn staged_dataset_meta(&self, path: &ObjectPathBuf) -> Option<StagedMeta> {
        if self.staged_object(path)?.kind != StagedKind::Dataset {
            return None;
        }
        let fd = self.staged.dataset_at(path)?;
        Some(StagedMeta {
            datatype: fd.dt.clone(),
            dimensions: fd.ds.dimensions.clone(),
            // Reported the way `Dataset::maxshape` reports an on-disk one: a
            // maximum equal to the current shape is a fixed-shape dataset.
            maxshape: fd.ds.extensible_max_dimensions().map(<[MaxExtent]>::to_vec),
            // The rule the commit applies: chunk options or an extensible shape
            // select chunked storage, anything else contiguous.
            chunked: fd.chunk_options.is_chunked() || fd.maxshape.is_some(),
            filters: fd
                .chunk_options
                .filters
                .iter()
                .map(|f| (f.kind.filter_id(), f.optional))
                .collect(),
        })
    }

    /// The direct children `parent` gains from this session's staged creations:
    /// the groups first and then the datasets, each in the order it was staged.
    ///
    /// `parent` may itself be staged, which is what lets a listing on a
    /// not-yet-committed group report its members. Only children the session
    /// named answer, for the reason [`staged_object`](Self::staged_object)
    /// gives.
    ///
    /// Unlike [`staged_object`](Self::staged_object) this does not ask the file
    /// whether each name is already taken: the caller is enumerating a group and
    /// already holds its on-disk links, so `replaces_link` is all it needs to
    /// apply the same rule without a path resolution per child.
    pub(crate) fn staged_children(&self, parent: &ObjectPathBuf) -> Vec<StagedChild> {
        if self.stages_no_creations() {
            return Vec::new();
        }
        // Whether this group is itself a replacement staged here — its on-disk
        // links go with the object being removed, so every creation under it
        // owns its name — and which links directly under it are removed by
        // name: the two ways a creation here is a replacement rather than a
        // collision. A deletion of the group that this session does *not* build
        // again is neither: the commit refuses that batch, so the file's own
        // children still own their names (see [`StagedEdits::deletes_hand_over`]).
        let base_deleted =
            self.staged.deletes_hand_over(parent) && self.staged.has_group_at(parent);
        let deleted_here: HashSet<&str> = self
            .staged
            .deletes
            .iter()
            .filter_map(|d| d.name_under(parent))
            .map(LinkNameBuf::as_str)
            .collect();

        let mut seen: HashSet<&str> = HashSet::new();
        let mut out: Vec<StagedChild> = Vec::new();
        for path in &self.staged.groups {
            if let Some(name) = path.name_under(parent) {
                let name = name.as_str();
                if seen.insert(name) {
                    out.push(StagedChild {
                        name: name.to_string(),
                        kind: StagedKind::Group,
                        replaces_link: base_deleted || deleted_here.contains(name),
                    });
                }
            }
        }
        for (p, fd) in &self.staged.datasets {
            if p == parent && seen.insert(fd.name.as_str()) {
                out.push(StagedChild {
                    name: fd.name.as_str().to_string(),
                    kind: StagedKind::Dataset,
                    replaces_link: base_deleted || deleted_here.contains(fd.name.as_str()),
                });
            }
        }
        out
    }

    /// Stage a new (empty) group at `path`, created on the next
    /// [`commit`](Self::commit). The parent must already exist or be created in
    /// the same session; populate the group with datasets via
    /// [`stage_created_dataset`](Self::stage_created_dataset) using a path under it.
    ///
    /// # Errors
    ///
    /// Returns [`Error::EditUnsupported`] if `path` is [`ObjectPathBuf::root`], which no link
    /// reaches, if a live buffered appender covers `path`, or if the name is taken on the terms
    /// [`refuse_creation_collision`](Self::refuse_creation_collision) sets out.
    pub fn create_group(&mut self, path: &ObjectPathBuf) -> Result<(), Error> {
        // Nothing links to the root group, and every file has one, so there is
        // no link here to create. Rejected here, as a deletion of the root is:
        // a creation staged at the root would mark it a node this commit builds
        // fresh, until the commit rejected the batch.
        if path.is_empty() {
            return Err(Error::EditUnsupported("cannot create the root group"));
        }
        self.refuse_if_claimed(path)?;
        self.refuse_creation_collision(path, StagedKind::Group)?;
        self.staged.push_group(path.clone());
        Ok(())
    }

    /// Stage an attribute add or replacement on a group, applied on the next
    /// [`commit`](Self::commit).
    ///
    /// `path` identifies the group to edit, and [`ObjectPathBuf::root`] the root group. The
    /// group may already exist or may be created earlier in the same session
    /// with [`create_group`](Self::create_group). Attributes — fixed-size or
    /// variable-length — are stored compactly in the rebuilt group header while
    /// they fit it, and in a fractal heap past that, on the terms
    /// [`plan_attr_ops`] sets out.
    pub fn set_group_attr(
        &mut self,
        path: &ObjectPathBuf,
        name: &str,
        value: AttrValue,
    ) -> Result<(), Error> {
        self.refuse_if_claimed(path)?;
        self.staged.group_attrs.push((
            path.clone(),
            AttrOp::Set {
                name: name.to_string(),
                value,
            },
        ));
        Ok(())
    }

    /// Stage removal of a compact attribute from a group, applied on the next
    /// [`commit`](Self::commit).
    ///
    /// `path` identifies the group to edit, and [`ObjectPathBuf::root`] the root group. The
    /// named attribute must exist in the committed group state after any earlier
    /// staged attribute operations for the same group have been applied.
    pub fn remove_group_attr(&mut self, path: &ObjectPathBuf, name: &str) -> Result<(), Error> {
        self.refuse_if_claimed(path)?;
        self.staged.group_attrs.push((
            path.clone(),
            AttrOp::Remove {
                name: name.to_string(),
            },
        ));
        Ok(())
    }

    /// Stage an attribute add or replacement on an **existing dataset**, applied on
    /// the next [`commit`](Self::commit).
    ///
    /// `path` names the dataset to edit. Attributes — fixed-size or
    /// variable-length — are stored in the rebuilt dataset header
    /// while they fit it, and in a fractal heap past that, on the terms
    /// [`plan_attr_ops`] sets out. Applying it relocates the dataset's object
    /// header (the header is rewritten and its single naming link repointed; the
    /// dataset's data and chunk index stay in place), so it is supported only when
    /// the dataset has a **single hard link**. To set attributes on a dataset being *created* in this
    /// session, use the builder's [`set_attr`](crate::DatasetBuilder::set_attr)
    /// instead.
    pub fn set_dataset_attr(
        &mut self,
        path: &ObjectPathBuf,
        name: &str,
        value: AttrValue,
    ) -> Result<(), Error> {
        self.refuse_if_claimed(path)?;
        self.staged.dataset_attrs.push((
            path.clone(),
            AttrOp::Set {
                name: name.to_string(),
                value,
            },
        ));
        Ok(())
    }

    /// Stage removal of a compact attribute from an **existing dataset**, applied on
    /// the next [`commit`](Self::commit).
    ///
    /// `path` names the dataset to edit; the named attribute must exist in the
    /// committed dataset state after any earlier staged attribute operations for the
    /// same dataset have been applied. Like [`set_dataset_attr`](Self::set_dataset_attr)
    /// it relocates the dataset header and requires a single hard link.
    pub fn remove_dataset_attr(&mut self, path: &ObjectPathBuf, name: &str) -> Result<(), Error> {
        self.refuse_if_claimed(path)?;
        self.staged.dataset_attrs.push((
            path.clone(),
            AttrOp::Remove {
                name: name.to_string(),
            },
        ));
        Ok(())
    }

    /// Stage removal of the link at `path` (the HDF5 `H5Ldelete`), applied on the
    /// next [`commit`](Self::commit). The link's object — and, for a group, its
    /// whole subtree — becomes unreachable. The bytes it occupied are returned to
    /// this session's free list (issue #21): a later commit reuses them for new
    /// objects instead of growing the file, and if a freed run reaches
    /// end-of-file the file is truncated. Contiguous and chunked datasets (their
    /// chunk index and chunk data blocks) and whole group subtrees are all
    /// reclaimed. Reclaim is best-effort — an object whose blocks this engine
    /// cannot enumerate exhaustively (variable-length global-heap storage, dense
    /// attribute/link heaps, a version 2 B-tree chunk index) is left as dead
    /// bytes rather than risk freeing a region that is still in use. Freed space is
    /// reused within the open session; for a file created with
    /// `H5Pset_file_space_strategy(persist = true)` it is also recorded on disk so
    /// it survives reopen, otherwise it is forgotten
    /// on close. After reuse, an object reference to a deleted object may resolve
    /// to an unrelated object (deleting a referenced object is undefined in HDF5).
    ///
    /// The path must exist. The link's parent group must itself be editable in
    /// place (compact links, single-chunk header); the target being removed has
    /// no such restriction.
    ///
    /// # Replacing an object
    ///
    /// Deleting a path and creating a new object at that same path in one commit
    /// is a *replacement*, and is accepted (issue #305): the removal is applied
    /// before the addition, and the commit's single superblock write publishes
    /// both, so the path is occupied at every instant a crash could interrupt.
    /// The new object need not resemble the old one — a dataset may replace a
    /// group, or the reverse — and replacing a group discards its whole subtree
    /// rather than inheriting it.
    ///
    /// Everything this commit adds *below* a replaced path lands in the
    /// replacement, whatever order the calls were made in: staging
    /// `create_dataset("g/x")` and then replacing `g` puts `x` in the new group,
    /// not the one being removed. A staged edit that could only mean the
    /// *original* is refused instead — a group under the replaced path that this
    /// commit does not itself create, an attribute set on the object being
    /// removed, a value overwrite inside it, or a [`copy`](crate::File::copy)
    /// reading from it.
    ///
    /// See [`Group::delete`](crate::Group::delete) for the public form and a
    /// worked example.
    ///
    /// A deletion may not overlap a staged change at a *different* path
    /// (deleting `/a` while adding `/a/b`, unless `/a` is itself replaced in the
    /// same commit), nor an edit to the object being removed (an attribute set
    /// on it, or a value overwrite of something inside it); split those into
    /// separate commits.
    /// Deleting something this session **staged** and has not committed
    /// withdraws that staging instead: there is no link in the file to unlink,
    /// so the session is left as though the creation had never been made — its
    /// attributes, appends and staged children go with it. When the file *also*
    /// holds a link at the path (the creation was replacing it), the withdrawal
    /// leaves the plain deletion of the file's own object behind.
    ///
    /// The root group, [`ObjectPathBuf::root`], is rejected with [`Error::EditUnsupported`]:
    /// no link reaches it, so there is no link to remove.
    pub fn delete(&mut self, path: &ObjectPathBuf) -> Result<(), Error> {
        // The root is not linked from anywhere, so there is no link to remove —
        // and an empty path is a prefix of every other, so a deletion staged
        // here would make every staged creation in the session look like a
        // replacement of a file object until the commit refused the batch.
        // Refused by name, as creating the root is.
        if path.is_empty() {
            return Err(Error::EditUnsupported(
                "cannot delete the root group; delete its members instead",
            ));
        }
        self.refuse_if_claimed(path)?;
        if self.staged.dataset_at(path).is_some() || self.staged.has_group_at(path) {
            self.refuse_mid_batch()?;
            self.staged.withdraw_at(path);
            // Withdrawing is the whole deletion unless the file holds a link
            // here too — and if it does, a deletion of it may already be staged,
            // which is what made this a replacement in the first place. A second
            // one would be an overlapping deletion the commit refuses.
            if self.staged.deletes_cover(path) || !self.path_in_file(path) {
                return Ok(());
            }
        }
        self.staged.deletes.push(path.clone());
        Ok(())
    }

    /// Stage a deep copy of the object at `src` to a new link at `dst` (the HDF5
    /// `H5Ocopy`), applied on the next [`commit`](Self::commit). The source — a
    /// dataset or a whole group subtree — is duplicated: fresh copies of every
    /// object's data and header are written, internal links and the contiguous
    /// data address are repointed to the copies, and a link named by `dst`'s last
    /// component is added to `dst`'s parent group. The original is untouched.
    ///
    /// The copy reflects the file's on-disk state at commit time. `src` must
    /// exist and `dst` must not (and may not lie inside `src`). A chunked (and
    /// filtered) dataset is copied with its chunk payloads and filter pipeline
    /// preserved byte-for-byte (the index is rebuilt at the new location, so a
    /// source using a B-tree-v1 or implicit index is reproduced with an equivalent
    /// v4 index). The source subtree must otherwise be copyable in place: compact
    /// links and attributes, single-chunk headers, and a chunk index this engine
    /// can enumerate (a version-2 B-tree, or a sparse/unallocated chunk grid, is
    /// refused) — otherwise `commit` reports [`Error::EditUnsupported`].
    ///
    /// A *contiguous* dataset whose storage was never allocated is copied as the
    /// storage it has — none — rather than as the fill value a read answers with,
    /// which is what keeps a schema-only source from arriving materialized at the
    /// size its shape declares. A dataset storing its elements in external files
    /// (`H5Pset_external`) carries that same empty storage over data this crate
    /// does not read, and is refused by name so the two do not share an answer
    /// (issue #336).
    pub fn copy(&mut self, src: &ObjectPathBuf, dst: &ObjectPathBuf) -> Result<(), Error> {
        // Both ends matter: the source is read and the destination is written,
        // and a commit relocates headers along either path.
        self.refuse_if_claimed(src)?;
        self.refuse_if_claimed(dst)?;
        self.staged.copies.push((src.clone(), dst.clone()));
        Ok(())
    }

    /// Stage a deep copy of the object at `src` in another open file `source` to a
    /// new link at `dst` in this file — a *cross-file* HDF5 `H5Ocopy` — applied on
    /// the next [`commit`](Self::commit). Like [`copy`](Self::copy) but the source
    /// lives in a separate, independently-opened [`File`](crate::File) reader
    /// rather than the file being edited.
    ///
    /// The source — a dataset or a whole group subtree — is duplicated faithfully:
    /// fresh, byte-identical copies of every object's header and data are appended
    /// to this file, internal links repointed, and a link named by `dst`'s last
    /// component added to `dst`'s parent group (which must already exist or be
    /// created earlier in this session). Both files are left otherwise untouched;
    /// the destination only changes on `commit`.
    ///
    /// Unlike the same-file [`copy`](Self::copy), the source is read **eagerly**
    /// here (the `source` borrow need not outlive the call), so this returns
    /// `Result`: the source subtree is resolved, validated, and read out before
    /// returning, and only an already-validated copy is queued for `commit`.
    ///
    /// # Errors
    ///
    /// Returns [`Error::EditUnsupported`] if the copy cannot be reproduced exactly
    /// in another file. Because the copy is byte-for-byte verbatim, anything that
    /// embeds a *source-file* absolute address is refused (it would dangle here):
    /// **variable-length** or **reference** datasets and attributes (including a
    /// chunked dataset whose elements are variable-length or references, whose
    /// chunk payloads embed such addresses), and any **shared header message** (a
    /// committed datatype, or an SOHM-shared dataspace, fill value, or filter
    /// pipeline). As with [`copy`](Self::copy) a chunked/filtered source is copied
    /// with its chunk payloads and pipeline preserved (index rebuilt at the new
    /// location); the source must use compact links and attributes, single-chunk
    /// version-2 headers, and a chunk index this engine can enumerate (a
    /// version-2 B-tree, or a sparse chunk grid, is refused). The
    /// `source` must be a buffered file ([`File::open`](crate::File::open) or
    /// [`File::from_bytes`](crate::File::from_bytes), not
    /// [`open_streaming`](crate::File::open_streaming)) using 8-byte offsets and no
    /// userblock, and `src` must exist in it and not be the root group.
    ///
    /// # Note
    ///
    /// The source is read under the mode its own file is open under, read-only for
    /// every source this copy accepts, so an object header message of a type this
    /// crate cannot name that a decoder with write access must understand is read
    /// past. The copy reproduces an object's header records verbatim, so such a
    /// message on `src`'s own header reaches the destination. `H5Ocopy` copies a
    /// message of an unknown type into the destination header the same way, since a
    /// message without a `copy_file` callback is copied raw (`H5Ocopy.c`,
    /// HDF5 2.2.0).
    pub fn copy_from(
        &mut self,
        source: &crate::reader::File,
        src: &ObjectPathBuf,
        dst: &ObjectPathBuf,
    ) -> Result<(), Error> {
        // The source bytes must be addressable: a streaming file is refused.
        let src_data = source.in_memory_image().ok_or(Error::EditUnsupported(
            "cross-file copy requires a buffered source file (File::open or File::from_bytes), not a streaming one",
        ))?;
        let src_sb = source.superblock();
        if src_sb.offset_size != OFFSET_SIZE || src_sb.length_size != LENGTH_SIZE {
            return Err(Error::EditUnsupported(
                "cross-file copy requires the source file to use 8-byte offsets and lengths",
            ));
        }
        if !source.base_address().is_zero() {
            return Err(Error::EditUnsupported(
                "cross-file copy requires the source file to have no userblock (base address 0)",
            ));
        }

        if src.is_empty() {
            return Err(Error::EditUnsupported("cannot copy the root group"));
        }
        if dst.is_empty() {
            return Err(Error::EditUnsupported("copy destination path is empty"));
        }

        let src_addr = crate::group_v2::resolve_path_any(
            src_data,
            source.access_mode(),
            src_sb,
            &src.as_path(),
        )?;
        // Read (and foreign-address-screen) the whole subtree now, while `source`
        // is borrowed; the owned tree carries every byte the commit will write. The
        // source is gated to base 0 above, so its stored addresses are absolute.
        let tree = Self::read_copy_subtree(
            &BytesSource::new(src_data),
            source.access_mode(),
            src_addr,
            0,
            true,
            BaseAddress::ZERO,
        )?;
        self.refuse_if_claimed(dst)?;
        self.staged.cross_copies.push((dst.clone(), tree));
        Ok(())
    }

    /// Apply all staged additions and deletions to the file in place and flush.
    ///
    /// Appends each new dataset (its data — a contiguous blob, or the chunk data
    /// and index for a chunked/filtered dataset — plus its object header) and
    /// each new group, then appends rewritten object headers for every touched
    /// group and its ancestors up to the root (omitting any deleted links), then
    /// repoints the superblock at the new root. On success the staged set is
    /// cleared and the session can be reused. On any [`Error::EditUnsupported`]
    /// the file on disk is left untouched: the checks that raise it — including
    /// each dataset's filter-pipeline and chunk-geometry validation — all run
    /// before the first byte is written. Should a later step fail mid-apply (an
    /// I/O error, or a residual build error), the superblock — repointed last —
    /// still names the prior root, so the file stays valid and the appended bytes
    /// are unreferenced slack.
    ///
    /// **A refused commit costs the session nothing it had staged.** The
    /// staged set is whole afterwards — so the batch can be committed again,
    /// and refuses again identically rather than applying the part of itself
    /// the refusal was not about — and the free regions the attempt drew from
    /// are given back, so an attempt that never became visible costs the
    /// session neither staged work nor reusable space (issue #316; see
    /// [`StagedEdits`] and [`FreeSnapshot`]).
    ///
    /// A failure *past* the first write is the other case, and it clears the
    /// staged set rather than restoring it: the file is valid but some of the
    /// batch is in it as slack, and a later `commit` must not re-issue the rest
    /// as if nothing had happened.
    ///
    /// **What the file holds is a separate promise: a commit that fails before
    /// its repoint leaves every dataset reading what it read before.** "Slack"
    /// describes all but one of the edits such a commit applies. The exception
    /// is a same-length value overwrite, which writes straight over the
    /// dataset's existing data block and so is live the moment it lands, with
    /// no repoint to withhold; the prior bytes are written back over every one
    /// of them on the way out (issue #344).
    ///
    /// Past the repoint the promise is the other one, and it has to be: the
    /// commit has published, so its overwrites are the file's values and are
    /// left standing. A failure there — `repoint_stored_references` is the step
    /// that can produce one — returns an error over a file that holds the whole
    /// batch, which is what
    /// `a_commit_that_fails_past_its_repoint_keeps_the_value_it_published`
    /// pins.
    ///
    /// # Errors
    ///
    /// Returns [`Error::CommitPartiallyApplied`] — carrying the write failure,
    /// not the refusal — when that restore itself failed, which is the one
    /// refusal after which a dataset the batch named may hold either value.
    /// Every other refusal leaves them all as they were.
    pub fn commit(&mut self) -> Result<(), Error> {
        let snapshot = self.snapshot_free();
        self.publish_attempted = false;
        // Cleared at entry, like `superseded_heaps` and for the same reason: a
        // previous attempt that unwound past this function — a panic caught by
        // the session lock, which recovers from poisoning — would otherwise
        // leave entries behind, and this commit's rollback would replay bytes
        // captured before it over whatever lives at those addresses now. The
        // clear on the way out is a separate duty: it stops a journal being
        // held across the idle time between commits.
        self.inplace_undo.clear();
        // The staged set comes out for the duration of the attempt and goes back
        // if the attempt refuses, so a refusal costs the session no staged work
        // (issue #316). What goes back is whatever the attempt did not take: its
        // preflight only reads, and its apply phase takes the whole set in one
        // move at the point of no return, so a preflight refusal restores every
        // edit and a failure past that point restores nothing.
        let mut staged = std::mem::take(&mut self.staged);
        let mut result = self.commit_inner(&mut staged);
        if result.is_err() && !self.publish_attempted {
            // The three legs of the rollback: the file's values, the free
            // lists, and the staged set. Nothing observes the order — the first
            // touches the file and the others only in-memory state, inside one
            // synchronous call — so it is written the way it reads: put the
            // file back, then re-offer what the attempt drew from (issue #344).
            let restored = self.undo_inplace_writes();
            self.restore_free(snapshot);
            self.staged = staged;
            // The set is back, so nothing this session staged has been consumed
            // and every handle onto one of those creations is still pending.
            if let Err(restore) = restored {
                // Both are worth carrying: the refusal is what the caller has to
                // fix, and the write failure is what proves the file changed.
                let refusal = result.expect_err("only a failed attempt is rolled back");
                result = Err(Error::CommitPartiallyApplied {
                    refusal: Box::new(refusal),
                    restore: Box::new(restore),
                });
            }
        } else {
            // The set is gone — published, or in the file as slack a later
            // commit must not re-issue — so a handle onto one of its creations
            // stops being pending and addresses the file from here on.
            self.staged_generation += 1;
        }
        // For the paths that took no rollback — a commit that succeeded, whose
        // overwrites are now the file's values, and one that failed with the
        // publish already issued, where undoing anything is unsound. A rollback
        // drained the journal on its way through, so it owes nothing here.
        self.inplace_undo.clear();
        result
    }

    /// The free lists as they stand now, for [`commit`](Self::commit) to restore
    /// if its apply loop draws from them and then fails.
    fn snapshot_free(&self) -> FreeSnapshot {
        FreeSnapshot {
            free: self.free.clone(),
            reserved: self.reserved.clone(),
            paged: self
                .paged
                .as_ref()
                .map(|pg| (pg.meta.clone(), pg.raw.clone())),
            vl_overwrite_heaps: self.vl_overwrite_heaps.clone(),
        }
    }

    /// Put the free lists back as [`snapshot_free`](Self::snapshot_free) found
    /// them. Called only for a commit that failed before publishing anything, so
    /// every region it restores is dead again.
    ///
    /// That rests on nothing the live tree reaches naming a span the attempt
    /// allocated, which takes two rules rather than one. Every object a commit
    /// *places* is unreachable until the superblock repoint; and the one write
    /// that does not wait for the repoint — a same-length value overwrite — is
    /// planned only over bytes that carry no allocated address, because
    /// [`prepare_write`](Self::prepare_write) relocates a staged
    /// variable-length overwrite instead of writing it in place, which is
    /// exactly the counterexample (issue #321).
    ///
    /// [`undo_inplace_writes`](Self::undo_inplace_writes) has nevertheless run
    /// by the time this does, so the file's values are back before its regions
    /// are offered again (issue #344).
    fn restore_free(&mut self, snapshot: FreeSnapshot) {
        self.vl_overwrite_heaps = snapshot.vl_overwrite_heaps;
        self.free = snapshot.free;
        self.reserved = snapshot.reserved;
        if let (Some(pg), Some((meta, raw))) = (self.paged.as_mut(), snapshot.paged) {
            pg.meta = meta;
            pg.raw = raw;
        }
    }

    /// Overwrites `raw` at `at`, keeping the bytes it replaces so a commit that
    /// fails before its repoint can put them back.
    ///
    /// The read covers exactly the range about to be written, which the write
    /// preflight sized and bounds-checked against the on-disk dataset — a
    /// same-length overwrite by construction.
    fn write_inplace_journaled(&mut self, at: u64, raw: &[u8]) -> Result<(), Error> {
        let prior = self.image().read_exact_at(at, raw.len())?;
        self.inplace_undo.push((at, prior));
        self.write_at(at, raw)
    }

    /// Put back the values the refused commit's same-length in-place overwrites
    /// replaced, and drop the journal.
    ///
    /// The third leg of a refused commit's rollback, beside
    /// [`restore_free`](Self::restore_free) and the staged set: those two give
    /// back what the attempt *consumed*, this gives back what it *changed*. It
    /// is the only one of the three that has to touch the file, because a
    /// same-length overwrite is the only edit a commit writes over bytes the
    /// live root already reaches (issue #344).
    ///
    /// # Errors
    ///
    /// Every entry is attempted even after one fails, so the file is restored as
    /// far as it can be, and the first failure met is what comes back. That is the
    /// one outcome leaving values from a batch that was refused, which is why
    /// [`commit`](Self::commit) reports it as
    /// [`Error::CommitPartiallyApplied`] rather than as the refusal.
    fn undo_inplace_writes(&mut self) -> Result<(), Error> {
        let journal = core::mem::take(&mut self.inplace_undo);
        if journal.is_empty() {
            return Ok(());
        }
        let mut failed = None;
        // Newest first, and load-bearing rather than conventional. The commit
        // refuses to overwrite the same *path* twice, which is not the same as
        // the same dataset: an object reachable through two hard links has two
        // paths and one data block, and a same-length overwrite through either
        // is allowed precisely because it rewrites the block every link sees.
        // Two such writes in one commit therefore journal the same address
        // twice, the second entry holding what the first write put there.
        // Replaying oldest-first would finish on that value — one from the very
        // batch being rolled back.
        for (at, prior) in journal.into_iter().rev() {
            if let Err(e) = self.write_at(at, &prior) {
                failed.get_or_insert(e);
            }
        }
        match failed {
            Some(e) => Err(e),
            // A restore that is not a barrier can be gathered and reordered
            // past a later write, which is the whole hazard write gathering
            // introduced (issue #288); order it here instead. The data-only
            // barrier is the one that fits, because putting bytes back into
            // blocks that already existed moves no end-of-file — it is a
            // narrower barrier than the `barrier()` the commit itself uses
            // after these writes, deliberately.
            None => self.barrier_data(),
        }
    }

    fn commit_inner(&mut self, staged: &mut StagedEdits) -> Result<(), Error> {
        if staged.is_empty() {
            return Ok(());
        }

        // An attempt that failed partway may have recorded superseded collections
        // it never freed; this one must not free them on its behalf, since it is
        // not the commit that replaced what they hold.
        self.superseded_heaps.clear();
        // Drop every heap-collection provenance record this batch could falsify,
        // before any of them is read below.
        self.invalidate_heap_provenance(staged);

        // A paged file (`H5F_FSPACE_STRATEGY_PAGE`) that does not persist its free
        // space has no on-disk record of which pages hold metadata and which hold
        // raw data, so this commit could not keep the two segregated and would
        // silently degrade the paging. Refuse up front, before any writes, exactly
        // as the bounded backend does. A paged *persisting* file is committed
        // through the page-aware tail below (issue #198).
        if self.paged.is_some() && self.persist.is_none() {
            return Err(Error::EditUnsupported(
                "committing an edit to a paged file (H5F_FSPACE_STRATEGY_PAGE) requires \
                 persisted free space; recreate the file with \
                 with_file_space_strategy(FileSpaceStrategy::Page, true, ..) to edit it in place",
            ));
        }

        // Invalidate the in-place-append geometry cache before doing any work. A
        // commit that reaches here rewrites and relocates object headers, frees
        // vacated regions into `self.free`, and may truncate the file — any of
        // which can leave a cached `Located` pointing at a moved header or into a
        // now-free-eligible region. Clearing at *entry* (rather than the success
        // tail) means a later failure — including one after the durable root flip,
        // which leaves the session reusable — never strands a stale cache. The
        // no-op fast return above does no such work, so it keeps the cache. The
        // The next in-place append re-locates against the fresh file.
        self.located.clear();
        self.resolved.clear();
        // Past this point the commit may relocate object headers, so every address
        // a caller captured earlier is suspect; see `committed`.
        self.committed = true;

        // On a file with a userblock, stored addresses are relative to this base
        // and the editor converts at every disk boundary (read `stored + base`,
        // write `file_offset - base`). Userblock support covers value overwrites,
        // additions of contiguous and chunked/filtered datasets, in-place and
        // relocating overwrites of every layout (chunked, contiguous, compact) with
        // reclaim, object deletion (with base-aware subtree reclaim), object copy
        // (in-file, and cross-file into a userblock destination), group creation,
        // and compact group attributes. Cross-file copy still requires a base-0
        // *source* (see [`copy_from`](Self::copy_from)).
        let base = self.superblock.base_address;

        // --- Preflight value overwrites (`write_dataset`) before any write, under
        // the same all-or-nothing contract as additions. Each is resolved,
        // validated (datatype and shape must match the on-disk dataset exactly),
        // and classified: a same-length contiguous overwrite is applied straight
        // in place (no header rewrite, no superblock flip), while a resize or
        // compact rewrite relocates the header and is staged against its parent
        // group so the commit below rebuilds it and patches the link. ---
        let mut inplace_writes: Vec<(u64, OverwriteBytes)> = Vec::new();
        let mut moving_writes: Vec<(ObjectPathBuf, LinkNameBuf, u64, MovingWrite)> = Vec::new();
        let mut write_targets: Vec<ObjectPathBuf> = Vec::new();
        // The file-wide hard-link count, computed lazily the first time a commit
        // relocates a header: such a write moves the object's header and patches
        // only the one parent link that names it, so an object reachable through
        // more than one hard link would have its other links left pointing at the
        // stale header. Refuse that rather than silently diverge the aliases (a
        // same-length in-place overwrite is unaffected — it rewrites the shared
        // data block, which every link sees).
        //
        // Read by four places, which is every way a header moves: a relocating
        // overwrite, a staged append, a dataset attribute edit, and — since
        // issue #327 — a rebuilt group, which had the same defect and none of
        // the guard.
        let mut incoming_links: Option<Option<HashMap<u64, u32>>> = None;
        for (full, fd) in &staged.writes {
            // A path named twice in one commit would write it twice (and double-
            // free a resized extent); require separate commits.
            if write_targets.contains(full) {
                return Err(Error::EditUnsupported(
                    "the same dataset is overwritten twice in one commit; use separate commits",
                ));
            }
            let addr = crate::group_v2::resolve_path_any_from_source(
                &self.image(),
                AccessMode::ReadWrite,
                &self.superblock,
                &full.as_path(),
            )?;
            match Self::prepare_write(&self.image(), addr, fd, base, full)? {
                WritePlan::InPlace { data_addr, bytes } => {
                    inplace_writes.push((data_addr, bytes));
                }
                // A chunked in-place overwrite never carries staging — a staged
                // one is a `ChunkPayload::Deferred`, which relocates.
                WritePlan::InPlaceChunks { writes } => inplace_writes.extend(
                    writes
                        .into_iter()
                        .map(|(at, raw)| (at, OverwriteBytes::ready(raw))),
                ),
                WritePlan::Moving(mw) => {
                    // A relocating overwrite rewrites the dataset's header and data
                    // address. Every variant is base-aware on a userblock file: the
                    // chunked one rebuilds the chunk blob with stored addresses and
                    // reclaims the old storage base-relative, the contiguous one
                    // stores the relocated data address base-relative (and frees the
                    // old extent at its absolute offset), and the compact one carries
                    // its data inline. The parent link to the rewritten header is
                    // patched base-relative below.
                    //
                    // A relocating overwrite is safe only when this is the
                    // dataset's sole hard link. Compute the link graph once.
                    let counts = incoming_links
                        .get_or_insert_with(|| self.count_incoming_hard_links())
                        .as_ref();
                    match counts.and_then(|c| c.get(&addr)) {
                        Some(&1) => {}
                        _ => {
                            return Err(Error::EditUnsupported(
                                "overwriting a dataset that resizes or relocates its header is \
                                 only supported when it has a single hard link",
                            ));
                        }
                    }
                    let Some((parent, leaf)) = full.split_leaf() else {
                        return Err(Error::EditUnsupported("cannot overwrite the root group"));
                    };
                    moving_writes.push((parent, leaf, addr, mw));
                }
            }
            write_targets.push(full.clone());
        }

        // --- Preflight appends (`append_dataset`) under the same all-or-nothing,
        // single-hard-link contract. Each plans a relocating append — existing
        // chunk data stays in place; the appended (and any rewritten trailing)
        // chunks and a rebuilt Extensible-Array index are staged, and the whole is
        // treated like a relocating overwrite of the dataset's header (staged
        // against its parent group so the commit patches the link). A zero-length
        // append is a no-op and is dropped here. ---
        for (full, ab) in &staged.appends {
            let Some((parent, leaf)) = full.split_leaf() else {
                return Err(Error::AppendUnsupported("cannot append to the root group"));
            };
            if ab.raw.is_empty() {
                continue; // nothing to append
            }
            // A dataset overwritten or appended earlier in this commit would be
            // planned against a stale header and its old storage double-freed;
            // require separate commits.
            if write_targets.contains(full) {
                return Err(Error::AppendUnsupported(
                    "the same dataset is edited more than once in one commit; use separate commits",
                ));
            }
            let addr = crate::group_v2::resolve_path_any_from_source(
                &self.image(),
                AccessMode::ReadWrite,
                &self.superblock,
                &full.as_path(),
            )?;
            let mw = Self::prepare_append(&self.image(), addr, ab, base)?;
            // A relocating append moves the dataset's object header and patches only
            // the one parent link that names it, so it is safe only when this is the
            // dataset's sole hard link (same rule as a relocating overwrite).
            let counts = incoming_links
                .get_or_insert_with(|| self.count_incoming_hard_links())
                .as_ref();
            match counts.and_then(|c| c.get(&addr)) {
                Some(&1) => {}
                _ => {
                    return Err(Error::AppendUnsupported(
                        "appending relocates the dataset header; only supported when it \
                         has a single hard link",
                    ));
                }
            }
            moving_writes.push((parent, leaf, addr, mw));
            write_targets.push(full.clone());
        }

        // --- Preflight dataset attribute edits (`set_dataset_attr` /
        // `remove_dataset_attr`) under the same all-or-nothing, single-hard-link
        // contract. Each gathers the dataset's verbatim object-header region,
        // applies the compact attribute ops to it, and stages a relocating
        // `AttrEdit` header rewrite against the parent group — like a value
        // overwrite, but the data-layout message (and thus the chunk data and index)
        // is preserved verbatim, so only the header moves. ---
        if !staged.dataset_attrs.is_empty() {
            // Collect the ops per dataset in first-seen path order, so multiple edits
            // to one dataset produce a single relocating header rewrite.
            let mut order: Vec<ObjectPathBuf> = Vec::new();
            let mut ops_by_path: HashMap<&ObjectPathBuf, Vec<&AttrOp>> = HashMap::new();
            for (path, op) in &staged.dataset_attrs {
                if !ops_by_path.contains_key(path) {
                    order.push(path.clone());
                }
                ops_by_path.entry(path).or_default().push(op);
            }
            for full in order {
                let ops = ops_by_path.remove(&full).unwrap();
                let Some((parent, leaf)) = full.split_leaf() else {
                    return Err(Error::EditUnsupported(
                        "cannot set a dataset attribute on the root group; use set_group_attr",
                    ));
                };
                // A dataset already overwritten or appended in this commit would be
                // planned against a stale header; require separate commits.
                if write_targets.contains(&full) {
                    return Err(Error::EditUnsupported(
                        "the same dataset is edited more than once in one commit (an attribute \
                         edit plus another edit); use separate commits",
                    ));
                }
                let addr = crate::group_v2::resolve_path_any_from_source(
                    &self.image(),
                    AccessMode::ReadWrite,
                    &self.superblock,
                    &full.as_path(),
                )?;
                // An attribute edit relocates the dataset's object header and patches
                // only the one naming link, so it is safe only when this is the
                // dataset's sole hard link (same rule as a relocating overwrite).
                let counts = incoming_links
                    .get_or_insert_with(|| self.count_incoming_hard_links())
                    .as_ref();
                match counts.and_then(|c| c.get(&addr)) {
                    Some(&1) => {}
                    _ => {
                        return Err(Error::EditUnsupported(
                            "editing a dataset attribute relocates its header; only supported \
                             when it has a single hard link",
                        ));
                    }
                }
                let region = Self::gather_oh_messages(&self.image(), addr, base)?;
                let edits = plan_attr_ops(&self.image(), base, Some(addr), &region, &ops)?;

                moving_writes.push((
                    parent,
                    leaf,
                    addr,
                    MovingWrite::AttrEdit {
                        region: edits.region,
                        attrs: edits.attrs,
                    },
                ));
                write_targets.push(full);
            }
        }

        // Pre-commit object-header addresses this commit rewrites elsewhere, for
        // `InvalidatedAddresses` below: a reference naming one of these keeps an
        // address the commit is vacating (issue #317). Read off the one list every
        // relocating write is staged on, so a write added later cannot be left out
        // of the screen without failing to compile.
        let mut moved_headers: Vec<u64> =
            moving_writes.iter().map(|&(_, _, addr, _)| addr).collect();

        // Fast path: when the only staged edits are same-length in-place
        // overwrites, apply them straight to their data blocks and return without
        // rebuilding any header or flipping the superblock root. The commit's
        // linearization point is the synced data write — there is no tree to
        // repoint, so each overwrite stands alone. (A persisting file takes the
        // same path: no free-space change occurs.)
        //
        // Because this path never rewrites the superblock, it deliberately leaves
        // it untouched — including a pre-existing stale consistency flag (e.g. one
        // left by a crashed SWMR writer). A lone same-length value overwrite does
        // not introduce any inconsistency, so it does not clear one either; an edit
        // that takes the full path below (any header/root change) clears the flag
        // as usual.
        //
        // A staged variable-length overwrite cannot reach here at all: it
        // relocates, so `moving_writes` is non-empty for it. That is enforced
        // where the plan is chosen (`prepare_write`) rather than re-tested here,
        // and it has to hold for a second reason besides the one that made it
        // relocate — such an overwrite places a global heap collection, and an
        // append moves end-of-file, which only the superblock this path leaves
        // untouched records (issue #321).
        debug_assert!(
            inplace_writes.iter().all(|(_, b)| b.vlen.is_none()),
            "a staged variable-length overwrite must relocate, not write in place"
        );
        if moving_writes.is_empty()
            && staged.datasets.is_empty()
            && staged.groups.is_empty()
            && staged.group_attrs.is_empty()
            && staged.deletes.is_empty()
            && staged.copies.is_empty()
            && staged.cross_copies.is_empty()
        {
            // This path's own point of no return, and it takes the staged set
            // for the same reason the main one below does (issue #316): every
            // refusal that applies to these overwrites has already run, and a
            // batch some of which has been written is not one a later `commit`
            // may re-issue. It is also what this path already did, back when the
            // write preflight drained the staged set on its way here.
            //
            // Leaving the set in place would make a half-written batch
            // retryable — these overwrites are same-length writes to fixed
            // addresses, so repeating them is idempotent — but that reads the
            // rule the other way round for one path, on an argument nothing
            // enforces. What the file holds is settled separately and the same
            // way on both paths: a `write_at` that fails partway is undone
            // through the journal below, so the batch is dropped but the
            // dataset's values are the ones it had (issue #344).
            drop(std::mem::take(staged));
            for (data_addr, bytes) in &inplace_writes {
                // Every entry here stages nothing (the guard above), so this
                // hands back the staged bytes unchanged and unallocated.
                let raw = self.resolve_overwrite_bytes(bytes)?;
                self.write_inplace_journaled(*data_addr, &raw)?;
            }
            self.barrier()?;
            return Ok(());
        }

        // --- Plan: build the tree of "dirty" groups (root plus every group on a
        // path to an addition or deletion), validating every target before any
        // write. `add_targets` records the full paths created this commit, used
        // to reject a deletion that overlaps an addition. ---
        let mut nodes: BTreeMap<ObjectPathBuf, Node> = BTreeMap::new();
        nodes.entry(ObjectPathBuf::root()).or_default(); // root is always dirty
        let mut add_targets: Vec<ObjectPathBuf> = Vec::new();
        // Where each in-file `copy` reads from. A copy takes its bytes from the
        // *pre-commit* file, so a source this same commit replaces would copy the
        // object being removed while the replacement lands at the same path — see
        // the delete-staging loop, which refuses that. Cross-file copies are not
        // tracked: their source is in another file, so no path here can name it.
        let mut copy_sources: Vec<ObjectPathBuf> = Vec::new();
        let mut attr_targets: Vec<ObjectPathBuf> = Vec::new();

        // Mark explicitly-created new groups, ensuring their ancestor chain.
        for path in &staged.groups {
            if path.is_empty() {
                return Err(Error::EditUnsupported("cannot create the root group"));
            }
            ensure_ancestors(&mut nodes, path);
            nodes.entry(path.clone()).or_default().is_new = true;
            add_targets.push(path.clone());
        }

        // Make each staged dataset's parent group a node (with its ancestor
        // chain). The datasets themselves stay in the staged set, which the
        // preflight reads but must not empty (issue #316); `datasets_by_group`
        // below is the grouped view both the preflight and the apply loop use.
        for (parent, fd) in &staged.datasets {
            add_targets.push(parent.join(&fd.name));
            ensure_ancestors(&mut nodes, parent);
        }

        // Attach relocating value overwrites (resized contiguous or compact) to
        // their parent group nodes: the new header is written below and the
        // parent's existing link patched to it, like an existing child group.
        for (parent, leaf, old_oh, mw) in moving_writes {
            ensure_ancestors(&mut nodes, &parent);
            nodes
                .entry(parent)
                .or_default()
                .writes
                .push((leaf, old_oh, mw));
        }

        // Stage group attribute edits against their target groups. A target may
        // be a newly-created group from this same commit, but not a copied
        // destination or a dataset being added in the same commit. The ops
        // themselves stay in the staged set and are looked up by path where they
        // are applied, so a refusal below still gives them back (issue #316).
        for (path, _) in &staged.group_attrs {
            ensure_ancestors(&mut nodes, path);
            attr_targets.push(path.clone());
        }

        // Stage copies: validate the source subtree is copyable (read-only),
        // then treat the destination like an addition to its parent group.
        for (src, dst) in &staged.copies {
            if src.is_empty() {
                return Err(Error::EditUnsupported("cannot copy the root group"));
            }
            let Some((parent, leaf)) = dst.split_leaf() else {
                return Err(Error::EditUnsupported("copy destination path is empty"));
            };
            if dst.starts_with(src) {
                return Err(Error::EditUnsupported(
                    "cannot copy an object into itself or its own subtree",
                ));
            }
            let src_addr = crate::group_v2::resolve_path_any_from_source(
                &self.image(),
                AccessMode::ReadWrite,
                &self.superblock,
                &src.as_path(),
            )?;
            // Read the source subtree from this file's own mirror (`cross_file`
            // false: same address space, so verbatim addresses stay valid). On a
            // userblock file the stored addresses are base-relative, so pass this
            // session's base for the read to absolutize them.
            let tree = Self::read_copy_subtree(
                &self.image(),
                AccessMode::ReadWrite,
                src_addr,
                0,
                false,
                base,
            )?;
            copy_sources.push(src.clone());
            add_targets.push(dst.clone());
            ensure_ancestors(&mut nodes, &parent);
            nodes.entry(parent).or_default().copies.push((leaf, tree));
        }

        // Stage cross-file copies: their subtrees were already read out of the
        // source file (with foreign-address screening) when `copy_from` was
        // called, so here they are simply linked into the destination parent like
        // any other addition.
        for (dst, _) in &staged.cross_copies {
            let Some((parent, leaf)) = dst.split_leaf() else {
                return Err(Error::EditUnsupported("copy destination path is empty"));
            };
            add_targets.push(dst.clone());
            ensure_ancestors(&mut nodes, &parent);
            nodes.entry(parent).or_default().cross_copies.push(leaf);
        }

        // Stage deletions: each must exist, must not overlap any other staged
        // change *unless* this commit replaces what it removes, and is recorded
        // against its parent group (which becomes dirty). `deleted_addrs` keeps
        // each removed object's header address so its owned blocks can be
        // reclaimed after the commit lands (issue #21).
        let delete_targets = &staged.deletes;
        let mut deleted_addrs: Vec<u64> = Vec::new();
        for (i, d) in delete_targets.iter().enumerate() {
            let Some((parent, leaf)) = d.split_leaf() else {
                return Err(Error::EditUnsupported("cannot delete the root group"));
            };
            let del_addr = crate::group_v2::resolve_path_any_from_source(
                &self.image(),
                AccessMode::ReadWrite,
                &self.superblock,
                &d.as_path(),
            )?;
            deleted_addrs.push(del_addr);
            // A deletion may overlap other staged work when this commit
            // *replaces* what it removes: an addition names exactly `d`, so the
            // removal and the new object at the same path are one rotation
            // rather than an edit of something being deleted (issue #305).
            let recreated = add_targets.iter().any(|t| t == d);
            // A replacement also requires that every group node at or below `d`
            // is one this commit builds fresh. A node that is *not* new is
            // rebuilt from the old object's on-disk header — the object being
            // replaced — and both ways that lands are wrong, in the two shapes
            // this guard was measured against:
            //
            // * At `d` itself (a group attribute set on a path this commit
            //   replaces with a *dataset*), the node and the replacement share a
            //   `path_addr` key. The replacement's address overwrites the group's,
            //   the parent's link is then patched to the address it already had,
            //   and the rebuilt group header is left orphaned — a commit that
            //   returns `Ok` having **silently discarded** the staged attribute.
            // * Strictly below `d`, the parent is a freshly built region with no
            //   existing link to patch, so `patch_link_target` reports a missing
            //   child link partway through the apply. That leaves a valid file
            //   (the superblock is never repointed) but appends dead bytes and
            //   reports the wrong thing.
            //
            // Refusing here keeps both in the preflight, where the commit is
            // all-or-nothing and the message can name the actual conflict.
            if recreated
                && !nodes
                    .iter()
                    .all(|(key, node)| !key.starts_with(d) || node.is_new)
            {
                return Err(Error::EditUnsupported(
                    "a staged edit names a group at or under a replaced path that this \
                     commit does not itself create; create it in the same commit, or use \
                     separate commits",
                ));
            }
            // A copy reading from a path this commit replaces is the same conflict
            // seen from the other side: `read_copy_subtree` already took its bytes
            // from the pre-commit file, so the commit would place the *original*
            // at the copy's destination while placing something else at `d` — two
            // different objects from one path, in one commit, with no error. A
            // source that is merely deleted and not replaced is unambiguous (it is
            // a move) and stays allowed.
            if recreated {
                for t in &copy_sources {
                    if t.starts_with(d) {
                        return Err(Error::EditUnsupported(
                            "a copy in this commit reads from a path the same commit \
                             replaces; use separate commits",
                        ));
                    }
                }
            }
            // Past that return `recreated` means the whole touched subtree is
            // fresh, so an addition at or below `d` lands in the replacement. The
            // apply loop already removes a group's deleted links before appending
            // any new one (`remove_link_from_region` runs first), so a
            // replacement needs no further sequencing here.
            for t in &add_targets {
                if recreated && t.starts_with(d) {
                    continue;
                }
                if t.starts_with(d) || d.starts_with(t) {
                    return Err(Error::EditUnsupported(
                        "a deletion overlaps an addition in the same commit; \
                         replace the path instead, or use separate commits",
                    ));
                }
            }
            for t in &attr_targets {
                if recreated && t.starts_with(d) {
                    continue;
                }
                if t.starts_with(d) {
                    return Err(Error::EditUnsupported(
                        "a deletion overlaps a group-attribute edit in the same commit; use separate commits",
                    ));
                }
            }
            for t in &write_targets {
                if t.starts_with(d) {
                    // `write_targets` holds three kinds — a value overwrite, a
                    // dataset-attribute edit, and a staged append — so the
                    // message names what they have in common rather than only
                    // the first of them.
                    return Err(Error::EditUnsupported(
                        "a deletion overlaps a staged edit to a dataset in the same \
                         commit; use separate commits",
                    ));
                }
            }
            for (j, d2) in delete_targets.iter().enumerate() {
                if i != j && d2.starts_with(d) {
                    return Err(Error::EditUnsupported(
                        "overlapping deletions in one commit; delete the common parent only",
                    ));
                }
            }
            ensure_ancestors(&mut nodes, &parent);
            nodes.entry(parent).or_default().deletes.push(leaf);
        }

        // Resolve / validate each node's base object-header region up front.
        // Every existing dirty group is rewritten to a freshly-appended header,
        // so its old header becomes dead bytes once the superblock is repointed;
        // `superseded_addrs` records those old headers for reclamation (#21),
        // paired with the path whose rebuilt header replaces each — the two
        // halves of one relocation, which is what an object reference stored
        // elsewhere in the file has to be repointed across (issue #324).
        let keys: Vec<ObjectPathBuf> = nodes.keys().cloned().collect();
        let mut superseded_addrs: Vec<(ObjectPathBuf, u64)> = Vec::new();
        for key in &keys {
            let is_new = nodes[key].is_new;
            if is_new {
                nodes.get_mut(key).unwrap().base_region = fresh_group_region();
            } else {
                let addr = crate::group_v2::resolve_path_any_from_source(
                    &self.image(),
                    AccessMode::ReadWrite,
                    &self.superblock,
                    &key.as_path(),
                )?;
                // Rebuilding this group moves its header and patches only the
                // link this commit resolved it through, so every other hard link
                // to it would be left naming the old header — which this commit
                // then frees. The aliases show the pre-commit group until
                // something reuses the span, and are unreadable after (issue
                // #327). Same rule, and the same lazily-computed count, as the
                // three relocating dataset writes above.
                //
                // The root group is exempt because it is named by the superblock
                // rather than by a link, so it has no entry in the count and
                // nothing to strand. Every other group was resolved by path and
                // therefore has at least one.
                if !key.is_empty() {
                    let counts = incoming_links
                        .get_or_insert_with(|| self.count_incoming_hard_links())
                        .as_ref();
                    match counts.and_then(|c| c.get(&addr).copied()) {
                        Some(1) => {}
                        // Known, and more than one: the aliases are the problem.
                        Some(_) => {
                            return Err(Error::EditUnsupported(
                                "editing a group relocates its object header; only supported \
                                 when it has a single hard link",
                            ));
                        }
                        // Not known: the file-wide walk that counts links gave up
                        // — an object header it could not parse, a group it could
                        // not enumerate, or a link graph past its bound. A
                        // different refusal because it is a different problem, and
                        // "it has more than one hard link" would send the reader
                        // looking for a second link that may not exist.
                        None => {
                            return Err(Error::EditUnsupported(
                                "editing a group relocates its object header, and this file's \
                                 links could not be walked to establish that nothing else \
                                 names it",
                            ));
                        }
                    }
                }
                let info = self.inspect_group(addr)?;
                superseded_addrs.push((key.clone(), addr));
                let node = nodes.get_mut(key).unwrap();
                node.base_region = info.region;
                node.existing_links = info.link_names;
            }
        }

        // A rebuilt group's old header is vacated just as a relocated dataset's is,
        // so the two join one list for the screen below.
        moved_headers.extend(superseded_addrs.iter().map(|&(_, a)| a));

        // Apply and validate group attribute edits before any writes. This keeps
        // unsupported attribute edits under the same all-or-nothing preflight
        // contract as unsupported dataset additions. A variable-length attribute
        // is not fully resolved here — its global heap collection is built (it
        // is self-contained, no address needed yet) but placed and patched into
        // `base_region` only in the apply loop below, once its address is known.
        let attrs_by_group = group_by_parent(staged.group_attrs.iter().map(|(p, op)| (p, op)));
        // The pre-commit header address of each group this commit rewrites. An
        // edit that sends a group's attributes to a fractal heap reads the set it
        // rebuilds back out of the file, which needs the address; a group this
        // commit *creates* has neither an address nor any stored attribute, and
        // its `None` says so.
        let existing_group_addrs: HashMap<ObjectPathBuf, u64> =
            superseded_addrs.iter().cloned().collect();
        for key in &keys {
            if let Some(ops) = attrs_by_group.get(key) {
                let region = std::mem::take(&mut nodes.get_mut(key).unwrap().base_region);
                let edits = plan_attr_ops(
                    &self.image(),
                    base,
                    existing_group_addrs.get(key).copied(),
                    &region,
                    ops,
                )?;
                let node = nodes.get_mut(key).unwrap();
                node.base_region = edits.region;
                node.attrs = edits.attrs;
            }
        }

        // Map each node to its direct child group nodes, each with the link name
        // it has in that parent (for link wiring). The root group has no name in
        // a parent, so `split_leaf` gives `None` for it and the loop skips it.
        let mut children: BTreeMap<ObjectPathBuf, Vec<(ObjectPathBuf, LinkNameBuf)>> =
            BTreeMap::new();
        for key in &keys {
            if let Some((parent, leaf)) = key.split_leaf() {
                children
                    .entry(parent)
                    .or_default()
                    .push((key.clone(), leaf));
            }
        }

        // Each group's added datasets, in the order the apply loop will place
        // them, for the guards below to read. Borrowed from the staged set:
        // every one of those guards may still refuse, and a refusal gives the
        // caller back every edit it was holding (issue #316). The apply loop
        // rebuilds the same grouping, by the same rule, once it owns the set.
        let datasets_by_group = group_by_parent(staged.datasets.iter().map(|(p, fd)| (p, fd)));

        // A group that tracks *link* creation order takes an addition — the
        // apply loop numbers each added link from the running maximum its Link
        // Info message records — up to the point the addition would send its
        // links dense, which is refused before anything is written. The count
        // this screens is the one the group would hold afterwards: the links it
        // has now, less the ones this commit deletes, plus the ones it adds.
        for key in &keys {
            let node = &nodes[key];
            let added = node.copies.len()
                + node.cross_copies.len()
                + datasets_by_group.get(key).map_or(0, Vec::len)
                + children.get(key).map_or(0, |kids| {
                    kids.iter().filter(|(child, _)| nodes[child].is_new).count()
                });
            if added > 0 {
                let kept = node
                    .existing_links
                    .iter()
                    .filter(|name| !node.deletes.iter().any(|d| d.as_str() == *name))
                    .count();
                reject_dense_link_creation_order(&node.base_region, kept + added)?;
            }
        }

        // Validate names: no addition may collide with an existing link or with
        // another addition under the same parent. A link this same commit
        // deletes is not one of the existing ones — the apply loop removes it
        // from the region before any addition is appended, so replacing an
        // object at its own path is a rotation, not a collision (issue #305).
        for key in &keys {
            let node = &nodes[key];
            let mut adding: Vec<&str> = Vec::new();
            for fd in datasets_by_group.get(key).into_iter().flatten() {
                adding.push(fd.name.as_str());
            }
            for (child, leaf) in children.get(key).into_iter().flatten() {
                if nodes[child].is_new {
                    adding.push(leaf.as_str());
                }
            }
            for (leaf, _) in &node.copies {
                adding.push(leaf.as_str());
            }
            for leaf in &node.cross_copies {
                adding.push(leaf.as_str());
            }
            for (i, name) in adding.iter().enumerate() {
                let survives = node.existing_links.iter().any(|n| n == name)
                    && !node.deletes.iter().any(|n| n.as_str() == *name);
                if survives || adding[..i].contains(name) {
                    return Err(Error::EditUnsupported(
                        "a link with this name already exists in the target group",
                    ));
                }
            }
        }

        // Content the caller's library-version bound cannot carry is refused
        // here, before any write, so a rejected addition leaves the commit
        // unapplied.
        self.check_libver_admits(staged.datasets.iter().map(|(_, fd)| fd))?;

        // Enumerate what this commit's deletions reclaim, from the current
        // on-disk layout and before any byte moves. It is read here, ahead of
        // every remaining refusal, because two of them screen against it: an
        // object reference this commit writes must not name space the same
        // commit frees (issue #317). The spans are carried to `to_free` below
        // rather than walked a second time, so the screen and the allocator
        // always mean the same thing by "removed".
        //
        // An object's storage is reclaimed only when the link being removed is
        // its LAST hard link: HDF5 objects can have several hard links, and one
        // reachable through a surviving link is still live (freeing it would
        // corrupt the survivor). Count every hard link in the pre-commit file
        // and reclaim a deleted object only when its count is exactly 1.
        // `deleted_addrs` is de-duplicated first so two delete paths that are
        // hard links to the same object are not visited (and freed) twice. If
        // the link graph cannot be walked in full, no deleted object is
        // reclaimed (a safe leak) — and none is screened either, which is sound
        // in the same direction: nothing is freed, so no reference dangles.
        // Superseded group headers and relocated dataset headers are vacated too,
        // and are screened separately: `moved_headers` holds them, and they are
        // addresses rather than spans.
        let mut deleted_free: Vec<(u64, u64, FreeClass)> = Vec::new();
        deleted_addrs.sort_unstable();
        deleted_addrs.dedup();
        if !deleted_addrs.is_empty() {
            if let Some(incoming) = self.count_incoming_hard_links() {
                for &a in &deleted_addrs {
                    self.collect_free_spans(a, 0, &incoming, &mut deleted_free);
                }
            }
        }
        // An in-file copy re-emits its source's element bytes verbatim, so a
        // copied object reference keeps naming whatever it named in the source —
        // including an object this same commit is removing (issue #317). The page
        // types the walk records are for the allocator, not for this.
        //
        // Removals only, and this is the one place the two halves part company. A
        // *supplied* address is a fresh claim about the file, and a commit that
        // moves the object falsifies it. A *copied* one repeats a claim the file
        // already made, and since issue #324 a move no longer falsifies either
        // one: the commit's last act walks the tree it published and repoints
        // every reachable stored address, the copy's among them
        // ([`crate::reference_patch`], and `a_copy_made_in_the_same_commit_that_
        // moves_its_target_is_repointed`). A removal is the case that stays,
        // because there is no new address to point at.
        //
        // Refusing on `moved` here would also cost what it cannot buy: every
        // commit that reaches here rebuilds its root group, so `moved` is never
        // empty, and a chunked object-reference dataset is refused on a non-empty
        // screen alone rather than on a matching address — so every such copy
        // would be refused, in every commit.
        let for_copied = InvalidatedAddresses {
            removed: deleted_free
                .iter()
                .map(|&(off, len, _)| (off, len))
                .collect(),
            moved: Vec::new(),
            base: self.superblock.base_address,
        };
        {
            // Framed at the base address: a shared-message address is stored
            // relative to it, and `SourceResolver` reads its references as
            // absolute within the view it is given.
            let image = self.image();
            let framed = BaseOffsetSource {
                inner: image,
                base: self.superblock.base_address,
            };
            for key in &keys {
                for (_, tree) in &nodes[key].copies {
                    screen_copied_references(tree, &for_copied, &framed)?;
                }
            }
        }

        // The same screen the copies just got, plus the half a supplied address
        // earns and a copied one does not.
        let for_supplied = InvalidatedAddresses {
            moved: moved_headers,
            ..for_copied
        };

        // A shared-message index record can hold an object-header address too,
        // and it is the one stored address the commit's closing repoint walk
        // cannot reach. Screen it here, against the same two lists.
        self.screen_shared_message_index(&for_supplied)?;

        // References a builder already resolved to addresses never reach
        // `resolve_reference_target`, so they are screened out of the element
        // bytes instead — additions and value overwrites alike, since neither
        // form's bytes are touched again before they are written. A slot still
        // holding a placeholder screens as the zero it is, and is screened for
        // real by `resolve_reference_target` when it resolves.
        //
        // Cross-file copies need no screen for the opposite reason:
        // `reject_foreign_addresses` refuses a reference datatype outright on
        // that path.
        for (_, fd) in staged.datasets.iter().chain(&staged.writes) {
            screen_resolved_references(&fd.dt, &fd.raw, &for_supplied)?;
        }

        // Anything this commit adds could be a reference container the last walk
        // did not see, so it retires that walk's "no references in this file"
        // finding (issue #324). See `proved_free_of_references` for which of
        // these four clauses is load-bearing and which are belt-and-braces, and
        // for why the other five staged collections need no clause at all.
        //
        // A staged dataset's *attributes* are screened beside its element
        // datatype. Neither can carry a reference today — a committed one is
        // refused by `flatten_dataset`, and `AttrValue` has no reference variant
        // — but that is a refusal in another function rather than a property of
        // this collection, and the check that means what it says is the one that
        // looks.
        if staged.datasets.iter().chain(&staged.writes).any(|(_, fd)| {
            datatype_holds_object_address(&fd.dt)
                || fd
                    .attrs
                    .iter()
                    .any(|a| datatype_holds_object_address(&a.datatype))
        }) || !staged.copies.is_empty()
            || !staged.cross_copies.is_empty()
        {
            self.proved_free_of_references = false;
        }

        // Prove every object-reference target resolves before any write (see
        // `preflight_reference_targets`'s doc comment): otherwise a reference
        // resolution failure discovered mid-apply-loop would leave every
        // earlier-processed group's real writes (headers, data, copied
        // subtrees) orphaned in the file despite `commit()` returning `Err`.
        Self::preflight_reference_targets(
            &keys,
            &datasets_by_group,
            &nodes,
            &add_targets,
            &write_targets,
            delete_targets,
            &for_supplied,
            &self.image(),
            &self.superblock,
        )?;

        // --- The point of no return. Every refusal is behind us, so the staged
        // set is taken here rather than at the top of this function: a refusal
        // above restores *every* edit the caller staged, and a failure below —
        // an I/O error mid-apply, which leaves the file valid because the
        // superblock is repointed last — restores none of them, so no later
        // `commit` can finish a batch this one abandoned (issue #316). Nothing
        // above this line may consume from `staged`. ---
        let mut taken = std::mem::take(staged);
        // The same paths as the borrow above, owned now that the set has moved.
        let delete_targets = std::mem::take(&mut taken.deletes);
        // The same grouping as `datasets_by_group` above, by the same rule and
        // so in the same order — which is what keeps the apply loop's placement
        // order the one `preflight_reference_targets` proved.
        let mut flat = group_by_parent(taken.datasets.drain(..));
        // A cross-file copy's subtree can move now, so it joins its destination
        // group's in-file copies. The parent node exists: the preflight made one
        // for every destination.
        for (dst, tree) in taken.cross_copies.drain(..) {
            // A destination that identifies no link is rejected above.
            let (parent, leaf) = dst.split_leaf().expect("a copy destination names a link");
            nodes.get_mut(&parent).unwrap().copies.push((leaf, tree));
        }

        // Gather the regions this commit will vacate, read from the current
        // on-disk layout before any byte moves: every deleted object's owned
        // blocks plus every superseded group header. They stay out of the free
        // list until every append this commit makes is behind it, so none of
        // them is reused while it is still live. (On the non-persisting tail
        // that point is a few lines *before* the superblock write rather than
        // after it — the apply loop is where the appends happen, and it is long
        // done by then.) Enumeration is
        // best-effort — `collect_free_spans` simply omits anything it cannot
        // account for exhaustively, so the worst case is unreclaimed dead bytes,
        // never a freed-but-live region.
        // It starts as the deleted objects' blocks, enumerated above the preflight
        // because a refusal there screens against them.
        let mut to_free: Vec<(u64, u64, FreeClass)> = deleted_free;

        // A superseded group header is dead once the root is repointed. Its chunk
        // spans are enumerated base-aware (`oh_chunk_spans` shifts continuation
        // addresses by the userblock base and returns absolute file offsets), as is
        // the delete path (`collect_free_spans`), so all of this reclamation works
        // on userblock files too.
        for &(_, a) in &superseded_addrs {
            if let Ok(spans) = self.oh_chunk_spans(a) {
                to_free.extend(meta_spans(spans));
            }
        }

        // A relocating overwrite (`write_dataset` resize, or any compact rewrite)
        // vacates the dataset's old object header, and a resized contiguous one
        // also vacates its old data block: both become dead once the parent's
        // relinked header lands. `superseded_addrs` covers only the rebuilt group
        // headers, not the relocated dataset's own header, so record that here too.
        // The pre-commit dataset-header address rides on the write plan (`old_oh`),
        // recorded where the plan was made; its chunks and old data extent are
        // freed only after the superblock repoint.

        // The single-hard-link guard in the write preflight makes freeing the old
        // header safe (no surviving link still points at it).
        for key in &keys {
            for (_leaf, old_oh, mw) in &nodes[key].writes {
                match mw {
                    MovingWrite::Contiguous {
                        old_extent: Some(extent),
                        ..
                    } => to_free.push((extent.0, extent.1, FreeClass::Page(PageType::Raw))),
                    // A relocated chunked dataset vacates its old chunk index and
                    // chunk data blocks. `chunked_storage_spans` returns `None` for
                    // anything it cannot enumerate exhaustively (leaving dead bytes
                    // rather than freeing a region still in use); the old header
                    // chunks are freed generically below.
                    MovingWrite::Chunked { old_addr, .. } => {
                        if let Some(spans) = self.chunked_storage_spans(*old_addr) {
                            to_free.extend(spans);
                        }
                    }
                    // A relocating append keeps the existing chunk *data* in place
                    // (shared by both indexes during the commit), so only the old
                    // index structure and the relocated old trailing chunk are dead.
                    // The old header chunks are freed by the generic path below.
                    MovingWrite::AppendedChunks {
                        old_addr,
                        old_tail_extent,
                        kept_chunks,
                        ..
                    } => {
                        if let Some(spans) = self.chunked_index_spans(*old_addr) {
                            // The old index is reclaimed as raw only where it
                            // provably sits in a raw page, and recorded as dead
                            // otherwise. See `index_is_provably_raw`. The
                            // dataset's old chunk data is the kept chunks
                            // (base-relative) plus the trailing partial chunk
                            // this append relocated, both already in hand, so
                            // the proof needs no second walk of the index.
                            let data: Vec<(u64, u64)> = kept_chunks
                                .iter()
                                .filter_map(|c| {
                                    Some((base.absolute(c.address).ok()?, c.compressed_size))
                                })
                                .chain(*old_tail_extent)
                                .collect();
                            let class = if self.index_is_provably_raw(&data, &spans) {
                                FreeClass::Page(PageType::Raw)
                            } else {
                                FreeClass::Dead
                            };
                            to_free.extend(spans.into_iter().map(|(a, l)| (a, l, class)));
                        }
                        if let Some(ext) = old_tail_extent {
                            // The relocated old trailing chunk is raw data.
                            to_free.push((ext.0, ext.1, FreeClass::Page(PageType::Raw)));
                        }
                    }
                    _ => {}
                }
                // The relocated dataset's old header chunks are dead too.
                if let Ok(spans) = self.oh_chunk_spans(*old_oh) {
                    to_free.extend(meta_spans(spans));
                }
            }
        }

        // Defense in depth: never hand the free list an out-of-bounds or
        // overlapping span. The last-link guard plus the per-object checks
        // should already make the accumulated spans disjoint; this enforces it
        // as a whole-commit invariant against the pre-commit end-of-file. Any
        // dropped span (which should not occur for a well-formed file) only
        // leaks, never corrupts.
        retain_disjoint_in_bounds(&mut to_free, self.image.len());

        // --- Apply: process deepest groups first so each parent sees its
        // children's new addresses, then repoint the superblock last.
        // `path_addr` accumulates every group's and dataset's address as it is
        // placed — read by `resolve_reference_target` to resolve a same-commit
        // object-reference target (see the dataset-placement loop below for the
        // group/dataset key convention: a group's own path, or a dataset's
        // full parent+name path). ---
        let mut path_addr: BTreeMap<ObjectPathBuf, u64> = BTreeMap::new();
        // Where each object-header address this commit vacates has been rewritten
        // to, for repointing the object references the rest of the file already
        // stores (issue #324). Filled from the two places a header moves: a
        // relocating value overwrite, recorded in the loop below as its plan is
        // applied, and a rebuilt group, joined from `superseded_addrs` once every
        // group's new address is known.
        let mut relocations: BTreeMap<u64, u64> = BTreeMap::new();
        let mut by_depth = keys.clone();
        // Stable on purpose: `keys` comes from a `BTreeMap`, so equal-depth
        // groups arrive in path order, and this order is the order they are
        // placed in — which fixes every address the commit writes. An unstable
        // sort may permute a depth group, and the file's layout with it.
        by_depth.sort_by_key(|k| std::cmp::Reverse(k.len())); // deepest first
        for key in &by_depth {
            let (mut region, deletes, copies, writes, attrs) = {
                let node = nodes.get_mut(key).unwrap();
                (
                    std::mem::take(&mut node.base_region),
                    std::mem::take(&mut node.deletes),
                    std::mem::take(&mut node.copies),
                    std::mem::take(&mut node.writes),
                    std::mem::take(&mut node.attrs),
                )
            };

            // Remove deleted links first (verbatim-preserving the rest).
            //
            // First is a correctness requirement rather than a convention, and
            // has been since a replacement became one commit (issue #305):
            // `remove_link_from_region` matches by *name*, so a removal running
            // after the additions below would take the replacement's link with
            // the original's and leave the path gone. Every link this loop could
            // collide with — a copy's, a dataset's, a new child group's — is
            // appended after it, which is what keeps that unreachable. (A
            // relocating write and an existing child group *patch* a link rather
            // than appending one, so neither can be a replacement's.)
            for name in &deletes {
                region = remove_link_from_region(&region, name.as_str())?;
            }

            // Every link appended below takes the next creation index from this
            // group's running maximum, on the groups that track link creation
            // order and on no others; `link_order.record` writes the bumped
            // maximum back once the last of them is placed. Read after the
            // deletions above because a deletion leaves the maximum alone — the
            // gap it opens is never handed out again.
            let mut link_order = LinkCreationOrder::for_region(&region)?;

            // Write each staged source subtree and link its root into this group.
            for (leaf, tree) in copies {
                let root = self.write_copy_subtree(&tree)?;
                region.push_link(leaf.as_str(), base.relative(root)?, link_order.take()?);
            }

            // Datasets directly under this group. Appended addresses are absolute
            // file offsets, and the contiguous data-layout address is stored
            // relative to the base address. Placed
            // non-reference datasets first (recording each into `path_addr`), then
            // reference datasets — a reference to a *non-reference* sibling added
            // in the same group's batch resolves regardless of `staged.datasets`
            // call order (`Vec::sort_by_key` is stable, so within each of the two
            // groups the original order is preserved). Two reference datasets that
            // target each other in the same batch are still call-order-dependent —
            // whichever is placed first resolves the other, and the reverse
            // direction is safely refused as "still writing" (never corrupted),
            // caught up front by `preflight_reference_targets`.
            let mut group_datasets: Vec<FlatDataset> =
                flat.remove(key).into_iter().flatten().collect();
            // Stable on purpose: this is a partition, and the relative order
            // within each half is what the paragraph above promises.
            group_datasets.sort_by_key(|fd| fd.reference_targets.is_some());
            for mut fd in group_datasets {
                // Place each variable-length attribute's global heap collection
                // and patch its placeholder heap address. Unlike VL-string
                // *data* (`vl_string_staging`, refused when chunked below), a
                // chunked/extensible dataset can carry a VL *attribute* just
                // fine — attributes live in the object header, not inside a
                // chunk, so patching them here before either apply branch runs
                // covers both.
                for (idx, collections) in std::mem::take(&mut fd.vl_attrs) {
                    let addrs = self.place_vl_collections(&collections)?;
                    patch_vl_refs(&mut fd.attrs[idx].raw_data, &addrs);
                }
                // Resolve an object-reference dataset's per-element targets now
                // that every earlier-placed object in this commit is in
                // `path_addr` (chunked datasets never carry these —
                // `flatten_dataset` refuses that combination).
                if let Some(patches) = fd.reference_targets.take() {
                    for patch in &patches {
                        let addr = Self::resolve_reference_target(
                            &patch.target,
                            &path_addr,
                            &nodes,
                            &add_targets,
                            &write_targets,
                            &delete_targets,
                            &for_supplied,
                            &self.image(),
                            &self.superblock,
                        )?;
                        write_reference_address(&mut fd.raw, patch.byte_offset, addr);
                    }
                }
                let oh = if fd.chunk_options.is_chunked() || fd.maxshape.is_some() {
                    self.build_chunked_dataset(&fd)?
                } else {
                    // A staged variable-length-string dataset's element
                    // references still carry a placeholder heap address; place
                    // its collection and patch them before `raw` is appended
                    // (chunked datasets never carry staging — refused above).
                    if let Some(staging) = fd.vl_string_staging.take() {
                        if !staging.collections.is_empty() {
                            let addrs = self.place_vl_collections(&staging.collections)?;
                            patch_vl_refs_masked(&mut fd.raw, &staging.patch_offsets, &addrs);
                        }
                    }
                    // A zero-element dataset has no data block to allocate; its
                    // layout address is the undefined-address sentinel (never
                    // base-relative — see `build_dataset_oh`'s empty-data callers
                    // in the whole-file writer), matching every reader's and the
                    // reference C library's convention for "no storage allocated".
                    let data_addr = if fd.raw.is_empty() {
                        StoredAddress::new(UNDEF)
                    } else {
                        base.relative(self.alloc_or_append_typed(&fd.raw, PageType::Raw)?)?
                    };
                    // Attributes this dataset keeps in a fractal heap are placed
                    // now — after the variable-length patching above, so the heap
                    // holds resolved references — and the header names the heap
                    // instead of carrying them inline.
                    let attr_info = self.place_dense_attrs_if_needed(&fd)?;
                    build_dataset_oh(
                        &fd.dt,
                        // Committed datatypes are refused when a dataset is
                        // staged (`flatten_dataset`), so every type here is
                        // written into the dataset's own header.
                        &DatatypeLocation::Inline,
                        &fd.ds,
                        data_addr,
                        fd.raw.len() as u64,
                        &fd.attrs,
                        attr_info.as_deref(),
                        fd.fill.as_deref(),
                        self.libver(),
                    )?
                };
                let oh_addr = self.alloc_or_append_typed(&oh, PageType::Meta)?;
                region.push_link(
                    fd.name.as_str(),
                    base.relative(oh_addr)?,
                    link_order.take()?,
                );
                path_addr.insert(key.join(&fd.name), oh_addr);
            }

            // Relocating value overwrites under this group: write the new data and
            // rewritten header, then patch this group's existing link to it. On a
            // userblock file only the chunked variant reaches here (contiguous and
            // compact resizes are refused in the write preflight).
            for (leaf, old_oh, mw) in &writes {
                let new_oh = self.write_moving(mw)?;
                patch_link_target(&mut region, leaf.as_str(), base.relative(new_oh)?)?;
                relocations.insert(*old_oh, new_oh);
            }

            // Wire links to dirty child groups (new → add a link; existing →
            // patch the existing link to the child's new address).
            for (child, leaf) in children.get(key).into_iter().flatten() {
                let child_name = leaf.as_str();
                let child_addr = base.relative(path_addr[child])?;
                if nodes[child].is_new {
                    region.push_link(child_name, child_addr, link_order.take()?);
                } else {
                    patch_link_target(&mut region, child_name, child_addr)?;
                }
            }

            // Every addition to this group is placed, so its Link Info message
            // can record the maximum creation index it has now assigned. A group
            // that gained no link, or that does not track the order, is left
            // byte-identical.
            link_order.record(&mut region)?;

            // Whatever this group's attribute edits left to place: a
            // variable-length attribute's heap collection, or a whole dense set.
            self.place_edited_attrs(&mut region, attrs)?;

            let oh = build_v2_object_header(&region)?;
            let addr = self.alloc_or_append_typed(&oh, PageType::Meta)?;
            path_addr.insert(key.clone(), addr);
        }

        // Same-length in-place overwrites (`write_dataset`) write straight into
        // their existing, already-referenced data blocks. Those blocks are
        // reachable from both the old and the new root (the dataset's header is
        // unchanged), so the write is independent of the superblock flip; it is
        // ordered before the barrier sync below so the new bytes are durable
        // alongside everything else this commit appended.
        // Nothing resolved here allocates: `prepare_write` plans a staged
        // variable-length overwrite as a relocating write rather than an
        // in-place one (issue #321), so every entry hands its staged bytes back
        // untouched — which the `vlen.is_none()` assertion guarding the fast
        // path states for the whole list, this one included. The journal below
        // is what costs here, and it is not free: see `inplace_undo`.
        for (data_addr, bytes) in &inplace_writes {
            let raw = self.resolve_overwrite_bytes(bytes)?;
            self.write_inplace_journaled(*data_addr, &raw)?;
        }

        // Every group's new header address is known now that the apply loop has
        // run, so the rebuilt groups can join the relocation map. A group whose
        // key is missing would be one the loop did not place, which cannot happen:
        // `superseded_addrs` is filled from `keys`, and the loop visits all of it.
        for (key, old) in &superseded_addrs {
            if let Some(&new) = path_addr.get(key) {
                relocations.insert(*old, new);
            }
        }

        // Repoint the superblock at the new root last: this is the commit's
        // linearization point. Until it lands, the file on disk still points at
        // the old root (the appended objects are merely unreferenced trailing
        // bytes), so a failure here leaves a valid file.
        //
        // That ordering is only crash-safe if the appended objects are durable
        // before the root pointer is flipped; otherwise a power loss could
        // persist the flip ahead of the data it references, leaving the root
        // pointing at bytes that never reached disk. `flush` on a plain `File`
        // does not force a write-back, so sync the appended bytes to disk first
        // (the barrier), then flip the pointer, then sync the flip.
        let new_root = path_addr[&ObjectPathBuf::root()];

        // The heap collections this commit's value overwrites superseded. They
        // join `to_free` rather than being handed to the free list here, which
        // is what keeps them out of reach of this commit's own allocations:
        // nothing draws on `to_free` until the commit has finished placing, so
        // the elements naming them are still readable from the prior root for
        // as long as that root is the live one.
        // Freed as metadata because that is the page type they were placed as
        // (`place_vl_collections`), which is the only claim about them this
        // session can make from its own record.
        to_free.extend(
            self.superseded_heaps
                .drain(..)
                .map(|(addr, len)| (addr, len, FreeClass::Page(PageType::Meta))),
        );
        // Re-run the whole-commit invariant over the set these just joined, since
        // the pass above ran before the in-place writes resolved and so never saw
        // them. This is defensive and nothing reaches it: no test fails when it
        // is deleted, and no sequence has been found that puts a superseded
        // collection out of bounds or across another freed span. It is kept
        // because a heap collection is freed on the strength of a *record*
        // rather than of a structure read back out of the file, which is the one
        // span where a stale answer would not be caught by anything else.
        retain_disjoint_in_bounds(&mut to_free, self.image.len());

        // A persisting file keeps its freed space recorded on disk rather than
        // truncating it away, so its commit takes a different tail: one that
        // rewrites the on-disk managers rather than only repointing the superblock.
        if self.persist.is_some() {
            self.commit_persisting_releasing_reserve(new_root, to_free)?;
            return self.repoint_stored_references(&relocations);
        }

        // The new tree is fully written, so the regions this commit vacated are
        // now dead: hand them to the session free list. If the resulting free
        // space forms a run reaching end-of-file, the file can be physically
        // truncated to where that run starts; otherwise the end-of-file is
        // unchanged. `take_trailing` removes the trimmed run so it is not also
        // counted as reusable interior space.
        // The class is not consulted: a non-persisting commit is only reached on a
        // flat file (a paged one is refused without persistence), which has no page
        // types to keep apart and so vacates nothing it cannot place.
        for (a, l, _) in to_free.drain(..) {
            self.free.free(a, l);
        }
        let cur_eof = self.image.len();
        let trunc_to = self.free.take_trailing(cur_eof);
        let new_eof = trunc_to.unwrap_or(cur_eof);

        self.barrier()?;
        // The root address is stored relative to the base address; the end-of-file
        // address is absolute. After writing the relative root to disk, keep the
        // in-memory `root_group_address` absolute (the open-time convention).
        if self.superblock.version >= 2 {
            // Build the new superblock off a clone and adopt it only once the
            // write succeeds, so a failed write does not desync the in-memory
            // state. The v2/v3 superblock carries its own checksum.
            let mut new_sb = self.superblock.clone();
            new_sb.root_group_address = base.relative(new_root)?.get();
            new_sb.eof_address = new_eof;
            // Publish the flags this session *raised*, so a flag the source file
            // arrived carrying (left set by a crashed SWMR writer, say) is
            // scrubbed and this clean commit leaves the file properly closed for
            // the C library (issue #73). serialize() recomputes the v2/v3
            // checksum.
            //
            // Provably zero here: the only session that holds flags across a
            // commit is a page-buffered one, which requires a paged file, and a
            // paged file takes the tail below instead. Written as the field
            // rather than as `0` because "publish what this session holds" is the
            // rule, and a site that states it as a literal is one refactor away
            // from being wrong silently (issue #308).
            new_sb.consistency_flags = self.held_status_flags;
            let sb_bytes = new_sb.serialize();
            self.publish_attempted = true;
            self.write_at(self.sb_sig_off, &sb_bytes)?;
            self.barrier()?;
            new_sb.root_group_address = new_root;
            self.superblock = new_sb;
        } else {
            self.publish_attempted = true;
            self.repoint_v0v1_root(base.relative(new_root)?.get(), new_eof)?;
            self.barrier()?;
            self.superblock.root_group_address = new_root;
            self.superblock.eof_address = new_eof;
        }

        // Physically shrink the file only after the superblock — now carrying the
        // smaller end-of-file — is durable. A crash between the two leaves a file
        // whose superblock end-of-file is correct and whose trailing bytes are
        // mere unreferenced slack, which the next open ignores; the reverse order
        // could advertise an end-of-file past the actual file length.
        if let Some(cut) = trunc_to {
            self.image.truncate(cut)?;
            self.barrier()?;
        }
        self.repoint_stored_references(&relocations)
    }

    /// Repoint every object reference the file already stores at a header this
    /// commit moved (issue #324).
    ///
    /// Called at the end of each commit tail, *after* the superblock repoint,
    /// which is where it belongs on both counts. The commit is atomic at the
    /// repoint and this is a fixup derived from it, so it must not run earlier:
    /// a crash before the repoint has to leave the pre-commit file, its stored
    /// references included. And the tree it walks has to be the committed one —
    /// a rebuilt group's header is a fresh copy of the pre-commit one and
    /// carries the old address of anything it referenced, so patching the
    /// superseded header instead would correct bytes that are already dead.
    ///
    /// The writes are followed by a [`barrier`](Self::barrier), so a commit that
    /// returns under [`SyncPolicy::Always`] has put its references on disk and
    /// not merely its tree. That rests on `barrier`'s own contract rather than
    /// on a crash test — nothing here sweeps the window — and it costs nothing
    /// on the files that do not need it: a plan with nothing in it returns
    /// first, and on a file holding no object reference there is never anything
    /// in it.
    ///
    /// A crash between the repoint and that barrier leaves the commit standing
    /// with some references still stale, which is exactly the state every commit
    /// left before this existed.
    ///
    /// Bounded by [`MAX_LINK_GRAPH_NODES`], the same budget
    /// [`count_incoming_hard_links`](Self::count_incoming_hard_links) walks the
    /// link graph under.
    fn repoint_stored_references(&mut self, relocations: &BTreeMap<u64, u64>) -> Result<(), Error> {
        if self.proved_free_of_references {
            return Ok(());
        }
        let plan = crate::reference_patch::plan(
            &self.image(),
            &self.superblock,
            relocations,
            MAX_LINK_GRAPH_NODES,
        )?;
        // Record the walk's verdict before applying it, and only ever the
        // positive proof: a walk that found a reference, or could not read some
        // object, leaves the question open rather than answering "yes there are
        // references", because the next commit's answer could differ.
        if plan.proved_free_of_references() {
            self.proved_free_of_references = true;
        }
        if plan.is_empty() {
            return Ok(());
        }
        plan.apply(self)?;
        self.barrier()
    }

    /// Commit tail for a file that persists its free space (issue #21). Unlike
    /// the non-persisting path, freed space is *retained* and recorded on disk —
    /// matching the reference library's persistent free-space strategy — so a
    /// later reopen (by this crate or the C library) recovers it.
    ///
    /// The post-commit free list (this commit's vacated regions plus the now-dead
    /// old free-space-manager and extension blocks) is serialized into a fresh
    /// `FSHD`/`FSSE` pair and a rewritten superblock-extension File Space Info
    /// message. The three are written as one contiguous tail, placed in space an
    /// *earlier* commit freed where it fits ([`flat_tail_layout`](Self::flat_tail_layout))
    /// and appended at end-of-file where it does not. Nothing live or
    /// still-referenced is overwritten either way, and the superblock — repointed
    /// last — is the linearization point, so a crash before it leaves the prior
    /// file (root, extension, and managers) wholly intact.
    ///
    /// The tail used to always append. Every commit then grew the file by one
    /// tail and freed its predecessor's, so a delete-and-recreate workload grew
    /// without bound while reporting all of that as reusable — the same defect
    /// [`commit_persisting_paged`](Self::commit_persisting_paged) had for its own
    /// tail (issue #286), on the strategy the paged one was measured against
    /// (issue #358).
    fn commit_persisting(
        &mut self,
        new_root: u64,
        to_free: Vec<(u64, u64, FreeClass)>,
        placement: TailPlacement,
    ) -> Result<(), Error> {
        // A paged file records its free space in per-page-type managers and keeps
        // its allocation page-aligned, so it takes its own tail (issue #198).
        if self.paged.is_some() {
            return self.commit_persisting_paged(new_root, to_free, placement);
        }
        let os = self.superblock.offset_size;
        let (strategy, threshold, page_size, old_blocks) = {
            // Copy what we need so no borrow of `self.persist` is held across the
            // `&mut self` writes below; the old state stays in place so a failure
            // leaves the session reusable.
            let ps = self
                .persist
                .as_ref()
                .expect("commit_persisting is only called when persistence is armed");
            (
                ps.strategy,
                ps.threshold,
                ps.page_size,
                ps.old_blocks.clone(),
            )
        };
        let base = self.superblock.base_address;

        // The extension address is stored relative to the base address, and every
        // reader of the extension below indexes the image, so it is shifted once
        // here for all of them.
        let old_ext_addr = base.absolute(
            self.superblock
                .superblock_extension_address
                .map(StoredAddress::new)
                .filter(|a| !a.is_undefined(os))
                .ok_or(Error::EditUnsupported(
                    "a persisting file has no superblock extension to update",
                ))?,
        )?;

        // The persist File Space Info message is fixed-size, so the rewritten
        // extension's length is independent of the addresses it will carry: size
        // it with a placeholder to place the FSM blocks that follow it.
        let placeholder =
            FileSpaceInfo::persistent_single_manager(strategy, threshold, page_size, 0, 0);
        let ext_len =
            build_v2_object_header(&self.rewrite_extension_region(old_ext_addr, &placeholder)?)?
                .len() as u64;

        // Place the tail — the rewritten extension and the manager blocks — in a
        // hole an *earlier* commit freed where one fits, and at end-of-file where
        // none does. `self.free` holds only durable free space (this commit's own
        // frees stay in `to_free` until the repoint), so a reused span holds bytes
        // already unreachable from the on-disk root, which is the guarantee
        // [`reserve`](Self::reserve) rests on too.
        let (post, placed_at, tail_len, eoa) =
            self.flat_tail_layout(&to_free, &old_blocks, ext_len, os);
        if placed_at.is_none() && placement == TailPlacement::ReuseOnly {
            // The shrink pass appends nothing: growing the file by a tail is the
            // opposite of what it was called for. Nothing has been written and the
            // layout handed its reservation back, so the session is exactly as the
            // commit before it left it.
            return Ok(());
        }
        let reused = placed_at.is_some();
        let ext_addr = placed_at.unwrap_or_else(|| self.image.len());
        let sections = self.persisted_sections(&post);
        let fshd_addr = self.persisted_address(ext_addr + ext_len);
        let fsse_addr = self.persisted_address(ext_addr + ext_len + fshd_len(os));
        // A reused tail sits inside the file, which ends it at the end-of-allocation
        // the layout settled on — the current end-of-file, less any run of free
        // space reaching it, which `post` no longer records and the truncation
        // below gives back to the filesystem (issue #418). An appended tail ends
        // the file itself.
        let final_eof = if reused { eoa } else { ext_addr + tail_len };
        debug_assert!(
            ext_addr + tail_len <= final_eof,
            "the tail must end at or below the end-of-allocation it was placed under"
        );

        // Build the real extension and the FSM blocks. With no free space to
        // record we still refresh the extension (persist on, managers undefined).
        let (ext_oh, fsm_blocks) = if sections.is_empty() {
            let info = FileSpaceInfo::persistent_empty(strategy, threshold, page_size);
            let ext_oh =
                build_v2_object_header(&self.rewrite_extension_region(old_ext_addr, &info)?)?;
            (ext_oh, None)
        } else {
            // `eoa_pre_fsm` is the end-of-allocation before the free-space-manager
            // section blocks (`FSHD`/`FSSE`) were allocated: a consumer may shrink
            // back to here and rebuild them. For an appended tail that is the FSHD,
            // not the extension — the extension sits below it and persists, so
            // shrinking leaves the superblock and its extension pointer valid (only
            // the manager blocks, which are rewritten every commit, are discarded).
            // A tail placed *inside* the file gave the allocation back nothing to
            // shrink, so it records the end-of-allocation itself. Either way it is
            // defined, which a persisting file's message must be or an
            // assertion-enabled libhdf5 aborts on open (issue #178), and is the
            // value `H5Fget_freespace` accounts for correctly (verified in the
            // crosscheck).
            let eoa_pre_fsm = if reused { final_eof } else { fshd_addr.get() };
            let info = FileSpaceInfo::persistent_single_manager(
                strategy,
                threshold,
                page_size,
                fshd_addr.get(),
                eoa_pre_fsm,
            );
            let ext_oh =
                build_v2_object_header(&self.rewrite_extension_region(old_ext_addr, &info)?)?;
            let (fshd, fsse) =
                serialize_file_fsm(&sections, fshd_addr, fsse_addr, os, SECT_CLASS_SIMPLE);
            (ext_oh, Some((fshd, fsse)))
        };
        // Both forms of the message carry the same twelve manager slots, so either
        // one measures what the placeholder did. The tail was reserved for that
        // length, and the manager blocks are placed after it.
        debug_assert_eq!(
            ext_oh.len() as u64,
            ext_len,
            "extension length must be stable across the placeholder and real messages"
        );

        // Write the extension, then the FSM blocks. Both forms are safe against a
        // crash here: an appended tail is past everything live, a reused one sits
        // in space an earlier commit freed, and neither is referenced until the
        // repoint below.
        let region = (ext_addr, tail_len);
        self.write_tail_block(region, ext_addr, &ext_oh)?;
        if let Some((fshd, fsse)) = fsm_blocks {
            self.write_tail_block(region, base.absolute(fshd_addr)?, &fshd)?;
            self.write_tail_block(region, base.absolute(fsse_addr)?, &fsse)?;
        }
        // Exactly the bytes written, contiguous from the extension, which is what
        // the next commit supersedes.
        let new_old_blocks = vec![(ext_addr, tail_len)];

        // Barrier, then repoint the superblock (root, eof, and the new extension)
        // — the linearization point — and sync it.
        self.barrier()?;
        let mut new_sb = self.superblock.clone();
        new_sb.root_group_address = new_root;
        new_sb.eof_address = final_eof;
        new_sb.superblock_extension_address = Some(ext_addr);
        // As above, and zero for the same reason: this is the *unpaged* persisting
        // tail, which no page-buffered session reaches (issue #73, issue #308).
        new_sb.consistency_flags = self.held_status_flags;
        let sb_bytes = new_sb.serialize();
        self.publish_attempted = true;
        self.write_at(self.sb_sig_off, &sb_bytes)?;
        self.barrier()?;
        self.superblock = new_sb;

        // Physically shrink the file only after the superblock — now carrying the
        // smaller end-of-file — is durable, exactly as the non-persisting tail
        // does: a crash between the two leaves trailing bytes the next open
        // ignores, where the reverse order would advertise an end-of-file past
        // the file's actual length. The managers just written describe `post`,
        // which no longer names anything above this cut.
        if final_eof < self.image.len() {
            self.image.truncate(final_eof)?;
            self.barrier()?;
        }

        // The repoint is durable: the prior free list plus this commit's vacated
        // regions are now genuinely free, and the freshly written blocks become
        // the ones a future commit will supersede.
        self.free = post;
        self.persist = Some(PersistState {
            strategy,
            threshold,
            page_size,
            old_blocks: new_old_blocks,
        });
        // The managers describe the file at this length, so nothing is owed until
        // it grows past them again.
        self.fsm_len = self.image.len();
        Ok(())
    }

    /// The free regions of `free` as the sections a free-space manager records.
    fn persisted_sections(&self, free: &FreeList) -> Vec<FreeSection> {
        free.sections()
            .into_iter()
            .map(|(addr, size)| FreeSection {
                addr: self.persisted_address(addr),
                size,
            })
            .collect()
    }

    /// The address a persisting commit records for the image position `at`.
    ///
    /// The free lists hold positions in this session's image, and a manager block
    /// records the addresses the file stores. The two are the same number here,
    /// since [`load_persisted_free_space`](Self::load_persisted_free_space) arms
    /// persistence for a file whose image begins at byte zero alone, so this
    /// changes the frame and not the value.
    ///
    /// # Panics
    ///
    /// Panics if the file has a userblock, where the two frames differ and a
    /// manager block would record every free region a userblock's length below
    /// where it lies.
    fn persisted_address(&self, at: u64) -> StoredAddress {
        assert!(
            self.superblock.base_address.is_zero(),
            "a persisting commit requires a file whose image begins at byte zero"
        );
        StoredAddress::new(at)
    }

    /// The free space a flat persisting commit is about to record: the session's
    /// durable list plus the regions this commit vacated and the superseded
    /// extension and manager blocks, which are dead once the superblock is
    /// repointed.
    ///
    /// Returned as a temporary rather than folded into the session, for the reason
    /// [`paged_post_free`](Self::paged_post_free) does the same: every region
    /// gathered here is still *live* until that repoint, and everything in between
    /// can fail, so a session that believed them free after a failed commit would
    /// hand them out over the objects still occupying them.
    ///
    /// Called once to *size* the tail and again for each round its placement takes
    /// to settle, since what remains free depends on the space the tail has drawn
    /// for itself ([`flat_tail_layout`](Self::flat_tail_layout)).
    fn flat_post_free(
        &self,
        to_free: &[(u64, u64, FreeClass)],
        old_blocks: &[(u64, u64)],
    ) -> FreeList {
        let mut post = self.free.clone();
        // A flat file keeps one list for the whole of it, so the class each freed
        // region carries for the paged path is not consulted here — and a flat
        // file produces no [`FreeClass::Dead`] region to begin with, since the
        // proof that classes one short-circuits where there are no page types to
        // keep apart.
        for &(a, l, _) in to_free {
            post.free(a, l);
        }
        for &(a, l) in old_blocks {
            post.free(a, l);
        }
        post
    }

    /// Reserve free space for a flat persisting commit's tail, returning the free
    /// list the tail leaves behind, the address it got — `None` when nothing
    /// reusable fits, which is the caller's cue to append — its length, which the
    /// caller needs either way, and the end-of-allocation the commit should
    /// publish, which is the current end-of-file less any run of free space that
    /// reaches it (issue #418).
    ///
    /// The reservation and the length define each other, exactly as they do for the
    /// paged tail ([`tail_layout`](Self::tail_layout)): drawing bytes out of the
    /// free list changes the very sections the manager blocks record, and a
    /// different section set is a different length. This proposes a length, sizes
    /// the blocks against the list as it stands once that length is taken, and
    /// accepts any plan that *fits*. A longer plan is refused — it would write past
    /// the reservation into live bytes — and the proposal becomes the length that
    /// plan asked for, which is the natural next candidate.
    ///
    /// A plan **shorter** than its reservation is accepted, and the reservation
    /// rather than the plan is what the tail extent covers, so those spare bytes
    /// are freed with the rest of the tail when the next commit supersedes it.
    /// Rejecting them instead is what the paged tail does, and it can afford to:
    /// its fallback is a fresh page, recorded in full. Here the fallback is
    /// end-of-file, and the two lengths genuinely need not meet — best fit picks a
    /// different hole as the proposal crosses a hole's size, and the pair can
    /// oscillate by one section record indefinitely. Measured on the workload
    /// `persisting_churn_reaches_a_steady_size` runs, insisting on an exact fit sent
    /// about one commit in eight to end-of-file, which is the growth this exists to
    /// stop.
    ///
    /// What that costs is bounded and small: a commit that takes spare bytes and is
    /// then the last of its session leaves them behind, since a reopened session
    /// reconstructs the tail extent from the blocks themselves
    /// ([`load_persisted_free_space`](Self::load_persisted_free_space)) and cannot
    /// see a reservation that outlived the process. The spare is a section record
    /// and, where the reservation emptied a size group, that group's own header
    /// too — tens of bytes, once per session, against a whole tail on one commit in
    /// eight.
    ///
    /// Iteration is capped rather than trusted to converge: the length is a step
    /// function of the section set, so a proposal can in principle keep growing.
    /// Appending is always available and always correct, so giving up costs space
    /// rather than a guarantee — and the cap bounds the work, since each round
    /// clones the free list and sizes the section list for real.
    ///
    /// Draws from `self.free` before anything is written, which
    /// [`commit`](Self::commit) puts back if the attempt then fails. The one caller
    /// that does not snapshot is [`finalize_persist`](Self::finalize_persist), whose
    /// failure leaves this session's list short by the reservation — in memory only,
    /// on a session already being torn down, with the on-disk managers unchanged.
    ///
    /// When `self.free` has no hole for it, the tail may take one out of
    /// [`reserved`](Self::reserved) instead. Only the rewrite that publishes an
    /// append's draw ([`reserve_for_immediate_append`](Self::reserve_for_immediate_append))
    /// reaches this with anything there — every other caller releases the reserve
    /// first — and for that rewrite the fallback is what keeps the draw from
    /// growing the file: a draw takes every hole the append fits in, and on a
    /// flat file that is the one list the tail is placed from, so without it the
    /// rewrite appended its tail at end-of-file every time and the file grew by a
    /// tail per draw (issue #413). Space in the reserve is exactly what this
    /// rewrite publishes as *not* free, so a tail placed there is live in bytes
    /// the managers do not advertise, which is the same standing a tail placed
    /// from `self.free` has once the repoint lands. `self.free` first, so that an
    /// ordinary hole is spent before the append's own space is; best fit within
    /// the reserve then takes its smallest run, leaving the largest for the
    /// append that drew it. The paged tail ([`tail_layout`](Self::tail_layout))
    /// has no such fallback: it opens a page when nothing fits, and
    /// `paged_group_churn_with_populated_datasets_reaches_a_steady_size` holds it
    /// to not doing so once per draw.
    fn flat_tail_layout(
        &mut self,
        to_free: &[(u64, u64, FreeClass)],
        old_blocks: &[(u64, u64)],
        ext_len: u64,
        os: u8,
    ) -> (FreeList, Option<u64>, u64, u64) {
        /// Enough rounds for the section set to settle after a reservation shrinks
        /// it, without letting a proposal that keeps growing spin.
        const ROUNDS: usize = 4;

        let eof = self.image.len();
        // The first proposal: what the tail would measure with nothing reserved.
        // A manager block's length depends only on the sections it records, never
        // on where it sits. It is also the answer for a tail that ends up appended,
        // since every round hands its reservation back before this returns.
        let probe = self.flat_post_free(to_free, old_blocks);
        let appended_len = ext_len + file_fsm_blocks_len(&self.persisted_sections(&probe), os);
        let mut proposed = appended_len;

        for _ in 0..ROUNDS {
            let (at, from_reserve) = match self.free.alloc(proposed) {
                Some(at) => (at, false),
                None => match self.reserved.alloc(proposed) {
                    Some(at) => (at, true),
                    None => break,
                },
            };
            let mut post = self.flat_post_free(to_free, old_blocks);
            // Free space that reaches end-of-file is released rather than
            // recorded: the file is truncated to where the run starts, so the
            // sections these blocks are sized from must already leave it out
            // (issue #418). The reservation was taken before this list was built,
            // so the run can only begin at or above the tail's own end.
            let eoa = release_trailing_run(&mut post, eof, proposed);
            let len = ext_len + file_fsm_blocks_len(&self.persisted_sections(&post), os);
            if len <= proposed {
                debug_assert!(
                    at + proposed <= eoa,
                    "a reused tail at {at} of {proposed} bytes runs past the end-of-allocation \
                     {eoa} it was placed under"
                );
                return (post, Some(at), proposed, eoa);
            }
            if from_reserve {
                self.reserved.free(at, proposed);
            } else {
                self.free.free(at, proposed);
            }
            proposed = len;
        }
        // Nothing reusable fits, so the tail is appended past end-of-file and no
        // trailing run is released: whatever free space reaches end-of-file was
        // too small for the tail, and stays recorded below it.
        (probe, None, appended_len, eof)
    }

    /// Commit tail for a genuine paged file (`H5F_FSPACE_STRATEGY_PAGE`, issue
    /// #198). The paged counterpart of [`commit_persisting`](Self::commit_persisting).
    ///
    /// Two things differ from the flat tail. Free space is recorded in *per-page-type*
    /// managers — SUPER (slot 0) for metadata, DRAW (slot 2) for small raw, and the
    /// generic-large manager (slot 6) for whole free pages and large-raw fragments —
    /// rather than one generic manager, so a paged file reopened by the reference
    /// library still finds its free space segregated. And the file is padded to a
    /// page before the tail is laid down, so the end-of-allocation stays a whole
    /// number of pages, matching the paged file the from-scratch writer produces.
    ///
    /// The tail itself is placed like any other run of metadata, in free space
    /// where some fits ([`tail_layout`](Self::tail_layout)), and only opens a page
    /// at end-of-file when none does. It used to always append into a page of its
    /// own and pad the remainder out untracked, which cost a page per commit and
    /// recorded none of it — a paged file under delete-and-recreate churn grew
    /// without bound while reporting a fraction of that as reusable (issue #286).
    /// The reference library does not page-align these blocks either: its manager
    /// headers sit at whatever offset within a metadata page they are allocated at.
    ///
    /// Crash atomicity is identical to the flat path: the tail is either past the
    /// live file or inside space an earlier commit freed, and is unreferenced
    /// either way until the superblock repoint, which is the linearization point.
    fn commit_persisting_paged(
        &mut self,
        new_root: u64,
        to_free: Vec<(u64, u64, FreeClass)>,
        placement: TailPlacement,
    ) -> Result<(), Error> {
        let os = self.superblock.offset_size;
        let (strategy, threshold, page_size, old_blocks) = {
            let ps = self
                .persist
                .as_ref()
                .expect("commit_persisting is only called when persistence is armed");
            (
                ps.strategy,
                ps.threshold,
                ps.page_size,
                ps.old_blocks.clone(),
            )
        };

        let base = self.superblock.base_address;

        // Page-align the file before anything else, so the rewritten extension and
        // the manager blocks begin on a page boundary and stay in metadata pages.
        // The padded tail becomes free space of whatever type that page held.
        self.pad_to_page()?;

        // As on the flat path: the extension address is stored relative to the base
        // address, and every reader of the extension below indexes the image.
        let old_ext_addr = base.absolute(
            self.superblock
                .superblock_extension_address
                .map(StoredAddress::new)
                .filter(|a| !a.is_undefined(os))
                .ok_or(Error::EditUnsupported(
                    "a persisting file has no superblock extension to update",
                ))?,
        )?;

        // The 12-slot persist message is fixed-size, so a placeholder sizes the
        // rewritten extension before its manager addresses are known — and before
        // its address is, which is what lets the tail be sized before it is placed.
        let placeholder = FileSpaceInfo::persistent_managers(
            strategy,
            threshold,
            page_size,
            [UNDEF; NUM_FILE_FSM_MANAGERS],
            0,
        );
        let ext_len =
            build_v2_object_header(&self.rewrite_extension_region(old_ext_addr, &placeholder)?)?
                .len() as u64;

        // Place the tail — the rewritten extension and the manager blocks — as an
        // ordinary run of metadata, in a hole an *earlier* commit freed where one
        // fits. That is what stops a file under delete-and-recreate churn from
        // growing: every commit frees its predecessor's tail exactly, so in the
        // steady state a tail lands in a hole an earlier one vacated — the one two
        // commits back, since a tail is freed into `pg`'s lists only at the repoint
        // that follows its successor — and the file never has to open a page for it
        // (issue #286). `pg`'s lists hold only
        // durable free space — this commit's own frees stay in `to_free` until the
        // repoint — so the bytes overwritten are already unreachable from the
        // on-disk root, the guarantee [`reserve`](Self::reserve) relies on.
        //
        // The tail is sized from the very lists it draws on, so its length and its
        // placement define each other: reserving space removes sections, and fewer
        // sections need fewer bytes to record. `tail_layout` settles that by
        // proposing a length, planning against it, and accepting the proposal only
        // when the plan comes out exactly that long — no shorter, which would leave
        // untracked bytes inside the reservation, and no longer, which would run
        // past it into whatever lives next. A few rounds settle it; a proposal that
        // will not converge falls through to the append below, which has no length
        // to satisfy.
        let placed = self.tail_layout(&to_free, &old_blocks, ext_len, page_size, os);
        if placed.is_none() && placement == TailPlacement::ReuseOnly {
            // As on the flat path: the shrink pass opens no page of its own.
            return Ok(());
        }
        let reused = placed.is_some();
        let (post, plan, ext_addr, blocks_len, eoa) = match placed {
            Some(layout) => layout,
            None => {
                // Nothing fits: open a metadata page at end-of-file. `pad_to_page`
                // above already left the file page-aligned, so this only records the
                // tail page's type. No trailing run is released here — whatever
                // free space reaches end-of-file was too small for the tail, and
                // stays recorded below it.
                self.begin_page(PageType::Meta)?;
                let at = self.image.len();
                let post = self.paged_post_free(&to_free, &old_blocks);
                let plan = plan_paged_managers(
                    &self.persisted_sections(&post.meta),
                    &self.persisted_sections(&post.raw),
                    &self.persisted_sections(&post.unclassified),
                    page_size,
                    self.persisted_address(at + ext_len),
                    os,
                );
                let blocks_len = plan.end_of_managers.get().max(at + ext_len) - at;
                (post, plan, at, blocks_len, at)
            }
        };
        let final_eof = if reused {
            // The tail landed inside the file; the end-of-allocation is where this
            // commit's appends left it — page-aligned by `pad_to_page`, less the
            // whole pages of free space reaching it, which `post` no longer
            // records and the truncation below gives back (issue #418).
            eoa
        } else {
            align_up(ext_addr + blocks_len, page_size)
        };
        debug_assert!(
            ext_addr + blocks_len <= final_eof,
            "the tail must end at or below the end-of-allocation it was placed under"
        );

        let ext_oh = if plan.is_empty() {
            // No free space to track: an empty persist message, page-aligned.
            let info = FileSpaceInfo::persistent_empty(strategy, threshold, page_size);
            build_v2_object_header(&self.rewrite_extension_region(old_ext_addr, &info)?)?
        } else {
            // Paged convention (matching the from-scratch writer): the managers are
            // ordinary metadata below a page-aligned end-of-allocation.
            let info = FileSpaceInfo::persistent_managers(
                strategy,
                threshold,
                page_size,
                plan.slots.map(StoredAddress::get),
                final_eof,
            );
            build_v2_object_header(&self.rewrite_extension_region(old_ext_addr, &info)?)?
        };
        debug_assert_eq!(
            ext_oh.len() as u64,
            ext_len,
            "extension length must be stable across the placeholder and real messages"
        );

        // Write the extension, then every manager block. Both forms are safe against
        // a crash here: an appended tail is past everything live, and a reused one
        // sits in space an earlier commit freed. Neither is referenced until the
        // repoint below.
        let region = (ext_addr, blocks_len);
        self.write_tail_block(region, ext_addr, &ext_oh)?;
        for b in &plan.blocks {
            let (fshd, fsse) =
                serialize_file_fsm(&b.sections, b.fshd_addr, b.fsse_addr, os, b.class);
            self.write_tail_block(region, base.absolute(b.fshd_addr)?, &fshd)?;
            self.write_tail_block(region, base.absolute(b.fsse_addr)?, &fsse)?;
        }
        // An appended tail ends mid-page; pad it out, so the end-of-allocation stays
        // a whole number of pages. A reused tail is already inside the file, and
        // matched its reservation exactly, so there is nothing to pad — and where
        // it released a trailing run its end-of-allocation is *below* the current
        // end-of-file, which the truncation after the repoint gives back.
        if !reused {
            self.pad_zeros_to(final_eof)?;
        }
        // Exactly the bytes written, contiguous from the extension. A session that
        // reopens this file records the same extents from the message and the
        // manager headers, so nothing depends on remembering this across a close.
        let new_old_blocks = vec![(ext_addr, blocks_len)];

        // Barrier, then repoint the superblock (root, eof, and the new extension)
        // — the linearization point — and sync it.
        self.barrier()?;
        let mut new_sb = self.superblock.clone();
        new_sb.root_group_address = new_root;
        new_sb.eof_address = final_eof;
        new_sb.superblock_extension_address = Some(ext_addr);
        // The one publish site a page-buffered session actually reaches, and so
        // the one where this must not be a literal zero: a crash mark has to
        // outlive every commit the session makes (issue #308).
        new_sb.consistency_flags = self.held_status_flags;
        let sb_bytes = new_sb.serialize();
        self.publish_attempted = true;
        self.write_at(self.sb_sig_off, &sb_bytes)?;
        self.barrier()?;
        self.superblock = new_sb;

        // Physically shrink the file only after the superblock — now carrying the
        // smaller end-of-file — is durable, for the reason the flat tail gives:
        // a crash between the two leaves trailing bytes the next open ignores,
        // where the reverse order would advertise an end-of-file past the file's
        // actual length.
        if final_eof < self.image.len() {
            self.image.truncate(final_eof)?;
            self.barrier()?;
        }

        // The repoint is durable. Only now are this commit's vacated regions
        // genuinely free, so adopt the lists built above and drop the padding tails
        // they already account for. The blocks just written become the ones the next
        // commit supersedes.
        if let Some(pg) = self.paged.as_mut() {
            pg.meta = post.meta;
            pg.raw = post.raw;
            pg.dead = post.dead;
            // This list is otherwise constant for the session, but a released
            // trailing run may have cut into it, and a section naming bytes the
            // file no longer has would be written back to the managers by the
            // next commit.
            pg.unclassified = post.unclassified;
            pg.meta_pad.clear();
            pg.raw_pad.clear();
            if !reused {
                // The tail page is fresh metadata: the managers sit in it, so a
                // following append of metadata may keep packing that page, and the
                // rest of it is free metadata this session can spend. The managers
                // just written cannot say so — they would have to describe space
                // whose size their own length decides — so the next commit records
                // it, from here. A reused tail leaves end-of-file where the data put
                // it, so the outgoing page type stands and there is no padding.
                pg.last = Some(PageType::Meta);
                pg.meta
                    .free(ext_addr + blocks_len, final_eof - (ext_addr + blocks_len));
            }
        }
        self.persist = Some(PersistState {
            strategy,
            threshold,
            page_size,
            old_blocks: new_old_blocks,
        });
        // The managers describe the file at this length, so nothing is owed until
        // it grows past them again.
        self.fsm_len = self.image.len();
        Ok(())
    }

    /// Pad a paged file to a page boundary if its tail page is partially filled,
    /// recording the padding as free space of the tail page's type. A no-op on a
    /// non-paged file or an already-aligned one.
    fn pad_to_page(&mut self) -> Result<(), Error> {
        let len = self.image.len();
        let pad = match &self.paged {
            Some(pg) if len % pg.page_size != 0 => {
                Some((pg.last, pg.page_size - len % pg.page_size))
            }
            _ => None,
        };
        if let Some((last, pad_len)) = pad {
            let pad_at = len;
            self.append(&vec![0u8; pad_len.to_usize()?])?;
            if let Some(pg) = self.paged.as_mut() {
                match last {
                    // A partially-filled tail page at commit time is a raw page:
                    // the last thing the apply loop writes for a dataset is its
                    // header, but a commit that only wrote raw data ends on one.
                    Some(PageType::Meta) => pg.meta_pad.push((pad_at, pad_len)),
                    Some(PageType::Raw) => pg.raw_pad.push((pad_at, pad_len)),
                    // No typed append this commit, so the tail page is one a
                    // previous session left non-aligned — a crash, since a clean
                    // close pads. Its type is unknown, and recording the padding
                    // under a guess would advertise it for reuse of that type and
                    // mix the page, so leave it untracked (see `PagedEdit::begin`,
                    // which makes the same call for the same reason). Reuse made
                    // this reachable: before it, every commit appended at least the
                    // root group header, so `last` was always known here.
                    None => {}
                }
            }
        }
        Ok(())
    }

    /// The free lists a paged commit is about to persist: the session's durable
    /// lists plus the page-padding tails this commit's appends left behind, the
    /// regions it vacated, and the superseded extension and manager blocks (all
    /// metadata, dead once the superblock is repointed).
    ///
    /// Every page the result leaves wholly empty is promoted to a free page before
    /// it is returned ([`PagedEdit::promote_whole_free_pages`]), so the managers
    /// this commit writes already describe it. Deferring the promotion to the
    /// repoint would leave the session holding free space its own on-disk record
    /// does not name.
    ///
    /// Returned as temporaries rather than folded into the session, exactly as the
    /// flat path builds `post`: every region gathered here is still *live* until
    /// that repoint, so the session's own lists must not learn about it until the
    /// repoint succeeds. Everything in between can fail (the extension rewrite,
    /// each write, each barrier), and a session that survived a failed commit
    /// while believing live extents were free would hand them out on the next
    /// commit — silently destroying the objects still occupying them.
    ///
    /// Called once to *size* the tail and again for each round the tail's
    /// placement takes to settle, since what remains depends on the space the tail
    /// has drawn for itself ([`tail_layout`](Self::tail_layout)). Each call clones
    /// both lists; they hold one region per hole, not per byte, so the copies are
    /// small.
    fn paged_post_free(
        &self,
        to_free: &[(u64, u64, FreeClass)],
        old_blocks: &[(u64, u64)],
    ) -> PagedPostFree {
        let pg = self
            .paged
            .as_ref()
            .expect("commit_persisting_paged is only called on a paged file");
        let (mut meta, mut raw, mut dead) = (pg.meta.clone(), pg.raw.clone(), pg.dead.clone());
        // This engine never frees into that list, and nothing it places comes out
        // of it.
        let unclassified = pg.unclassified.clone();
        let mut free = |a: u64, l: u64, class: FreeClass| {
            PagedEdit::route_free(&mut meta, &mut raw, &mut dead, a, l, class);
        };
        for &(a, l) in &pg.meta_pad {
            free(a, l, PageType::Meta.into());
        }
        for &(a, l) in &pg.raw_pad {
            free(a, l, PageType::Raw.into());
        }
        for &(a, l, class) in to_free {
            free(a, l, class);
        }
        // The superseded extension and manager blocks, which are metadata wherever
        // they sat: the tail is placed as metadata, and a page it had to open for
        // itself was opened as metadata too.
        for &(a, l) in old_blocks {
            free(a, l, PageType::Meta.into());
        }
        PagedEdit::promote_whole_free_pages(&mut meta, &mut raw, &mut dead, pg.page_size);
        PagedPostFree {
            meta,
            raw,
            dead,
            unclassified,
        }
    }

    /// Reserve free metadata space for a paged commit's tail and plan the manager
    /// blocks for the address it got, or `None` when nothing reusable fits — the
    /// caller then appends instead.
    ///
    /// The reservation and the plan are mutually dependent: drawing `len` bytes out
    /// of the free lists changes the sections those very blocks record, and a
    /// different section set is a different length. This proposes a length, plans
    /// against the lists as they stand once that length is taken, and accepts only
    /// an exact agreement. Anything else hands the reservation back and proposes
    /// the length the plan just came out at, which is the natural next candidate.
    ///
    /// Both kinds of disagreement have to be rejected, not just the obvious one. A
    /// plan **longer** than its reservation would write past it into whatever lives
    /// after. A plan **shorter** would leave bytes inside the reservation that no
    /// manager records and the next commit does not free — the leak this whole
    /// change exists to remove, reintroduced a few bytes at a time.
    ///
    /// Iteration is capped rather than trusted to converge: the length is a
    /// step function of the section set, so a proposal can in principle oscillate
    /// between two values that each imply the other. Appending is always available
    /// and always correct, so giving up costs a page rather than a guarantee.
    fn tail_layout(
        &mut self,
        to_free: &[(u64, u64, FreeClass)],
        old_blocks: &[(u64, u64)],
        ext_len: u64,
        page_size: u64,
        os: u8,
    ) -> Option<(PagedPostFree, PagedManagerPlan, u64, u64, u64)> {
        /// Enough rounds for the section set to settle after a reservation shrinks
        /// it, without letting an oscillating proposal spin.
        const ROUNDS: usize = 4;

        let eof = self.image.len();

        // The first proposal: what the blocks would measure with nothing reserved.
        // A manager block's length depends only on the sections it records, never
        // on its address, so a start of 0 measures the blocks alone.
        let probe = self.paged_post_free(to_free, old_blocks);
        let mut proposed = ext_len
            + plan_paged_managers(
                &self.persisted_sections(&probe.meta),
                &self.persisted_sections(&probe.raw),
                &self.persisted_sections(&probe.unclassified),
                page_size,
                StoredAddress::new(0),
                os,
            )
            .end_of_managers
            .get();

        for _ in 0..ROUNDS {
            let pg = self
                .paged
                .as_mut()
                .expect("commit_persisting_paged is only called on a paged file");
            let at = pg.alloc_typed(proposed, PageType::Meta)?;
            let mut post = self.paged_post_free(to_free, old_blocks);
            // Whole free pages at the end of the file are released rather than
            // recorded, so the sections these blocks are sized from must already
            // leave them out (issue #418). The reservation was taken before this
            // list was built, so the run can only begin at or above the tail's own
            // end.
            let eoa = post.release_trailing(eof, page_size, proposed);
            // Class the free space into its managers and place their blocks after
            // the extension. Shared with the bounded backend so both lay out
            // identically.
            let plan = plan_paged_managers(
                &self.persisted_sections(&post.meta),
                &self.persisted_sections(&post.raw),
                &self.persisted_sections(&post.unclassified),
                page_size,
                self.persisted_address(at + ext_len),
                os,
            );
            // An empty plan writes no blocks at all, leaving the tail the extension
            // alone; `end_of_managers` is then its own start.
            let blocks_len = plan.end_of_managers.get().max(at + ext_len) - at;
            if blocks_len == proposed {
                debug_assert!(
                    at + blocks_len <= eoa,
                    "a reused tail at {at} of {blocks_len} bytes runs past the end-of-allocation \
                     {eoa} it was placed under"
                );
                return Some((post, plan, at, blocks_len, eoa));
            }
            let pg = self
                .paged
                .as_mut()
                .expect("the paged state outlives this loop");
            PagedEdit::route_free(
                &mut pg.meta,
                &mut pg.raw,
                &mut pg.dead,
                at,
                proposed,
                PageType::Meta.into(),
            );
            proposed = blocks_len;
        }
        None
    }

    /// Write one block of a persisting commit's tail — paged or flat — at the
    /// address its layout gave it, which is either the current end-of-file (the
    /// tail is being appended) or inside `region`, the span reserved from the free
    /// lists, when the tail is being reused into space an earlier commit left.
    ///
    /// The region is checked rather than asserted, for the reason
    /// [`place`](Self::place) checks its own reservation: a reused span is followed
    /// by live bytes, so a block running past it would silently destroy a
    /// neighboring object, and this is the one write in a commit that lands in the
    /// middle of a live file. The region's length is derived from the same layout
    /// that placed these blocks, so the two cannot disagree today; the comparison
    /// is what keeps that true if a layout ever learns a length this sizing does
    /// not model.
    fn write_tail_block(
        &mut self,
        region: (u64, u64),
        addr: u64,
        bytes: &[u8],
    ) -> Result<(), Error> {
        let (start, len) = region;
        if addr < start || addr + bytes.len() as u64 > start + len {
            return Err(Error::Format(FormatError::SerializationError(format!(
                "a commit's tail reserved [{start}, {}) but placed {} bytes at {addr}",
                start + len,
                bytes.len()
            ))));
        }
        if addr == self.image.len() {
            let written = self.append(bytes)?;
            debug_assert_eq!(written, addr, "an appended block must land at end-of-file");
            return Ok(());
        }
        self.write_at(addr, bytes)
    }

    /// Extend the file with zeros up to `target` (>= the current length), used by
    /// the paged tail to pad the final metadata page to its boundary.
    fn pad_zeros_to(&mut self, target: u64) -> Result<(), Error> {
        let len = self.image.len();
        if target > len {
            let pad = (target - len).to_usize()?;
            self.append(&vec![0u8; pad])?;
        }
        debug_assert_eq!(self.image.len(), target);
        Ok(())
    }

    /// Rebuilds the superblock-extension object header's message region with its
    /// File Space Info message replaced by `info` (every other message preserved
    /// verbatim), ready to wrap with [`build_v2_object_header`]. The persisting
    /// message is fixed-size, so this never changes the region's length.
    ///
    /// `ext_addr` is the extension header's absolute file offset, which is where
    /// the messages are read from, and not the address the superblock stores.
    fn rewrite_extension_region(
        &self,
        ext_addr: u64,
        info: &FileSpaceInfo,
    ) -> Result<OhRegion, Error> {
        let region =
            Self::gather_oh_messages(&self.image(), ext_addr, self.superblock.base_address)?;
        rewrite_extension_region_bytes(&region, info)
    }

    /// Repoint a version 0/1 superblock at the rebuilt (now v2) root group and
    /// update its end-of-file field, patching the raw bytes in place — these
    /// superblocks carry no checksum. The root symbol-table entry is switched to
    /// cache type 0 (its scratch-pad B-tree / local-heap addresses, which
    /// describe the old symbol-table group, no longer apply). The
    /// object-header-address write is done last so it is the linearization point.
    fn repoint_v0v1_root(&mut self, new_root: u64, new_eof: u64) -> Result<(), Error> {
        let os = self.superblock.offset_size;
        let addr_width = u64::from(os);
        // Field layout after the fixed prefix: base / free-space / EOF / driver
        // addresses, then the root symbol-table entry (link-name offset, object
        // header address, cache type(4), reserved(4), scratch(16)). The prefix is
        // 24 bytes for v0 and 28 for v1 (the latter adds indexed-storage-K).
        let var_start = if self.superblock.version == 0 { 24 } else { 28 };
        let base = self.sb_sig_off + var_start;
        let eof_off = base + 2 * addr_width;
        let ste = base + 4 * addr_width;
        let oh_addr_off = ste + addr_width;
        let cache_off = ste + 2 * addr_width;
        self.write_at(eof_off, &new_eof.to_le_bytes()[..usize::from(os)])?;
        self.write_at(cache_off, &[0u8; 4])?; // cache type = none
        self.write_at(cache_off + 8, &[0u8; 16])?; // clear scratch-pad
        self.write_at(oh_addr_off, &new_root.to_le_bytes()[..usize::from(os)])?;
        Ok(())
    }

    /// Collect every message of the object header at `addr` into one contiguous
    /// region, following continuation blocks across chunks and dropping the
    /// `Continuation` messages themselves. Re-emitting the result through
    /// [`build_v2_object_header`] collapses a multi-chunk header (as the
    /// reference C library often writes) into a single chunk, which is how this
    /// editor rebuilds headers. The chunk-0 prefix is validated by
    /// [`oh_region_at`]; each continuation block must be a well-formed `OCHK`
    /// block within the file.
    ///
    /// Reads each header chunk out of `src` as one bounded buffer rather than
    /// indexing a whole-file image, so this serves a session whose file is not
    /// mirrored in memory (issue #198).
    fn gather_oh_messages<S: Source + ?Sized>(
        src: &S,
        addr: u64,
        base: BaseAddress,
    ) -> Result<OhRegion, Error> {
        let chunks = read_oh_chunks(src, addr, base)?;
        // Every chunk of one header shares what chunk 0's prefix declared, which
        // is what makes concatenating their records into one region well defined,
        // and what carries the prefix's optional blocks through to the rebuild.
        let mut out = OhRegion::empty(chunks[0].props());
        for chunk in &chunks {
            let layout = chunk.layout();
            let (region, mut p) = chunk.message_region();
            while let Some((msg_type, _body, body_end)) = layout.next_message(region, p)? {
                if msg_type != MessageType::OBJECT_HEADER_CONTINUATION {
                    out.push_bytes(&region[p..body_end]);
                }
                p = body_end;
            }
        }
        Ok(out)
    }

    /// Reconstructs a version-1 (symbol-table) group as a fresh v2 compact-link
    /// message region: a LinkInfo message, one Link message per existing child,
    /// and the group's existing attributes (re-wrapped as v2 messages). The
    /// symbol-table message and other non-link/non-attribute messages
    /// (modification time, comment, …) are dropped — editing a v0/v1 group
    /// converts it to the latest format. Refuses an attribute it cannot
    /// reproduce (shared, or larger than a v2 message can hold).
    fn reconstruct_v1_group(&self, addr: u64) -> Result<GroupInfo, Error> {
        let os = self.superblock.offset_size;
        let ls = self.superblock.length_size;
        let base = self.superblock.base_address;
        let oh = ObjectHeader::parse_from_source(
            &self.image(),
            AccessMode::ReadWrite,
            addr,
            os,
            ls,
            base,
        )?;
        if oh
            .messages
            .iter()
            .any(|m| m.msg_type == MessageType::DATA_LAYOUT)
        {
            return Err(Error::EditUnsupported(
                "a target path names a dataset, not a group",
            ));
        }
        let entries = resolve_group_entries_from_source(&self.image(), &oh, os, ls, base)?;

        let mut region = fresh_group_region();
        let mut link_names = Vec::with_capacity(entries.len());
        for e in &entries {
            // A version 1 group has no link creation order, which the Link Info
            // message introduced, so these links get no creation index and the
            // Link Info message `fresh_group_region` writes tracks none.
            region.push_link(&e.name, e.object_header_address, None);
            link_names.push(e.name.clone());
        }
        for m in &oh.messages {
            if m.msg_type == MessageType::ATTRIBUTE {
                if m.data.len() > OBJECT_HEADER_MESSAGE_MAX {
                    return Err(Error::EditUnsupported(
                        "a v0/v1 group attribute is too large to convert in place",
                    ));
                }
                if !m.flags.difference(MessageFlags::SHARED).is_empty() {
                    // Every other flag bit says something about the message this
                    // rewrite would have to reproduce and does not — that it is
                    // constant, that it must not be shared, what a reader should
                    // do if it cannot decode it. The shared bit is the one this
                    // conversion knows how to carry.
                    return Err(Error::EditUnsupported(
                        "a v0/v1 group attribute message carries an object-header flag this \
                         conversion cannot reproduce",
                    ));
                }
                if m.flags.is_shared() {
                    // The body is a reference, not an attribute; the reference is
                    // what gets rewrapped, in the encoding a version 2 header
                    // uses. Reachable only from a file whose version 1 objects
                    // share messages, which the reference C library does not
                    // write — it moves an object to a version 2 header before
                    // sharing anything on it.
                    region.push_shared(
                        MessageType::ATTRIBUTE,
                        &modernize_shared_reference(&m.data, os, ls)?,
                    );
                    continue;
                }
                // Re-wrap the attribute message body (it is self-describing) in a
                // v2 message record. The rebuilt header does not track creation
                // order — a version 1 header cannot have carried any — so the
                // attribute needs no creation index.
                region.push(MessageType::ATTRIBUTE, &m.data);
            }
        }
        Ok(GroupInfo { region, link_names })
    }

    /// Parses and validates a group's object header, returning its message region
    /// — the bytes to copy when rewriting the header — and the names of its
    /// existing links. A version 2 header is rebuilt from its own message bytes
    /// (collapsing continuation chunks, preserving every message); a version 1
    /// symbol-table group is converted to v2 via [`reconstruct_v1_group`].
    ///
    /// [`reconstruct_v1_group`]: Self::reconstruct_v1_group
    fn inspect_group(&self, addr: u64) -> Result<GroupInfo, Error> {
        let sig = self.image().read_metadata_at(addr, 4);
        if sig.as_deref() != Ok(&b"OHDR"[..]) {
            return self.reconstruct_v1_group(addr);
        }
        let mut region =
            Self::gather_oh_messages(&self.image(), addr, self.superblock.base_address)?;
        let mut p = 0;
        let mut has_link_info = false;
        let mut link_names = Vec::new();
        while let Some((msg_type, body, body_end)) = region.next_message(p)? {
            match msg_type {
                MessageType::LINK_INFO => {
                    has_link_info = true;
                    // LinkInfo: version(1) flags(1) [max_creation_index(8) if
                    // flags&0x01] fractal_heap_addr(8) … — dense storage has a
                    // defined fractal-heap address. Bound the read by this
                    // message's own body, not just the region, so a short or
                    // malformed LinkInfo can't make us read the next message.
                    let mut q = body + 2;
                    if body_end - body >= 2 && region[body + 1] & 0x01 != 0 {
                        q += 8;
                    }
                    if q + 8 <= body_end {
                        let heap_addr = u64::from_le_bytes(region[q..q + 8].try_into().unwrap());
                        if heap_addr != u64::MAX {
                            return Err(Error::EditUnsupported(
                                "a target group uses dense (fractal-heap) link storage (not supported in place yet)",
                            ));
                        }
                    }
                }
                MessageType::LINK => {
                    if let Ok(link) = LinkMessage::parse(&region[body..body_end], OFFSET_SIZE) {
                        link_names.push(link.name);
                    }
                }
                MessageType::DATA_LAYOUT => {
                    return Err(Error::EditUnsupported(
                        "a target path names a dataset, not a group",
                    ));
                }
                _ => {}
            }
            p = body_end;
        }
        if !has_link_info {
            return Err(Error::EditUnsupported(
                "a target group's object header has no link-info message",
            ));
        }
        // Heal headers written by older hdf5-pure releases that omitted the
        // Group Info message, so the rewritten group stays writable by the C
        // library.
        ensure_group_info(&mut region)?;
        Ok(GroupInfo { region, link_names })
    }

    /// The refusals a staged value overwrite (`write_dataset`) makes from the
    /// staged builder alone, with no on-disk header involved.
    ///
    /// [`stage_dataset_write`](Self::stage_dataset_write) applies these as the
    /// write is staged, so the call that configured the builder is the one that
    /// reports the mistake. A refusal that reads only `fd` belongs here; one that
    /// needs the target's header belongs in
    /// [`prepare_write`](Self::prepare_write).
    fn refuse_unsupported_overwrite(fd: &FlatDataset) -> Result<(), Error> {
        // A value overwrite never introduces chunking, filters, or an extensible
        // shape: those would change the storage layout, not just the bytes.
        if fd.chunk_options.is_chunked() || fd.maxshape.is_some() {
            return Err(Error::EditUnsupported(
                "write_dataset overwrites values only; it cannot make a dataset \
                 chunked, filtered, or extensible",
            ));
        }

        // `write_dataset` overwrites element bytes only; it does not touch the
        // object header's attribute messages. Attributes staged on the returned
        // builder would otherwise be silently dropped (the in-place path rewrites
        // only the data block, and the moving path reuses the verbatim on-disk
        // header), so refuse rather than degrade — set them in a separate edit.
        if !fd.attrs.is_empty() {
            return Err(Error::EditUnsupported(
                "write_dataset overwrites values only; it cannot set attributes \
                 (set them with a separate edit)",
            ));
        }

        // `write_dataset` overwrites element bytes only; it reuses the dataset's
        // existing Fill Value message (the in-place path rewrites only the data
        // block, and the moving path keeps every header message but the layout
        // verbatim). A fill value staged on the returned builder would otherwise
        // be silently ignored, so refuse rather than degrade — set the fill value
        // when the dataset is first created.
        if fd.fill.is_some() {
            return Err(Error::EditUnsupported(
                "write_dataset overwrites values only; it cannot change the fill \
                 value (set it when the dataset is created)",
            ));
        }

        // `with_vlen_strings` stages placeholder element references, resolved by
        // placing their global heap collections and patching the addresses in
        // (issue #321). That happens where each plan is written, not here: see
        // `OverwriteBytes`.

        // `reference_targets` holds elements only the add path resolves, in
        // `preflight_reference_targets` and the apply loop after it; the
        // overwrite path never reads the field. Checked on the field rather than
        // on `with_path_references`, since every producer stages elements that
        // are equally unresolved. Left unrefused, a staged overwrite writes
        // address-zero placeholders over a working reference dataset and
        // `commit` reports `Ok` (issue #318). `with_reference_data` supplies
        // resolved addresses and overwrites like any other value.
        if fd.reference_targets.is_some() {
            return Err(Error::EditUnsupported(
                "write_dataset cannot overwrite an object-reference dataset's \
                 data in place yet",
            ));
        }

        Ok(())
    }

    /// Preflight a staged value overwrite (`write_dataset`): resolve the dataset
    /// at `addr`, validate that the staged `fd` matches it byte-exactly in
    /// datatype and shape, and classify how the bytes will be applied. No file
    /// bytes are written here — this is part of the all-or-nothing preflight, so a
    /// rejected write leaves the commit unapplied.
    ///
    /// Contiguous, compact, and chunked (including filtered) datasets are all
    /// supported; the chunk geometry, filter pipeline, and chunk index come from
    /// the on-disk header (a chunk index this engine cannot enumerate — a
    /// version-2 B-tree — is refused). A datatype or shape that differs from the
    /// on-disk dataset's is likewise refused — this is a value overwrite, not a
    /// reshape or retype.
    ///
    /// Every refusal `fd` can decide on its own has been made by then, where the
    /// write is staged.
    fn prepare_write<S: Source + ?Sized>(
        src: &S,
        addr: u64,
        fd: &FlatDataset,
        base: BaseAddress,
        path: &ObjectPathBuf,
    ) -> Result<WritePlan, Error> {
        // Enforced by construction: `staged.writes` has one producer, and it
        // refuses there. Asserted rather than re-refused because a second caller
        // that skipped it would not fail here — a chunked builder would fall
        // through and be written as a plain contiguous overwrite, with
        // `chunk_options` silently dropped.
        debug_assert!(
            Self::refuse_unsupported_overwrite(fd).is_ok(),
            "a staged write reached prepare_write without refuse_unsupported_overwrite"
        );

        let region = Self::gather_oh_messages(src, addr, base)?;

        // Locate the datatype, dataspace, and data-layout messages, and detect a
        // filter pipeline (filtered storage is always chunked, never contiguous).
        let mut datatype: Option<(usize, usize)> = None;
        let mut dataspace: Option<(usize, usize)> = None;
        let mut layout: Option<(usize, usize)> = None;
        let mut filter: Option<(usize, usize)> = None;
        // The dataset's own fill value, which the edge overhang of a partial
        // chunk must hold (issue #296). Both message forms, newest wins.
        let mut fill_msg: Option<(MessageType, usize, usize)> = None;
        let mut has_link = false;
        let mut p = 0;
        while let Some((msg_type, body, body_end)) = region.next_message(p)? {
            match msg_type {
                MessageType::DATATYPE => datatype = Some((body, body_end)),
                MessageType::DATASPACE => dataspace = Some((body, body_end)),
                MessageType::DATA_LAYOUT => layout = Some((body, body_end)),
                MessageType::FILTER_PIPELINE => filter = Some((body, body_end)),
                // Versioned beats legacy, and within a type the first wins —
                // the same rule `Dataset::fill_bytes` and `Located::from_walk`
                // apply, so all three agree on a header carrying more than one.
                MessageType::FILL_VALUE
                    if !matches!(fill_msg, Some((MessageType::FILL_VALUE, ..))) =>
                {
                    fill_msg = Some((msg_type, body, body_end));
                }
                MessageType::FILL_VALUE_OLD if fill_msg.is_none() => {
                    fill_msg = Some((msg_type, body, body_end));
                }
                MessageType::LINK | MessageType::LINK_INFO | MessageType::SYMBOL_TABLE => {
                    has_link = true;
                }
                _ => {}
            }
            p = body_end;
        }

        if has_link {
            return Err(Error::EditUnsupported(
                "write_dataset target is a group, not a dataset",
            ));
        }
        let (dt_b, dt_e) =
            datatype.ok_or(Error::EditUnsupported("dataset header has no datatype"))?;
        let (ds_b, ds_e) =
            dataspace.ok_or(Error::EditUnsupported("dataset header has no dataspace"))?;
        let (lb, le) = layout.ok_or(Error::EditUnsupported("dataset header has no data layout"))?;

        // Compare datatype and shape structurally against the staged data. A
        // value overwrite must keep both exactly: the datatype (including its
        // class, size, endianness, and any compound/array/enumeration layout) so
        // the bytes are interpreted the same, and the *current* dimensions so the
        // byte count is unchanged. Parsing both sides and comparing the decoded
        // values — rather than the raw message bytes — tolerates the harmless
        // encoding differences between this crate's writer and the reference C
        // library (e.g. the C library records a maximum-dimensions array equal to
        // the current dimensions, which this crate omits) while still refusing any
        // real retype or reshape.
        let (disk_dt, _) = Datatype::parse(&region[dt_b..dt_e])?;
        if disk_dt != fd.dt {
            return Err(Error::EditUnsupported(
                "write_dataset datatype does not match the on-disk dataset (overwrite, not retype)",
            ));
        }
        let disk_ds = Dataspace::parse(&region[ds_b..ds_e], LENGTH_SIZE)?;
        if disk_ds.space_type != fd.ds.space_type
            || disk_ds.rank != fd.ds.rank
            || disk_ds.dimensions != fd.ds.dimensions
        {
            return Err(Error::EditUnsupported(
                "write_dataset shape does not match the on-disk dataset (overwrite, not reshape)",
            ));
        }

        // Classify the layout. Version 3/4 compact (class 0), contiguous (class
        // 1), and chunked (class 2) are supported; an old-version layout or a
        // virtual layout (class 3) is refused.
        if le - lb < 2 {
            return Err(Error::EditUnsupported("malformed data-layout message"));
        }
        let version = region[lb];
        if version != 3 && version != 4 {
            return Err(Error::EditUnsupported(
                "an unsupported data-layout version cannot be overwritten in place yet",
            ));
        }
        match region[lb + 1] {
            // Compact: the data is inline in the header. Rebuild the header with
            // the new inline bytes (relocating it), patching the parent link.
            0 => Ok(WritePlan::Moving(MovingWrite::Compact {
                region,
                bytes: staged_bytes(fd, path),
            })),
            1 => {
                if le - lb < 18 {
                    return Err(Error::EditUnsupported("malformed contiguous data layout"));
                }
                let addr_off = lb + 2;
                let data_addr = StoredAddress::new(u64::from_le_bytes(
                    region[addr_off..addr_off + 8].try_into().unwrap(),
                ));
                let data_size = u64::from_le_bytes(region[lb + 10..lb + 18].try_into().unwrap());

                // Same length and a defined, in-bounds data block: overwrite the
                // bytes straight in place. No header rewrite, no relink. The
                // in-place write targets the data block's absolute file offset.
                //
                // Never for a staged variable-length overwrite, which relocates
                // instead. Its resolution *allocates* — a global heap collection,
                // drawn from a freed region where one fits — and an in-place write
                // is a live mutation of a block the current root already reaches.
                // Together those put a reference to a just-allocated span inside
                // bytes the live tree names, so a commit failing after the write
                // and before its repoint hands the span back to the free list
                // (`restore_free`) while the image still points into it. The next
                // commit is then free to place something else there, and the
                // dataset reads that object's bytes with every checksum intact.
                //
                // A relocating write has no such window: its new data block is
                // reachable from nothing until the superblock repoint, so a
                // failed attempt leaves the region genuinely dead and re-offering
                // it is sound — which is the invariant `restore_free` states.
                if fd.vl_string_staging.is_none()
                    && !data_addr.is_undefined(OFFSET_SIZE)
                    && data_size == fd.raw.len() as u64
                {
                    if let Ok(start) = base.absolute(data_addr) {
                        if start.checked_add(data_size).is_some_and(|e| e <= src.len()) {
                            return Ok(WritePlan::InPlace {
                                data_addr: start,
                                bytes: staged_bytes(fd, path),
                            });
                        }
                    }
                }

                // Length differs or the block was undefined/out of bounds: the new
                // data goes elsewhere and the old extent (if any) is freed. The
                // freed extent is recorded as an absolute file offset, which is the
                // frame the session free list uses.
                let old_extent = if !data_addr.is_undefined(OFFSET_SIZE) && data_size > 0 {
                    Some((base.absolute(data_addr)?, data_size))
                } else {
                    None
                };
                Ok(WritePlan::Moving(MovingWrite::Contiguous {
                    region,
                    addr_off,
                    bytes: staged_bytes(fd, path),
                    old_extent,
                }))
            }
            // Chunked: overwrite each chunk in place when every new (re-encoded)
            // chunk is the same byte length as its slot, else rebuild and relocate
            // the whole chunk storage. The chunk geometry, filter pipeline, and
            // index type all come from the existing on-disk header (the staged
            // builder carries none — chunked/filtered/extensible builders are
            // refused at the top of this function as "not a value overwrite").
            2 => {
                // Chunked overwrite (in-place or relocating). On a userblock file
                // every stored chunk-index and chunk address is relative to `base`:
                // the in-place path below walks the index on a base-relative view of
                // the file and shifts the resulting write offsets back by `base`,
                // and the relocating path rebuilds the chunk blob with stored
                // addresses (see `write_chunked_relocatable`).
                let dl = DataLayout::parse(&region[lb..le], OFFSET_SIZE, LENGTH_SIZE)?;
                let DataLayout::Chunked { index, .. } = dl else {
                    return Err(Error::EditUnsupported("dataset is not chunked"));
                };
                if !index.enumerable() {
                    return Err(Error::EditUnsupported(
                        "a chunked dataset with a version-2 B-tree chunk index cannot be \
                         overwritten in place yet",
                    ));
                }

                let ChunkedGeometry {
                    spatial,
                    element_size,
                    raw_size,
                    maxshape,
                } = chunked_geometry(&fd.dt, &disk_ds, &dl)?;

                // Split the new value into full-size chunk buffers in dense
                // row-major grid order, then re-encode through the on-disk
                // pipeline when the dataset is filtered.
                // The overhang past the dataset's edge holds the dataset's own
                // fill value, not zeros: an allocated chunk is expected to carry
                // it wherever nothing was written, and those slots are what a
                // reader returns once the dataset is extended into them (#296).
                let padding = fill_msg
                    .map_or(crate::fill_value::PaddingFill::Zero, |(mt, b, e)| {
                        crate::fill_value::PaddingFill::from_message(mt, &region[b..e])
                    });
                let pipeline_message: Option<Vec<u8>> =
                    filter.map(|(fb, fe)| region[fb..fe].to_vec());

                // Refuse a pipeline this engine cannot re-encode before anything
                // is split or encoded, so the refusal reaches the preflight on
                // both branches below — including the staged one, which does its
                // splitting in the apply phase and so has no other chance to make
                // it.
                if let Some(pm) = &pipeline_message {
                    let pipeline = FilterPipeline::parse(pm)?;
                    if !pipeline_reencodable(&pipeline) {
                        return Err(Error::EditUnsupported(
                            "a chunked dataset using a filter this engine cannot re-encode \
                             cannot be overwritten in place yet",
                        ));
                    }
                }

                // Element bytes still carrying unresolved variable-length
                // references cannot be split here: a filtered chunk's compressed
                // length depends on the heap addresses patched into it, so the
                // split has to follow the patch, which follows a placement the
                // preflight may not make. See `ChunkPayload::Deferred`.
                if fd.vl_string_staging.is_some() {
                    return Ok(WritePlan::Moving(MovingWrite::Chunked {
                        region,
                        shape: disk_ds.dimensions.clone(),
                        chunk_dims: spatial,
                        element_size,
                        maxshape,
                        pipeline_message,
                        payload: ChunkPayload::Deferred {
                            bytes: staged_bytes(fd, path),
                            padding,
                            dt: fd.dt.clone(),
                        },
                        old_addr: addr,
                    }));
                }

                let new_chunk_bytes = split_and_encode_chunks(
                    &fd.raw,
                    &disk_ds.dimensions,
                    &spatial,
                    element_size,
                    &padding,
                    pipeline_message.as_deref(),
                    &fd.dt,
                )?;

                // Fast path: overwrite each chunk straight in its slot when every
                // new chunk fits. No header rewrite and no superblock flip — the
                // chunk (and index) blocks are reachable from both roots. The index
                // is left untouched when chunks keep their size and rebuilt in place
                // when they shrink. The index walk runs on a base-relative view of
                // the file (so the layout's stored addresses index correctly), and
                // the returned write offsets are shifted back to absolute file
                // offsets by adding `base` (a no-op on a base-0 file).
                if let Some(writes) = try_inplace_chunk_writes(
                    &BaseOffsetSource { inner: src, base },
                    &dl,
                    &disk_ds,
                    &spatial,
                    raw_size,
                    &new_chunk_bytes,
                ) {
                    let writes = writes
                        .into_iter()
                        .map(|(off, b)| Ok((base.absolute(StoredAddress::new(off))?, b)))
                        .collect::<Result<Vec<_>, FormatError>>()?;
                    return Ok(WritePlan::InPlaceChunks { writes });
                }

                // Otherwise relocate: rebuild a fresh chunk blob + index at
                // end-of-file (carrying the re-encoded chunk bytes and the source
                // pipeline verbatim), swap the data-layout message in the verbatim
                // header, and free the old chunk storage after the commit lands.
                Ok(WritePlan::Moving(MovingWrite::Chunked {
                    region,
                    shape: disk_ds.dimensions.clone(),
                    chunk_dims: spatial,
                    element_size,
                    maxshape,
                    pipeline_message,
                    payload: ChunkPayload::Encoded(new_chunk_bytes),
                    old_addr: addr,
                }))
            }
            _ => Err(Error::EditUnsupported(
                "an unsupported data-layout class cannot be overwritten in place yet",
            )),
        }
    }

    /// Plan a relocating append to an existing chunked, unlimited,
    /// Extensible-Array-indexed dataset at `addr`. Validates the target, splits
    /// the appended elements into new (and one rewritten trailing) chunks —
    /// compressed through the on-disk pipeline when filtered — and gathers the
    /// existing complete chunks by metadata alone. Returns the
    /// [`MovingWrite::AppendedChunks`] plan; the commit machinery appends the new
    /// chunks and a rebuilt index and repoints the header (see
    /// [`write_appended_chunks`](Self::write_appended_chunks)).
    ///
    /// Reads only; no bytes are written here. `src` is the file image and `base`
    /// its userblock base; the dataset's stored (base-relative) structures are read
    /// through a `base`-shifted view.
    fn prepare_append<S: Source + ?Sized>(
        src: &S,
        addr: u64,
        ab: &AppendBuilder,
        base: BaseAddress,
    ) -> Result<MovingWrite, Error> {
        if ab.dt_conflict {
            return Err(Error::AppendUnsupported(
                "append mixes element types in one builder; use one element type per \
                 append_dataset call",
            ));
        }

        let region = Self::gather_oh_messages(src, addr, base)?;

        // Locate the datatype, dataspace, data-layout, and filter-pipeline
        // messages, and detect a group (link) header.
        let mut datatype: Option<(usize, usize)> = None;
        let mut dataspace: Option<(usize, usize)> = None;
        let mut layout: Option<(usize, usize)> = None;
        let mut filter: Option<(usize, usize)> = None;
        // The dataset's own fill value, which the edge overhang of a partial
        // chunk must hold (issue #296). Both message forms, newest wins.
        let mut fill_msg: Option<(MessageType, usize, usize)> = None;
        let mut has_link = false;
        let mut p = 0;
        while let Some((msg_type, body, body_end)) = region.next_message(p)? {
            match msg_type {
                MessageType::DATATYPE => datatype = Some((body, body_end)),
                MessageType::DATASPACE => dataspace = Some((body, body_end)),
                MessageType::DATA_LAYOUT => layout = Some((body, body_end)),
                MessageType::FILTER_PIPELINE => filter = Some((body, body_end)),
                // Versioned beats legacy, and within a type the first wins —
                // the same rule `Dataset::fill_bytes` and `Located::from_walk`
                // apply, so all three agree on a header carrying more than one.
                MessageType::FILL_VALUE
                    if !matches!(fill_msg, Some((MessageType::FILL_VALUE, ..))) =>
                {
                    fill_msg = Some((msg_type, body, body_end));
                }
                MessageType::FILL_VALUE_OLD if fill_msg.is_none() => {
                    fill_msg = Some((msg_type, body, body_end));
                }
                MessageType::LINK | MessageType::LINK_INFO | MessageType::SYMBOL_TABLE => {
                    has_link = true;
                }
                _ => {}
            }
            p = body_end;
        }
        if has_link {
            return Err(Error::AppendUnsupported(
                "append target is a group, not a dataset",
            ));
        }
        let (dt_b, dt_e) =
            datatype.ok_or(Error::AppendUnsupported("dataset header has no datatype"))?;
        let (ds_b, ds_e) =
            dataspace.ok_or(Error::AppendUnsupported("dataset header has no dataspace"))?;
        let (lb, le) = layout.ok_or(Error::AppendUnsupported(
            "dataset header has no data layout",
        ))?;

        let (disk_dt, _) = Datatype::parse(&region[dt_b..dt_e])?;
        let disk_ds = Dataspace::parse(&region[ds_b..ds_e], LENGTH_SIZE)?;
        let dl = DataLayout::parse(&region[lb..le], OFFSET_SIZE, LENGTH_SIZE)?;

        // Require a chunked dataset indexed by an Extensible Array, which only a
        // version 4 layout message stores.
        let DataLayout::Chunked { index, .. } = &dl else {
            return Err(Error::AppendUnsupported(
                "append requires a chunked dataset",
            ));
        };
        let ChunkIndexLayout::ExtensibleArray { address } = index else {
            return Err(Error::AppendUnsupported(
                "append requires an Extensible-Array-indexed chunked dataset (a single \
                 unlimited dimension under the latest format)",
            ));
        };

        // Require rank 1, unlimited along axis 0.
        if disk_ds.space_type != DataspaceType::Simple || disk_ds.dimensions.len() != 1 {
            return Err(Error::AppendUnsupported(
                "append requires a rank-1 dataset in this release",
            ));
        }
        if !matches!(
            disk_ds.max_dimensions.as_deref(),
            Some([MaxExtent::Unlimited, ..])
        ) {
            return Err(Error::AppendUnsupported(
                "append requires a dataset that is unlimited along its first dimension",
            ));
        }

        let ChunkedGeometry {
            spatial,
            element_size,
            ..
        } = chunked_geometry(&disk_dt, &disk_ds, &dl)?;
        let chunk_elems = spatial[0];
        if chunk_elems == 0 {
            return Err(Error::AppendUnsupported(
                "append requires a nonzero chunk length",
            ));
        }

        // Validate the appended bytes against the on-disk element type.
        if ab.raw.len() % element_size != 0 {
            return Err(Error::AppendUnsupported(
                "appended byte length is not a whole number of elements",
            ));
        }
        match &ab.elem_dt {
            // A typed append must match the on-disk datatype exactly (class, size,
            // and byte order) — this is a value append, not a retype.
            Some(expected) if *expected != disk_dt => {
                return Err(Error::AppendUnsupported(
                    "append datatype does not match the on-disk dataset (wrong element \
                     type or byte order)",
                ));
            }
            Some(_) => {}
            // A raw append trusts the caller's bytes but still refuses any datatype
            // whose flat little-endian bytes cannot be written verbatim: a
            // big-endian numeric leaf would silently misencode, and a
            // variable-length or reference leaf embeds heap/object addresses a byte
            // append cannot reproduce. A typed append is byte-order- and
            // class-checked by the datatype-equality arm above.
            None => {
                if !datatype_is_raw_appendable(&disk_dt) {
                    return Err(Error::AppendUnsupported(
                        "append_raw onto this dataset's datatype (non-little-endian, \
                         variable-length, or reference) could misencode the bytes; use a \
                         typed append",
                    ));
                }
            }
        }

        let new_elems = (ab.raw.len() / element_size) as u64;
        let current_dim0 = disk_ds.dimensions[0];
        let new_dim0 = current_dim0
            .checked_add(new_elems)
            .ok_or(Error::AppendUnsupported(
                "append would overflow the dataset dimension",
            ))?;

        // The filter pipeline is preserved verbatim in the rebuilt header; parse it
        // to re-encode the new chunks. An engine-unencodable filter is refused.
        let pipeline_message: Option<Vec<u8>> = filter.map(|(fb, fe)| region[fb..fe].to_vec());
        let has_filters = pipeline_message.is_some();
        let pipeline = match &pipeline_message {
            Some(pm) => {
                let parsed = FilterPipeline::parse(pm)?;
                if !pipeline_reencodable(&parsed) {
                    return Err(Error::AppendUnsupported(
                        "dataset uses a filter this engine cannot re-encode",
                    ));
                }
                Some(parsed)
            }
            None => None,
        };

        if base.get() > src.len() {
            return Err(Error::AppendUnsupported(
                "userblock base address past end-of-file",
            ));
        }
        let view = BaseOffsetSource { inner: src, base };

        // The rebuilt index's element format (bare address vs address+size+mask) is
        // chosen by `has_filters`; it must agree with the source index's client id,
        // or the kept chunks — carried by metadata into the new index — would be
        // re-encoded in the wrong element width.
        if let Some(idx_addr) = *address {
            let hdr = ExtensibleArrayHeader::parse_from_source(
                &view,
                idx_addr,
                OFFSET_SIZE,
                LENGTH_SIZE,
            )?;
            if (hdr.client_id == 1) != has_filters {
                return Err(Error::AppendUnsupported(
                    "dataset filter metadata is inconsistent (chunk-index client id \
                     disagrees with the filter pipeline)",
                ));
            }
        }

        // Enumerate the existing chunks (base-relative addresses) and require a
        // dense grid: `plan_dense_grid` returns the chunks in index order and
        // `None` on any hole, duplicate, or count mismatch against the dimension.
        let infos = enumerate_chunks_from_source(&view, &dl, &disk_ds, OFFSET_SIZE, LENGTH_SIZE)?;
        let grid = plan_dense_grid(infos, &disk_ds.dimensions, &spatial).ok_or(
            Error::AppendUnsupported(
                "dataset has a sparse or inconsistent chunk grid; cannot append",
            ),
        )?;
        let grid_order = grid.grid_order;

        // Complete chunks are kept by metadata; a trailing partial chunk (when the
        // current length is not chunk-aligned) is rewritten.
        let n_full = (current_dim0 / chunk_elems).to_usize()?;
        let has_partial = current_dim0 % chunk_elems != 0;

        // Rewriting that chunk decodes and re-encodes committed values, which a
        // *lossy* pipeline does not reproduce: a ZFP block re-quantizes to fit
        // its new contents, and float D-scale packs offsets from a per-chunk
        // minimum that a smaller appended value moves (measured: 1.45 became 1.5
        // at one decimal). Refused here as on the immediate path (#407).
        if has_partial
            && let Some(pl) = &pipeline
            && !pipeline_lossless(pl)
        {
            return Err(Error::AppendUnsupported(LOSSY_TAIL_REFUSAL));
        }

        let mut kept_chunks: Vec<WrittenChunk> = Vec::with_capacity(n_full);
        for ci in grid_order.iter().take(n_full) {
            kept_chunks.push(WrittenChunk {
                address: ci.address,
                compressed_size: ci.chunk_size.get(),
                // Preserve the source mask verbatim: a C/h5py file records a nonzero
                // mask for a chunk whose filter was skipped (e.g. deflate on
                // incompressible data), and forcing it to 0 would corrupt that chunk.
                filter_mask: ci.filter_mask,
            });
        }

        // Build the raw byte region for the tail (from the last chunk boundary to
        // the new end): the live prefix of any rewritten partial chunk, then the
        // appended bytes.
        let mut tail_raw: Vec<u8> = Vec::new();
        let mut old_tail_extent: Option<(u64, u64)> = None;
        if has_partial {
            let partial = &grid_order[n_full];
            let len = partial.chunk_size.to_usize()?;
            partial
                .address
                .get()
                .checked_add(len as u64)
                .filter(|&e| e <= view.len())
                .ok_or(Error::AppendUnsupported(
                    "trailing chunk extends past end-of-file",
                ))?;
            let stored = view.read_exact_at(partial.address.get(), len)?;
            let full = if let Some(pl) = &pipeline {
                let ctx = ChunkContext::from_datatype(&spatial, &disk_dt)?;
                decompress_chunk(&stored, pl, ctx, partial.filter_mask).map_err(Error::Format)?
            } else {
                stored
            };
            let live_elems = (current_dim0 % chunk_elems).to_usize()?;
            let live_bytes = live_elems * element_size.get();
            if full.len() < live_bytes {
                return Err(Error::AppendUnsupported(
                    "trailing chunk decoded shorter than its live element count",
                ));
            }
            tail_raw.extend_from_slice(&full[..live_bytes]);
            // The old partial chunk's data block is dead once the new index lands.
            old_tail_extent = Some((base.absolute(partial.address)?, partial.chunk_size.get()));
        }
        tail_raw.extend_from_slice(&ab.raw);

        // Split the tail into full chunk buffers and compress each through the
        // pipeline when filtered. The final chunk's overhang takes the dataset's
        // fill value, which is what a reader returns from those slots once a
        // later append or resize reaches them (#296).
        let tail_len_elems = new_dim0 - (n_full as u64) * chunk_elems;
        let padding = fill_msg.map_or(crate::fill_value::PaddingFill::Zero, |(mt, b, e)| {
            crate::fill_value::PaddingFill::from_message(mt, &region[b..e])
        });
        let split = split_into_chunks(
            &tail_raw,
            &[tail_len_elems],
            &spatial,
            element_size,
            padding.pattern(element_size),
        )
        .map_err(Error::Format)?;
        let new_chunk_bytes: Vec<Vec<u8>> = if let Some(pl) = &pipeline {
            let ctx = ChunkContext::from_datatype(&spatial, &disk_dt)?;
            let mut out = Vec::with_capacity(split.len());
            // One encoder across the appended tail; see `FilterScratch`.
            let mut scratch = FilterScratch::new();
            for buf in &split {
                out.push(compress_chunk_with(&mut scratch, buf, pl, ctx).map_err(Error::Format)?);
            }
            out
        } else {
            split
        };

        // Grow the dataspace along axis 0, preserving the (unlimited) max-dims.
        let mut grown = disk_ds.clone();
        grown.dimensions[0] = new_dim0;
        let new_dataspace_body = grown.serialize(LENGTH_SIZE);

        let chunk_dims_u32: Vec<u32> = spatial
            .iter()
            .map(|&dim| dim.narrow::<u32>())
            .collect::<Result<_, _>>()?;

        Ok(MovingWrite::AppendedChunks {
            region,
            new_dataspace_body,
            chunk_dims_u32,
            element_size,
            has_filters,
            kept_chunks,
            new_chunk_bytes,
            old_addr: addr,
            old_tail_extent,
        })
    }

    /// Parse the object header at `addr` into a copyable model, validating that
    /// every message can be reproduced faithfully (verbatim message bytes, with
    /// only the contiguous data address and child link targets repointed).
    /// Dense (fractal-heap) attribute storage is read out of the source heap into
    /// a parsed attribute set carried on the model (`dense_attrs`) and re-emitted
    /// into a fresh heap on write, within the bounds that heap declares (see
    /// `file_writer::dense_attrs_check`); an attribute too large to hold as a
    /// managed object is re-emitted as a *huge* object. Rejects multi-chunk
    /// headers, dense or soft/external links, old-version data layouts, external
    /// data storage ([`reject_external_storage`]), and headers that are neither a
    /// dataset nor a group. (A *chunked* layout is modelled, not rejected —
    /// [`ObjModel::DatasetChunked`].)
    fn read_object<S: Source + ?Sized>(
        src: &S,
        access_mode: AccessMode,
        addr: u64,
        base: BaseAddress,
    ) -> Result<ObjModel, Error> {
        let region = Self::gather_oh_messages(src, addr, base)?;

        // Ahead of the layout classification below, where external storage and
        // never-allocated storage would otherwise share one undefined address
        // and one answer — and ahead of the dense-attribute read, so a header
        // this cannot copy says why rather than failing as something else.
        reject_external_storage(&region)?;

        // First pass: detect whether attributes are stored densely (a defined
        // fractal-heap address in the Attribute Info message). A dense object is
        // copied by reading its attributes out of the source heap and rebuilding
        // a fresh heap on write, so its Attribute Info message and any inline
        // Attribute messages are dropped from the verbatim region — the rebuilt
        // region carries neither, and `dense_attrs` carries the parsed set.
        let mut dense = false;
        let mut p = 0;
        while let Some((msg_type, body, body_end)) = region.next_message(p)? {
            if msg_type == MessageType::ATTRIBUTE_INFO {
                // An Attribute Info message does not by itself mean dense
                // storage: the reference C library and h5py emit one (with an
                // *undefined* fractal-heap address) even for compact, inline
                // attributes in the latest format, to carry attribute
                // creation-order metadata. Only a *defined* heap address is real
                // dense (fractal-heap) storage. A message that cannot be parsed
                // is refused conservatively.
                let ai = crate::attribute_info::AttributeInfoMessage::parse(
                    &region[body..body_end],
                    OFFSET_SIZE,
                )?;
                if ai.fractal_heap_address.is_some() {
                    dense = true;
                }
            }
            p = body_end;
        }

        // If dense, read the attribute set out of the source fractal heap now (so
        // the source buffer need not outlive the read) and validate it can be
        // re-emitted into a fresh heap on write. The read is [`read_object_attrs`],
        // shared with the attribute editor: both need the source framed past its
        // userblock before the heap walk, and one home for that framing is what
        // keeps the two from coming to disagree about it.
        let dense_attrs = if dense {
            let stored = read_object_attrs(src, access_mode, addr, base)?;
            let set = dense_attr_set(&region, stored)?;
            // The typed error names the offending attribute, which the previous
            // blanket `EditUnsupported` message could not.
            crate::file_writer::dense_attrs_check(&set.attrs).map_err(Error::Format)?;
            set
        } else {
            DenseAttrSet::default()
        };

        let mut layout: Option<(usize, usize)> = None; // (body offset in kept, size)
        let mut has_link_info = false;
        // (name, creation index, target address) per hard link. The creation
        // index is `None` unless the source group tracks link creation order,
        // and is carried so a copy of one that does reproduces its order.
        let mut children: Vec<(String, Option<u64>, StoredAddress)> = Vec::new();
        // The rebuilt chunk-0 region: every message kept verbatim except hard
        // Link messages (carried as `children`) and, when dense, the Attribute
        // Info message and inline Attribute messages (carried as `dense_attrs`).
        let mut kept = OhRegion::empty(region.props());

        let mut p = 0;
        while let Some((msg_type, body, body_end)) = region.next_message(p)? {
            let mut keep = true;
            match msg_type {
                MessageType::ATTRIBUTE_INFO => {
                    // Already parsed in the first pass; drop the dense Attribute
                    // Info message so the rebuilt header references the fresh heap
                    // (spliced in on write) rather than the source one. A compact
                    // (undefined-heap) Attribute Info message is kept verbatim.
                    if dense {
                        keep = false;
                    }
                }
                MessageType::ATTRIBUTE => {
                    // A dense object should carry no inline Attribute messages,
                    // but drop any defensively so the rebuilt header's only
                    // attribute storage is the fresh heap.
                    if dense {
                        keep = false;
                    }
                }
                MessageType::LINK_INFO => {
                    has_link_info = true;
                    let mut q = body + 2;
                    if body_end - body >= 2 && region[body + 1] & 0x01 != 0 {
                        q += 8;
                    }
                    if q + 8 <= body_end {
                        let heap_addr = u64::from_le_bytes(region[q..q + 8].try_into().unwrap());
                        if heap_addr != u64::MAX {
                            return Err(Error::EditUnsupported(
                                "a group uses dense (fractal-heap) link storage (not supported in place yet)",
                            ));
                        }
                    }
                }
                MessageType::LINK => {
                    keep = false;
                    match LinkMessage::parse(&region[body..body_end], OFFSET_SIZE) {
                        Ok(LinkMessage {
                            name,
                            link_target:
                                LinkTarget::Hard {
                                    object_header_address,
                                },
                            creation_order,
                            ..
                        }) => children.push((name, creation_order, object_header_address)),
                        _ => {
                            return Err(Error::EditUnsupported(
                                "a group contains a soft/external link (not copyable in place yet)",
                            ));
                        }
                    }
                }
                MessageType::DATA_LAYOUT => {
                    // Record the layout body offset within the *kept* region so a
                    // contiguous dataset's data-address field can be repointed
                    // even after earlier messages were dropped.
                    layout = Some((kept.len() + (body - p), body_end - body));
                }
                _ => {}
            }
            if keep {
                kept.push_bytes(&region[p..body_end]);
            }
            p = body_end;
        }

        if let Some((lbody, lsize)) = layout {
            let version = kept[lbody];
            if !(version == 3 || version == 4) || lsize < 2 {
                return Err(Error::EditUnsupported(
                    "an unsupported data-layout version cannot be copied in place yet",
                ));
            }
            let class = kept[lbody + 1];
            match class {
                0 => Ok(ObjModel::DatasetVerbatim {
                    region: kept,
                    dense_attrs,
                }),
                1 => {
                    if lbody + 18 > kept.len() {
                        return Err(Error::EditUnsupported("malformed contiguous data layout"));
                    }
                    let data_addr = StoredAddress::new(u64::from_le_bytes(
                        kept[lbody + 2..lbody + 10].try_into().unwrap(),
                    ));
                    let data_size =
                        u64::from_le_bytes(kept[lbody + 10..lbody + 18].try_into().unwrap());
                    Ok(ObjModel::DatasetContiguous {
                        region: kept,
                        addr_off: lbody + 2,
                        data_addr,
                        data_size,
                        dense_attrs,
                    })
                }
                // Chunked: the verbatim header carries the data-layout and filter-
                // pipeline messages; `read_copy_subtree` (which holds the source
                // buffer) enumerates and captures the chunk bytes and rebuilds the
                // index on write.
                2 => Ok(ObjModel::DatasetChunked {
                    region: kept,
                    dense_attrs,
                }),
                _ => Err(Error::EditUnsupported(
                    "an unsupported data-layout class cannot be copied in place yet",
                )),
            }
        } else if has_link_info {
            // A copied group must carry a Group Info message so the copy stays
            // writable by the C library, even when the source omitted it.
            ensure_group_info(&mut kept)?;
            Ok(ObjModel::Group {
                non_link_region: kept,
                children,
                dense_attrs,
            })
        } else {
            Err(Error::EditUnsupported(
                "an object is neither a contiguous/compact dataset nor a group",
            ))
        }
    }

    /// Read the object at `addr` in the source buffer `d` — and, for a group, its
    /// whole subtree — into an owned [`CopyTree`], the read half of an object copy.
    /// No bytes are written; this both validates that the subtree is copyable and
    /// captures the bytes the write half ([`write_copy_subtree`](Self::write_copy_subtree))
    /// later appends, so the source buffer need not outlive the read.
    ///
    /// `src` is the image the source object lives in: this session's own file image
    /// for an in-file [`copy`](Self::copy), or another file's image for a cross-file
    /// [`copy_from`](Self::copy_from). `base` is that image's userblock base (the
    /// session's own base for an in-file copy, always 0 for a cross-file copy, whose
    /// source is gated to base 0): the stored, base-relative addresses read out of
    /// the source headers are shifted by it to index `src`. When `cross_file` is set,
    /// every copied object header is additionally screened by
    /// [`reject_foreign_addresses`] — verbatim bytes that embed a *source-file*
    /// absolute address (variable-length or reference data, a committed datatype)
    /// would dangle in another file and are refused, whereas an in-file copy keeps
    /// them valid by sharing the source file's heaps and objects.
    fn read_copy_subtree<S: Source + ?Sized>(
        src: &S,
        access_mode: AccessMode,
        addr: u64,
        depth: u32,
        cross_file: bool,
        base: BaseAddress,
    ) -> Result<CopyTree, Error> {
        if depth >= MAX_COPY_DEPTH {
            return Err(Error::EditUnsupported(
                "copy source nests too deeply (possible hard-link cycle)",
            ));
        }
        // `base` is the userblock base of the image `src`: this session's own base
        // for an in-file copy, and always 0 for a cross-file copy (the source is
        // gated to base 0 in `copy_from`). `addr` is an absolute offset into `src`;
        // the stored (base-relative) addresses `read_object` returns for contiguous
        // data, chunk storage, and child links are converted to absolute offsets by
        // adding `base` before `src` is read or a child is descended into.
        match Self::read_object(src, access_mode, addr, base)? {
            ObjModel::DatasetVerbatim {
                region,
                dense_attrs,
            } => {
                if cross_file {
                    reject_foreign_addresses(&region)?;
                    reject_foreign_dense_attrs(&dense_attrs.attrs)?;
                }
                Ok(CopyTree::DatasetVerbatim {
                    region,
                    dense_attrs,
                })
            }
            ObjModel::DatasetContiguous {
                region,
                addr_off,
                data_addr,
                data_size,
                dense_attrs,
            } => {
                if cross_file {
                    reject_foreign_addresses(&region)?;
                    reject_foreign_dense_attrs(&dense_attrs.attrs)?;
                }
                // Storage the source never allocated is copied as storage, not as
                // the values reading it answers with. The reference library does
                // not allocate a contiguous dataset's data until something is
                // written to it, leaving the layout message's address undefined,
                // and reading one answers the fill value for every element (#292)
                // — so a copy that materialized what it read would turn a
                // schema-only dataset into a fully written one of the size its
                // shape declares. `repack` carries the same shape through as
                // unallocated (#293); this is the copy path's half of it (#336).
                //
                // Only reached once external storage is out of the way: it uses
                // this same undefined address for a dataset that *does* hold
                // data, and `read_object` refuses it by name above.
                let data = if data_addr.is_undefined(OFFSET_SIZE) {
                    None
                } else {
                    // The data block is read at its absolute offset into `src`.
                    let start = base.absolute(data_addr)?;
                    start
                        .checked_add(data_size)
                        .filter(|&e| e <= src.len())
                        .ok_or(Error::EditUnsupported("dataset data is out of bounds"))?;
                    let len = data_size.to_usize()?;
                    Some(src.read_exact_at(start, len)?)
                };
                Ok(CopyTree::DatasetContiguous {
                    region,
                    addr_off,
                    data,
                    dense_attrs,
                })
            }
            ObjModel::DatasetChunked {
                region,
                dense_attrs,
            } => {
                // Screen the verbatim header on the cross-file path. This refuses a
                // variable-length or reference datatype (whose chunk payload embeds
                // source-file global-heap / object addresses that would dangle in
                // another file) and any shared message — exactly the forms repack
                // also refuses for a cross-file verbatim chunk copy. An in-file copy
                // keeps them valid by sharing the source file's heaps.
                if cross_file {
                    reject_foreign_addresses(&region)?;
                    reject_foreign_dense_attrs(&dense_attrs.attrs)?;
                }
                let ChunkedHeaderParts {
                    dt,
                    ds,
                    layout,
                    pipeline_message,
                } = parse_chunked_header(&region)?;
                let DataLayout::Chunked { index, .. } = layout else {
                    return Err(Error::EditUnsupported("dataset is not chunked"));
                };
                if !index.enumerable() {
                    return Err(Error::EditUnsupported(
                        "a chunked dataset with a version-2 B-tree chunk index cannot be \
                         copied in place yet",
                    ));
                }
                let ChunkedGeometry {
                    spatial: chunk_dims,
                    element_size,
                    raw_size: _,
                    maxshape,
                } = chunked_geometry(&dt, &ds, &layout)?;

                // The layout's chunk-index address and every chunk address it leads
                // to are stored base-relative, so enumerate and read on a
                // base-relative view of the source image (the identity on a base-0
                // file). The returned addresses are then offsets into `dview`.
                let dview = BaseOffsetSource { inner: src, base };

                // Enumerate the source chunks and map them onto a dense grid; a
                // sparse (holed/unallocated) dataset cannot be reproduced by the
                // verbatim layout path, which needs every grid slot filled.
                let infos =
                    enumerate_chunks_from_source(&dview, &layout, &ds, OFFSET_SIZE, LENGTH_SIZE)?;
                let grid = plan_dense_grid(infos, &ds.dimensions, &chunk_dims).ok_or(
                    Error::EditUnsupported(
                        "a chunked dataset with unallocated (sparse) chunks cannot be copied in place yet",
                    ),
                )?;
                if grid.grid_order.is_empty() {
                    return Err(Error::EditUnsupported(
                        "an empty chunked dataset cannot be copied in place yet",
                    ));
                }

                // Capture each chunk's already-compressed bytes (no decode) into an
                // owned buffer, in dense row-major grid order, so the copy can be
                // written after the source buffer is gone (cross-file copy reads at
                // staging time). Sizes and masks are carried verbatim.
                let mut meta = Vec::with_capacity(grid.grid_order.len());
                let mut chunk_bytes = Vec::with_capacity(grid.grid_order.len());
                for ci in &grid.grid_order {
                    let len = ci.chunk_size.to_usize()?;
                    ci.address
                        .get()
                        .checked_add(len as u64)
                        .filter(|&e| e <= dview.len())
                        .ok_or(Error::EditUnsupported("chunk data is out of bounds"))?;
                    chunk_bytes.push(dview.read_exact_at(ci.address.get(), len)?);
                    meta.push(ChunkMeta {
                        compressed_size: ci.chunk_size.get(),
                        filter_mask: ci.filter_mask,
                    });
                }

                Ok(CopyTree::DatasetChunked {
                    region,
                    shape: ds.dimensions.clone(),
                    chunk_dims,
                    element_size,
                    maxshape,
                    pipeline_message,
                    meta,
                    chunk_bytes,
                    dense_attrs,
                })
            }
            ObjModel::Group {
                non_link_region,
                children,
                dense_attrs,
            } => {
                if cross_file {
                    reject_foreign_addresses(&non_link_region)?;
                    reject_foreign_dense_attrs(&dense_attrs.attrs)?;
                }
                let mut kids = Vec::with_capacity(children.len());
                for (name, creation_order, child) in children {
                    // `read_copy_subtree` reads `src` at absolute offsets.
                    let child = base.absolute(child)?;
                    kids.push((
                        name,
                        creation_order,
                        Self::read_copy_subtree(
                            src,
                            access_mode,
                            child,
                            depth + 1,
                            cross_file,
                            base,
                        )?,
                    ));
                }
                Ok(CopyTree::Group {
                    non_link_region,
                    children: kids,
                    dense_attrs,
                })
            }
        }
    }

    /// Append the fresh copies described by `node` (data blobs and headers) into
    /// this session at end-of-file or into reusable freed regions, returning the
    /// new object-header address of the copied root. The write half of an object
    /// copy; children are written before their parent group so each parent links
    /// its children's new addresses, and a contiguous dataset's data-address field
    /// is repointed at the freshly-written copy. Every address the copy writes into
    /// a header (a contiguous data block, a child link) is stored relative to the
    /// userblock base (`- base`, a no-op on a base-0 file); the chunked storage and
    /// dense attribute heaps are laid out base-relative by their own builders.
    fn write_copy_subtree(&mut self, node: &CopyTree) -> Result<u64, Error> {
        let base = self.superblock.base_address;
        match node {
            CopyTree::DatasetVerbatim {
                region,
                dense_attrs,
            } => {
                let mut region = region.clone();
                self.append_dense_attrs(&mut region, dense_attrs.clone())?;
                let oh = build_v2_object_header(&region)?;
                self.alloc_or_append_typed(&oh, PageType::Meta)
            }
            CopyTree::DatasetContiguous {
                region,
                addr_off,
                data,
                dense_attrs,
            } => {
                let mut region = region.clone();
                // Storage the source never allocated is copied as storage: there
                // is no block to place, and the region already carries the
                // undefined address the source stored, so the copy declares the
                // same empty storage (issue #336).
                if let Some(data) = data {
                    let new_data_addr = self.alloc_or_append_typed(data, PageType::Raw)?;
                    // The placement is an absolute offset; the data-layout
                    // address field stores it relative to the userblock base.
                    let relative = base.relative(new_data_addr)?.get();
                    region.bytes_mut()[*addr_off..*addr_off + 8]
                        .copy_from_slice(&relative.to_le_bytes());
                }
                // The dense heap is placed independently of the data — it is
                // built for whatever address it gets (see `append_dense_attrs`),
                // so no ordering between the two is owed.
                self.append_dense_attrs(&mut region, dense_attrs.clone())?;
                let oh = build_v2_object_header(&region)?;
                self.alloc_or_append_typed(&oh, PageType::Meta)
            }
            CopyTree::DatasetChunked {
                region,
                shape,
                chunk_dims,
                element_size,
                maxshape,
                pipeline_message,
                meta,
                chunk_bytes,
                dense_attrs,
            } => self.write_chunked_relocatable(
                region,
                shape,
                chunk_dims,
                *element_size,
                maxshape.as_deref(),
                pipeline_message.as_deref(),
                meta,
                chunk_bytes,
                dense_attrs.clone(),
            ),
            CopyTree::Group {
                non_link_region,
                children,
                dense_attrs,
            } => {
                let mut region = non_link_region.clone();
                for (name, creation_order, child) in children {
                    let new_child = self.write_copy_subtree(child)?;
                    // The link target is stored relative to the userblock base.
                    // A copied link keeps the creation index the source recorded
                    // for it, so a copy of a group that tracks link creation
                    // order carries the same order as its source — the Link Info
                    // message naming that order is copied verbatim beside it.
                    region.push_link(name, base.relative(new_child)?, *creation_order);
                }
                // The dense heap is built for whatever address it is placed at
                // (see `append_dense_attrs`), so it needs no ordering against the
                // children's headers and data.
                self.append_dense_attrs(&mut region, dense_attrs.clone())?;
                let oh = build_v2_object_header(&region)?;
                self.alloc_or_append_typed(&oh, PageType::Meta)
            }
        }
    }

    /// Write a chunked dataset's storage and return its new object-header address
    /// — the shared write half of a chunked copy ([`CopyTree::DatasetChunked`])
    /// and a relocating chunked overwrite ([`MovingWrite::Chunked`]).
    ///
    /// A fresh chunk-data blob and index are laid out relocatably via
    /// [`plan_chunked_data_verbatim`] / [`emit_chunked_data_verbatim`], pulling
    /// each chunk's already-compressed bytes from `chunk_bytes` (in dense
    /// row-major grid order) and carrying `meta`'s sizes and filter masks and the
    /// source `pipeline_message` verbatim — no recompression, no filter-parameter
    /// reconstruction. Like [`build_chunked_dataset`](Self::build_chunked_dataset)
    /// the blob is sized from its plan before it is placed, so it can go into a
    /// freed region that fits it as readily as at end-of-file. The verbatim header
    /// `region`'s data-layout message is then swapped for the one the planner
    /// produced (every other message preserved), any dense attribute heap is
    /// placed, and the header is written into reusable freed space or at
    /// end-of-file.
    #[expect(
        clippy::too_many_arguments,
        reason = "the chunked rebuild needs the full geometry, \
        pipeline, and chunk payloads; bundling them into a struct would only move the list"
    )]
    fn write_chunked_relocatable(
        &mut self,
        region: &OhRegion,
        shape: &[u64],
        chunk_dims: &[u64],
        element_size: NonZeroUsize,
        maxshape: Option<&[MaxExtent]>,
        pipeline_message: Option<&[u8]>,
        meta: &[ChunkMeta],
        chunk_bytes: &[Vec<u8>],
        dense_attrs: DenseAttrSet,
    ) -> Result<u64, Error> {
        // Plan once at a provisional base purely to size the data region: the plan
        // walks chunk *sizes* and sizes the index from its layout, so it touches
        // no bytes at all, and every address it embeds sits in a fixed-width
        // field, so its total length is the same wherever the blob lands. That is
        // what lets the address be chosen — a freed region or end-of-file — before
        // the bytes exist. This plan is discarded; the one built at the real base
        // below is what the emit works from.
        let sizing = plan_chunked_data_verbatim(
            meta,
            shape,
            chunk_dims,
            element_size,
            pipeline_message,
            StoredAddress::new(0),
            maxshape,
        )?;
        let (_addr, layout_message) =
            self.place_relocatable(sizing.plan.total_len, PageType::Raw, |stored_base| {
                // Re-plan at the address the blob really occupies, so its embedded
                // addresses resolve to their real file offsets once the reader adds
                // the userblock base back (see `build_chunked_dataset`).
                let layout = plan_chunked_data_verbatim(
                    meta,
                    shape,
                    chunk_dims,
                    element_size,
                    pipeline_message,
                    stored_base,
                    maxshape,
                )?;
                let mut buf =
                    Vec::with_capacity(usize::try_from(layout.plan.total_len).unwrap_or(0));
                emit_chunked_data_verbatim(
                    &mut buf,
                    &layout.plan,
                    &SliceChunkProvider {
                        chunks: chunk_bytes,
                    },
                )?;
                Ok((buf, layout.layout_message))
            })?;
        // Swap the data-layout message for the rebuilt one; keep every other header
        // message (datatype, dataspace, fill value, filter pipeline, attributes)
        // verbatim.
        let mut new_region = replace_layout_message(region, &layout_message)?;
        self.append_dense_attrs(&mut new_region, dense_attrs)?;
        let oh = build_v2_object_header(&new_region)?;
        self.alloc_or_append_typed(&oh, PageType::Meta)
    }

    /// When `attrs` is non-empty, build a fresh dense (fractal-heap) attribute
    /// blob for it, place it, and splice the matching Attribute Info message onto
    /// `region`. A no-op for an empty set.
    ///
    /// The blob produced by [`DenseAttrPlan::build`] is fully
    /// relocatable: every address it embeds is `base + fixed offset`, and its
    /// length is the same for every base, so it can go into a freed metadata
    /// region as readily as at end-of-file — the base it is built for is whichever
    /// address it gets. The reservation comes from the plan the blob is then
    /// built from, so sizing it costs no bytes. The freshly built heap is always
    /// same-file, so it never aliases the source heap even for an in-file copy.
    /// The caller has already validated [`dense_attrs_check`].
    ///
    /// [`DenseAttrPlan::build`]: crate::file_writer::DenseAttrPlan::build
    /// [`dense_attrs_check`]: crate::file_writer::dense_attrs_check
    fn append_dense_attrs(
        &mut self,
        region: &mut OhRegion,
        set: DenseAttrSet,
    ) -> Result<(), Error> {
        if set.is_empty() {
            return Ok(());
        }
        let attr_info_message = self.place_dense_attrs(&set.attrs, set.creation)?;
        region.push(MessageType::ATTRIBUTE_INFO, &attr_info_message);
        Ok(())
    }

    /// Place dense attribute storage for a staged dataset whose set needs it,
    /// returning the Attribute Info message naming the heap — or `None` when the
    /// set belongs in the object header, where the header builders write it
    /// inline. Which it is was decided and validated when the dataset was staged
    /// ([`FlatDataset::attrs_are_dense`]).
    fn place_dense_attrs_if_needed(&mut self, fd: &FlatDataset) -> Result<Option<Vec<u8>>, Error> {
        if !fd.attrs_are_dense {
            return Ok(None);
        }
        self.place_dense_attrs(&fd.attrs, DenseAttrCreationOrder::Untracked)
            .map(Some)
    }

    /// Build a fresh dense attribute blob for `attrs`, place it, and return the
    /// Attribute Info message naming it — the header's whole share of dense
    /// storage. See [`append_dense_attrs`](Self::append_dense_attrs) for why the
    /// blob may be placed anywhere.
    fn place_dense_attrs(
        &mut self,
        attrs: &[crate::attribute::AttributeMessage],
        creation: DenseAttrCreationOrder,
    ) -> Result<Vec<u8>, Error> {
        let plan = crate::file_writer::dense_attrs_plan(attrs, creation);
        let (_addr, attr_info_message) =
            self.place_relocatable(plan.blob_len(), PageType::Meta, |stored_base| {
                let blob = plan.build(stored_base);
                Ok((blob.blob, blob.attr_info_message))
            })?;
        Ok(attr_info_message)
    }

    /// Resolve what an attribute edit left for this phase ([`plan_attr_ops`]) and
    /// append what names it to the object's message `region`.
    ///
    /// Both arms place a variable-length attribute's global heap collection and
    /// patch the placeholder references in its message; where they differ is what
    /// carries the result. A compact attribute becomes an inline Attribute
    /// message; a dense set is built into a fresh heap the header then names.
    ///
    /// For the dense arm the patching has to come first: the heap stores each
    /// attribute's *message bytes*, references and all, so a heap built before
    /// them would hold the placeholders — the same ordering the whole-file writer
    /// keeps for the same reason.
    ///
    /// A no-op for an edit that resolved entirely in the preflight, which is what
    /// makes it safe to call on every rebuilt header.
    fn place_edited_attrs(
        &mut self,
        region: &mut OhRegion,
        attrs: EditedAttrs,
    ) -> Result<(), Error> {
        match attrs {
            EditedAttrs::Compact(pending) => {
                for pending in pending {
                    let PendingVlAttr {
                        mut msg,
                        collections,
                        creation_index,
                    } = pending;
                    let addrs = self.place_vl_collections(&collections)?;
                    patch_vl_refs(&mut msg.raw_data, &addrs);
                    *region = put_attr_message(
                        region,
                        &msg.name,
                        &msg.serialize(LENGTH_SIZE),
                        creation_index,
                    )?;
                }
                Ok(())
            }
            EditedAttrs::Dense(mut dense) => {
                for (idx, collections) in std::mem::take(&mut dense.vl) {
                    let addrs = self.place_vl_collections(&collections)?;
                    patch_vl_refs(&mut dense.set.attrs[idx].raw_data, &addrs);
                }
                self.append_dense_attrs(region, dense.set)
            }
        }
    }

    /// Apply a relocating value overwrite (`write_dataset` resize / compact
    /// rewrite): write the new data and a rewritten object header at end-of-file
    /// (or into reusable freed space) and return the new header address. The
    /// caller patches the parent group's link to this address. The old data
    /// extent (for a resized contiguous dataset) is freed separately, after the
    /// commit's superblock repoint, so it is never reused mid-commit.
    fn write_moving(&mut self, mw: &MovingWrite) -> Result<u64, Error> {
        let base = self.superblock.base_address;
        match mw {
            MovingWrite::Contiguous {
                region,
                addr_off,
                bytes,
                ..
            } => {
                let raw = self.resolve_overwrite_bytes(bytes)?;
                let new_data_addr = self.alloc_or_append_typed(&raw, PageType::Raw)?;
                let mut region = region.clone();
                // The placement is an absolute file offset; the contiguous
                // data-layout field stores it relative to the userblock base (`-
                // base`, a no-op on a base-0 file).
                let relative = base.relative(new_data_addr)?.get();
                region.bytes_mut()[*addr_off..*addr_off + 8]
                    .copy_from_slice(&relative.to_le_bytes());
                // The data size field follows the 8-byte address in the contiguous
                // layout body; keep it in sync with the new length.
                let size_off = *addr_off + 8;
                let raw_len = raw.len() as u64;
                region.bytes_mut()[size_off..size_off + 8].copy_from_slice(&raw_len.to_le_bytes());
                let oh = build_v2_object_header(&region)?;
                self.alloc_or_append_typed(&oh, PageType::Meta)
            }
            MovingWrite::Compact { region, bytes } => {
                let raw = self.resolve_overwrite_bytes(bytes)?;
                let region = rebuild_compact_layout_region(region, &raw)?;
                let oh = build_v2_object_header(&region)?;
                self.alloc_or_append_typed(&oh, PageType::Meta)
            }
            MovingWrite::Chunked {
                region,
                shape,
                chunk_dims,
                element_size,
                maxshape,
                pipeline_message,
                payload,
                ..
            } => {
                let deferred;
                let chunk_bytes = match payload {
                    ChunkPayload::Encoded(chunk_bytes) => chunk_bytes,
                    ChunkPayload::Deferred { bytes, padding, dt } => {
                        // Resolve before splitting: the heap collections have to
                        // be placed, and their addresses patched into the element
                        // references, before those references are cut into chunks
                        // and — on a filtered dataset — compressed over.
                        let raw = self.resolve_overwrite_bytes(bytes)?;
                        deferred = split_and_encode_chunks(
                            &raw,
                            shape,
                            chunk_dims,
                            *element_size,
                            padding,
                            pipeline_message.as_deref(),
                            dt,
                        )?;
                        &deferred
                    }
                };
                let meta: Vec<ChunkMeta> = chunk_bytes
                    .iter()
                    .map(|c| ChunkMeta {
                        compressed_size: c.len() as u64,
                        filter_mask: 0,
                    })
                    .collect();
                self.write_chunked_relocatable(
                    region,
                    shape,
                    chunk_dims,
                    *element_size,
                    maxshape.as_deref(),
                    pipeline_message.as_deref(),
                    &meta,
                    chunk_bytes,
                    DenseAttrSet::default(),
                )
            }
            MovingWrite::AppendedChunks {
                region,
                new_dataspace_body,
                chunk_dims_u32,
                element_size,
                has_filters,
                kept_chunks,
                new_chunk_bytes,
                ..
            } => self.write_appended_chunks(
                region,
                new_dataspace_body,
                chunk_dims_u32,
                *element_size,
                *has_filters,
                kept_chunks,
                new_chunk_bytes,
            ),
            MovingWrite::AttrEdit { region, attrs } => {
                // `region` already carries what the commit preflight could resolve;
                // `place_edited_attrs` places the rest, exactly as the
                // group-attribute apply loop does. Then build and place the
                // relocated dataset header: the data-layout message is untouched,
                // so the dataset's chunk data and index stay in place and only the
                // header moves.
                let mut region = region.clone();
                self.place_edited_attrs(&mut region, attrs.clone())?;
                let oh = build_v2_object_header(&region)?;
                self.alloc_or_append_typed(&oh, PageType::Meta)
            }
        }
    }

    /// Apply a relocating append ([`MovingWrite::AppendedChunks`]): place the new
    /// (and any rewritten trailing) chunk bytes, rebuild a fresh
    /// Extensible Array over the kept plus appended chunks, grow the dataspace and
    /// repoint the data layout in the verbatim header `region`, and write the
    /// relocated header. Returns the new header address; the caller patches the
    /// parent link. The kept chunk data is untouched (referenced by both the old
    /// and new index during the commit); the old index/header/trailing chunk are
    /// freed only after the superblock repoint.
    ///
    /// The new chunks and the new index go down together where one freed region
    /// holds them and a paged file is what makes that worth asking, and one at a
    /// time otherwise; the body says why.
    #[expect(
        clippy::too_many_arguments,
        reason = "the append rebuild needs the header region, grown dataspace, chunk \
        geometry, and both chunk sets; bundling them into a struct would only move the list"
    )]
    fn write_appended_chunks(
        &mut self,
        region: &OhRegion,
        new_dataspace_body: &[u8],
        chunk_dims_u32: &[u32],
        element_size: NonZeroUsize,
        has_filters: bool,
        kept_chunks: &[WrittenChunk],
        new_chunk_bytes: &[Vec<u8>],
    ) -> Result<u64, Error> {
        let base = self.superblock.base_address;
        // Where the appended chunks and the rebuilt index go, and whether they go
        // together.
        //
        // Contiguity is worth something on a *paged* file. The index goes in a raw
        // page, not a metadata one, because that is where this crate puts every
        // chunk index — and the reclaim side can only tell a raw index from the
        // metadata one the reference library writes by the chunk data it abuts
        // ([`index_is_provably_raw`](Self::index_is_provably_raw)). An index that
        // abuts none of its chunks cannot be placed by the commit that supersedes
        // it, so its bytes are held as dead until the page around them empties.
        // Laying the new chunks and the new index down as one blob — the way the
        // from-scratch writer and `write_chunked_relocatable` lay out chunked
        // storage — keeps that proof available (issue #388).
        //
        // It is worth nothing on a file that is not paged, which has no page types
        // to keep apart, and it must never be bought with reuse: one blob needs one
        // freed region big enough for the lot, where the chunks placed one at a
        // time fill several smaller ones. So the blob form is taken only when a
        // single region already holds it, and otherwise each chunk is placed on its
        // own and the index after them. That fallback keeps the proof only where
        // the last chunk placed ends up immediately below the index — which is what
        // happens when both append at end-of-file, and not what happens when the
        // chunks find holes to sit in. An index left abutting nothing is recorded
        // as dead rather than free, which costs space and never correctness.
        //
        // Sizing runs at a provisional base, exactly as `write_chunked_relocatable`
        // sizes its blob: an Extensible Array's length follows the slot layout and
        // every address it embeds sits in a fixed-width field, so the total is the
        // same wherever the array lands, and `place` rejects any disagreement.
        //
        // The element width comes from the chunk geometry rather than from the
        // chunk list, so a rebuild declares the same width the original index did —
        // including when the dataset it grows was created empty.
        let chunk_bytes =
            full_chunk_bytes(chunk_dims_u32.iter().map(|&d| u64::from(d)), element_size);
        // Rank 1 and unlimited along axis 0 (`prepare_append` refuses anything
        // else), so every chunk's index slot is its position in the grid and the
        // array is dense from zero.
        //
        // This engine applies every filter to a new chunk (no per-chunk skipping),
        // so an appended chunk's mask is always 0. Kept chunks carry their own
        // (possibly nonzero) mask.
        let chunk_total: u64 = new_chunk_bytes.iter().map(|cb| cb.len() as u64).sum();
        let placed_chunks = |blob_stored: StoredAddress| -> Vec<WrittenChunk> {
            let mut combined: Vec<WrittenChunk> = kept_chunks.to_vec();
            let mut next = blob_stored;
            for cb in new_chunk_bytes {
                combined.push(WrittenChunk {
                    address: next,
                    compressed_size: cb.len() as u64,
                    filter_mask: 0,
                });
                next = next.offset(cb.len() as u64);
            }
            combined
        };
        let sizing = placed_chunks(StoredAddress::new(0));
        let ea_len = extensible_array_len(
            &crate::chunked_write::IndexSlots::dense(&sizing),
            chunk_bytes,
            OFFSET_SIZE,
            LENGTH_SIZE,
            has_filters,
        );
        let ea = |slots: &crate::chunked_write::IndexSlots<'_>,
                  at: StoredAddress|
         -> Result<Vec<u8>, Error> {
            build_extensible_array_at(
                slots,
                chunk_bytes,
                OFFSET_SIZE,
                LENGTH_SIZE,
                has_filters,
                at,
            )
            .map_err(Error::Format)
        };
        // Reserved by hand rather than through `reserve`, which would fall back to
        // end-of-file: this asks only whether a freed region holds the whole blob,
        // and takes the per-chunk path when none does.
        let blob = match self.paged {
            Some(_) => self.alloc_free(chunk_total + ea_len, PageType::Raw),
            None => None,
        };
        let ea_stored = match blob {
            Some(addr) => {
                // The region is out of the free lists from here, so every way the
                // build can fail has to put it back: a `?` straight out of this arm
                // would leave it neither free nor written for the rest of the
                // session. Collected into a `Result` and handed back on the way out.
                let placed = (|| -> Result<StoredAddress, Error> {
                    let blob_stored = base.relative(addr)?;
                    let combined = placed_chunks(blob_stored);
                    let mut buf =
                        Vec::with_capacity(usize::try_from(chunk_total + ea_len).unwrap_or(0));
                    for cb in new_chunk_bytes {
                        buf.extend_from_slice(cb);
                    }
                    buf.extend_from_slice(&ea(
                        &crate::chunked_write::IndexSlots::dense(&combined),
                        blob_stored.offset(chunk_total),
                    )?);
                    self.place(
                        Placement::Reused {
                            addr,
                            len: chunk_total + ea_len,
                        },
                        &buf,
                    )?;
                    Ok(blob_stored.offset(chunk_total))
                })();
                match placed {
                    Ok(ea_stored) => ea_stored,
                    Err(e) => {
                        self.release_raw_alloc(addr, chunk_total + ea_len);
                        return Err(e);
                    }
                }
            }
            None => {
                // A chunk embeds no addresses of its own, so it can go anywhere and
                // the index below simply records where it went.
                let mut combined: Vec<WrittenChunk> = kept_chunks.to_vec();
                for cb in new_chunk_bytes {
                    let abs = self.alloc_or_append_typed(cb, PageType::Raw)?;
                    combined.push(WrittenChunk {
                        address: base.relative(abs)?,
                        compressed_size: cb.len() as u64,
                        filter_mask: 0,
                    });
                }
                let (ea_addr, ()) = self.place_relocatable(ea_len, PageType::Raw, |at| {
                    Ok((
                        ea(&crate::chunked_write::IndexSlots::dense(&combined), at)?,
                        (),
                    ))
                })?;
                base.relative(ea_addr)?
            }
        };

        // Swap the dataspace (grown) and data-layout (repointed at the new index)
        // messages; every other header message is preserved verbatim.
        #[expect(
            clippy::cast_possible_truncation,
            reason = "element size is a datatype byte width that fits u32"
        )]
        let layout_body = serialize_v4_extensible_array(
            chunk_dims_u32,
            ea_stored,
            OFFSET_SIZE,
            element_size.get() as u32,
        );
        let region = replace_dataspace_message(region, new_dataspace_body)?;
        let region = replace_layout_message(&region, &layout_body)?;
        let oh = build_v2_object_header(&region)?;
        self.alloc_or_append_typed(&oh, PageType::Meta)
    }

    /// Append `bytes` at end-of-file, returning the absolute address they were
    /// written at.
    fn append(&mut self, bytes: &[u8]) -> Result<u64, Error> {
        self.image.append(bytes)
    }

    /// Overwrites bytes in place at `offset`. The caller guarantees the range
    /// already exists.
    fn write_at(&mut self, offset: u64, bytes: &[u8]) -> Result<(), Error> {
        self.image.write_at(offset, bytes)
    }

    /// Ensure the next allocation begins in a page holding page type `ty`, on a
    /// paged file. A no-op on the common non-paged file.
    ///
    /// A paged file never mixes metadata and raw data within one page, so when the
    /// tail page holds the *other* type and is only partially filled it is first
    /// padded to a page boundary, the padding being recorded as free space of the
    /// outgoing type.
    ///
    /// Call this **before** reading the image's end-of-file ([`Source::len`]) to compute an
    /// address that will be embedded in the bytes being built: several callers
    /// (the chunk blob, the extensible-array index, the dense-attribute blob)
    /// build content whose interior addresses assume it lands at the current
    /// end-of-file, and padding inserted after that read would shift the landing
    /// address out from under them.
    fn begin_page(&mut self, ty: PageType) -> Result<(), Error> {
        // Destructure so the page state and the image are borrowed as the
        // separate fields they are.
        let Self { image, paged, .. } = self;
        match paged.as_mut() {
            Some(pg) => pg.begin(image.as_mut(), ty),
            None => Ok(()),
        }
    }

    /// Place `bytes` as page type `ty`, reusing a free region where one fits.
    /// Equivalent to [`reserve`](Self::reserve) followed by [`place`](Self::place),
    /// for the callers whose bytes are already built.
    fn alloc_or_append_typed(&mut self, bytes: &[u8], ty: PageType) -> Result<u64, Error> {
        let at = self.reserve(bytes.len() as u64, ty)?;
        self.place(at, bytes)
    }

    /// Choose where `len` bytes of page type `ty` will go: a reusable free region
    /// left by a prior commit, or the current end-of-file. The returned
    /// [`Placement`] must be handed to [`place`](Self::place) with exactly `len`
    /// bytes.
    ///
    /// Splitting the choice from the write is what lets a *relocatable* blob — one
    /// whose interior addresses are all `base + fixed offset`, like a chunked
    /// dataset's data region or a dense attribute heap — be built for a freed
    /// region rather than only for end-of-file: its size is known from its plan
    /// before its bytes exist, so the address can be settled first
    /// ([`place_relocatable`](Self::place_relocatable) is that sequence).
    ///
    /// Reuse only ever draws from free space vacated by *earlier* commits in this
    /// session (or seeded from the on-disk managers at open) — never space the
    /// current commit is about to free, which stays in its `to_free` list until
    /// after the superblock repoint. The bytes it overwrites are therefore already
    /// unreachable from the on-disk root, so a mid-commit crash cannot corrupt the
    /// live tree: the superblock still points at the prior, intact root.
    ///
    /// A paged file reuses within the matching page type, or out of a page holding
    /// nothing at all, which belongs to no type ([`PagedEdit::alloc_typed`]); every
    /// page stays homogeneous either way. It never opens a page for a reused region
    /// — the tail page is untouched by a write into the middle of the file.
    fn reserve(&mut self, len: u64, ty: PageType) -> Result<Placement, Error> {
        if let Some(addr) = self.alloc_free(len, ty) {
            return Ok(Placement::Reused { addr, len });
        }
        self.begin_page(ty)?;
        Ok(Placement::Appended {
            addr: self.image.len(),
            len,
        })
    }

    /// Draw `len` bytes from the free space of page type `ty`, or `None` when no
    /// single region is large enough.
    ///
    /// A non-paged file keeps one list for the whole file. A paged file keeps one
    /// per page type and must be served from the list matching `ty`: handing a
    /// metadata hole to raw data (or the reverse) would mix the two within a page,
    /// which is the single invariant the paged strategy exists to hold. A page
    /// holding nothing at all is the exception, and
    /// [`alloc_typed`](PagedEdit::alloc_typed) is where it is spent.
    fn alloc_free(&mut self, len: u64, ty: PageType) -> Option<u64> {
        let Some(pg) = self.paged.as_mut() else {
            return self.free.alloc(len);
        };
        pg.alloc_typed(len, ty)
    }

    /// Hand `[addr, addr + len)`, drawn from
    /// [`alloc_free`](Self::alloc_free) with [`PageType::Raw`], back to the list
    /// it came from — for a placement refused *after* the draw, which would
    /// otherwise hold the region for the rest of the session without writing
    /// anything into it.
    ///
    /// A paged file takes it back into the raw list whichever list served it: a
    /// whole page [`PagedEdit::alloc_typed`] claimed from the metadata side for
    /// raw data is raw from the moment it is claimed, and that call already
    /// returns its alignment tail there. A page that ends up wholly free is
    /// promoted out of either list by
    /// [`promote_whole_free_pages`](PagedEdit::promote_whole_free_pages) as usual.
    fn release_raw_alloc(&mut self, addr: u64, len: u64) {
        match self.paged.as_mut() {
            Some(pg) => pg.raw.free(addr, len),
            None => self.free.free(addr, len),
        }
    }

    /// Write `bytes` at the address [`reserve`](Self::reserve) handed out,
    /// returning that address.
    ///
    /// The length is checked rather than asserted: a reused region is sized to the
    /// reservation and is followed by live bytes, so writing more than was reserved
    /// would silently destroy a neighboring object. That makes it the one internal
    /// miscount worth paying a comparison to catch in every build.
    fn place(&mut self, at: Placement, bytes: &[u8]) -> Result<u64, Error> {
        if bytes.len() as u64 != at.len() {
            return Err(Error::Format(FormatError::SerializationError(format!(
                "a placement reserved {} bytes but built {}",
                at.len(),
                bytes.len()
            ))));
        }
        match at {
            Placement::Reused { addr, .. } => {
                self.write_at(addr, bytes)?;
                Ok(addr)
            }
            Placement::Appended { addr, .. } => {
                let written = self.append(bytes)?;
                debug_assert_eq!(
                    written, addr,
                    "an appended placement must land at end-of-file"
                );
                Ok(written)
            }
        }
    }

    /// Place a *relocatable* blob of `len` bytes as page type `ty`: pick its
    /// address first, then build it for that address with `build`, then write it
    /// there. Returns the address and whatever else `build` produced (typically the
    /// object-header message naming the blob, which embeds the same address).
    ///
    /// `build` receives the address the blob will occupy. Every address the blob
    /// embeds is written in that same form, which a reader resolves against the
    /// file's base address.
    ///
    /// `len` must be the length `build` will produce; [`place`](Self::place)
    /// rejects a mismatch. For every blob placed this way the length is a function
    /// of the content alone — the addresses sit in fixed-width fields — so every
    /// caller derives it, and none builds the blob to measure it:
    /// [`DenseAttrPlan::blob_len`](crate::file_writer::DenseAttrPlan::blob_len)
    /// for a dense attribute heap, [`extensible_array_len`] for an appended
    /// chunk index, and [`chunked_data_len`] / [`plan_chunked_data_verbatim`]
    /// for a whole chunked data region, both of which size their index through
    /// [`chunk_index_len`](crate::chunked_write::chunk_index_len).
    fn place_relocatable<T>(
        &mut self,
        len: u64,
        ty: PageType,
        build: impl FnOnce(StoredAddress) -> Result<(Vec<u8>, T), Error>,
    ) -> Result<(u64, T), Error> {
        let at = self.reserve(len, ty)?;
        let (bytes, extra) = build(self.superblock.base_address.relative(at.address())?)?;
        let addr = self.place(at, &bytes)?;
        Ok((addr, extra))
    }

    /// Place one variable-length dataset's or attribute's already-built,
    /// self-contained global heap collections (from
    /// [`build_global_heap_collections`] or a
    /// [`VlStringStaging::collections`]) and return, in the same order, the
    /// base-relative addresses its variable-length references should be patched
    /// to. A `GCOL` blob embeds no addresses of its own, so it can be appended
    /// (or dropped into reused free space) at any point in the apply loop,
    /// unlike a group or dataset header, which must be built last so it can name
    /// its children's real addresses. Each collection is placed independently,
    /// so they need not land contiguously.
    fn place_vl_collections(&mut self, collections: &[Vec<u8>]) -> Result<Vec<u64>, Error> {
        collections
            .iter()
            .map(|collection| {
                let addr = self.alloc_or_append_typed(collection, PageType::Meta)?;
                Ok(self.superblock.base_address.relative(addr)?.get())
            })
            .collect()
    }

    /// Resolve a staged value overwrite's element bytes: place any global heap
    /// collections its variable-length references name, patch their addresses
    /// in, and return the bytes ready to write (issue #321). An overwrite that
    /// stages nothing — every one but a `with_vlen_strings` one — is handed
    /// back untouched.
    ///
    /// This allocates, so every caller is in the apply phase — the preflight
    /// that chose the plan only reads.
    ///
    /// It also carries the reclaim. The collections a *previous* overwrite of
    /// this same path placed are handed to the commit's free list, and the ones
    /// placed here are recorded in their stead, so rotating a dataset's strings
    /// reaches a steady state instead of leaking a generation per commit. Only
    /// collections this session placed are ever reclaimed, and only while
    /// nothing has been able to name them twice — see
    /// [`vl_overwrite_heaps`](Self::vl_overwrite_heaps) for why that provenance
    /// is the whole proof, and `repack` for the collections it cannot make.
    fn resolve_overwrite_bytes<'b>(
        &mut self,
        bytes: &'b OverwriteBytes,
    ) -> Result<Cow<'b, [u8]>, Error> {
        let Some(vlen) = &bytes.vlen else {
            // Nothing to resolve, and nothing to copy: every overwrite but a
            // variable-length one takes this path, and the in-place loop writes
            // straight through the borrow. Cloning here would charge each of
            // them a second copy of its whole payload, which the bounded
            // backing in particular is built not to pay.
            return Ok(Cow::Borrowed(&bytes.raw));
        };
        let staging = &vlen.staging;
        // Read rather than removed: the record is replaced (not consumed) when
        // the new collections land just below.
        let superseded = self
            .vl_overwrite_heaps
            .get(&vlen.path)
            .cloned()
            .unwrap_or_default();
        let mut raw = bytes.raw.clone();
        let base = self.superblock.base_address;
        // A staging with no collections patches nothing, which is the right
        // answer for it: an element with no heap object keeps the zero address
        // that reads back as null. It falls out of the general path rather than
        // being a case of its own — `patch_vl_refs_masked` has one offset per
        // object, so no offsets is no iterations — and it still records (an
        // empty record) and still supersedes.
        let addrs = self.place_vl_collections(&staging.collections)?;
        patch_vl_refs_masked(&mut raw, &staging.patch_offsets, &addrs);
        // `place_vl_collections` answers base-relative, since that is what an
        // element reference stores; the free list is absolute.
        let placed = addrs
            .iter()
            .zip(&staging.collections)
            .map(|(&a, c)| Ok((base.absolute(StoredAddress::new(a))?, c.len() as u64)))
            .collect::<Result<Vec<_>, FormatError>>()?;
        self.vl_overwrite_heaps.insert(vlen.path.clone(), placed);
        self.superseded_heaps.extend(superseded);
        Ok(Cow::Owned(raw))
    }

    /// Forget every heap collection this session recorded as exclusively one
    /// dataset's, because `staged` contains an edit that could name one of them
    /// a second time (issue #321).
    ///
    /// The provenance in [`vl_overwrite_heaps`](Self::vl_overwrite_heaps) says a
    /// collection was named once *when it was placed*. Two staged edits can
    /// falsify that afterwards, and both do it by re-emitting element bytes
    /// somebody else's references live in:
    ///
    /// - a **copy**, in-file or cross-file, which duplicates a dataset's element
    ///   references verbatim. This is not hypothetical: an in-file copy of a
    ///   variable-length dataset is exactly how two datasets come to name one
    ///   collection.
    /// - a **raw-bytes write** over a datatype that reaches a heap address
    ///   ([`datatype_holds_file_address`]). `with_raw_data` hands the engine
    ///   element bytes it does not interpret, so a caller that read them from
    ///   one dataset and wrote them to another has aliased whatever they named —
    ///   the same unscreened door issue #317 recorded for object references.
    ///
    /// Wholesale rather than per-path: what a copy's source references is not
    /// known without decoding it, and a proof that has to be argued per edit is
    /// the kind that goes stale. Reclaim is an optimization, and giving all of
    /// it up on a rare edit costs a leak, where keeping one entry too long costs
    /// a freed collection somebody still reads.
    fn invalidate_heap_provenance(&mut self, staged: &StagedEdits) {
        if self.vl_overwrite_heaps.is_empty() {
            return;
        }
        let aliasing_edit = !staged.copies.is_empty()
            || !staged.cross_copies.is_empty()
            // A dataset staged with element bytes of its own (`with_raw_data`)
            // rather than staging its strings here, whether it is being created
            // or overwritten.
            || staged
                .writes
                .iter()
                .chain(&staged.datasets)
                .any(|(_, fd)| {
                    fd.vl_string_staging.is_none()
                        && datatype_holds_file_address(&fd.dt)
                });
        if aliasing_edit {
            self.vl_overwrite_heaps.clear();
            return;
        }
        // A deleted dataset's record would otherwise outlive it and be applied
        // to whatever later takes its path.
        for path in &staged.deletes {
            self.vl_overwrite_heaps
                .retain(|recorded, _| !recorded.overlaps(path));
        }
    }

    /// Resolves one object-reference element's target to the address stored on
    /// disk. [`ObjectRefTarget::Raw`] is written back verbatim: a null or an
    /// undefined reference points at nothing, and the whole-file writer treats
    /// it the same way. [`ObjectRefTarget::Path`] resolves, in order:
    ///
    /// 1. Against `path_addr` — every group and dataset this commit has
    ///    already placed (a sibling dataset placed earlier in the same
    ///    group's batch — see the apply loop's non-reference-first ordering —
    ///    or a descendant subtree fully processed earlier in the deepest-first
    ///    walk).
    /// 2. Against the pre-commit on-disk file
    ///    ([`resolve_path_any`](crate::group_v2::resolve_path_any)), but only
    ///    when the path is untouched by this commit, so its pre-commit
    ///    address is guaranteed to still be valid post-commit. "Touched"
    ///    means: a dirty group (`nodes`, new or merely rewritten because an
    ///    addition lives under it — its own address changes either way); a
    ///    path this commit adds, or that lies under a subtree this commit
    ///    copies in (`add_targets`, checked by prefix so a copy's interior is
    ///    covered even though only its root is enumerated there); or a
    ///    `write_dataset` target (`write_targets`) — conservatively refused
    ///    even for a same-length overwrite that does not actually relocate,
    ///    since resolving that distinction here is not worth the complexity.
    /// 3. If the path resolves nowhere at all (neither this commit nor the
    ///    pre-commit file has ever heard of it), as an undefined reference
    ///    (`HADDR_UNDEF`) — mirroring [`ObjectRefTarget::Path`]'s existing
    ///    whole-file-writer resolution convention for the same builder type.
    ///
    /// A path that step 1 misses but step 2 identifies as commit-touched is
    /// refused with a clear [`Error::EditUnsupported`] rather than resolved to
    /// a stale or wrong address — the one case this engine cannot resolve
    /// without the whole-file writer's two-pass dummy/real-address scheme.
    ///
    /// A path this same commit *deletes* (`delete_targets`) is refused for a
    /// different reason, and so carries its own message: the address step 2
    /// would resolve is not stale but doomed, and the next commit to reuse the
    /// span leaves a reference that dereferenced cleanly reading whatever
    /// landed there. The test is by **prefix**, because a deletion takes the
    /// whole subtree with it (`collect_free_spans` walks it): a reference to a
    /// child of a deleted group dangles exactly as a reference to the group
    /// itself does (issue #314).
    ///
    /// It is a conservative test in the same sense `write_targets` is. A child
    /// whose object survives the delete through another hard link keeps its
    /// address — `collect_free_spans` reclaims nothing when the incoming count
    /// is not 1 — and is refused here anyway, so the message states what the
    /// commit does rather than what the allocator will conclude.
    ///
    /// The delete test runs *after* the other three so that a replacement this
    /// commit has merely not placed yet — a sibling group later in the same
    /// depth band — is reported as an ordering problem rather than as a
    /// deletion. That ordering is a better default, not a partition: because
    /// `add_targets` claims a replaced path's whole subtree by prefix, a child
    /// the replacement does *not* recreate is genuinely doomed and still
    /// reports "still writing". Distinguishing it would mean enumerating what
    /// a replacement actually rebuilds, which neither list holds. A path the
    /// commit puts back is resolved by step 1 before either test is reached.
    fn resolve_reference_target(
        target: &ObjectRefTarget,
        path_addr: &BTreeMap<ObjectPathBuf, u64>,
        nodes: &BTreeMap<ObjectPathBuf, Node>,
        add_targets: &[ObjectPathBuf],
        write_targets: &[ObjectPathBuf],
        delete_targets: &[ObjectPathBuf],
        invalidated: &InvalidatedAddresses,
        src: &(impl Source + ?Sized),
        superblock: &Superblock,
    ) -> Result<StoredAddress, Error> {
        let path = match target {
            // An address carries no name to test, so the delete check the path
            // arm makes by prefix is made here on the address itself.
            //
            // It fires on nothing today: every `Raw` this crate stages is a
            // sentinel, because `repack`'s faithful re-emit resolves a real
            // target to a `Path` and leaves only the null and undefined
            // references raw. It is here because the variant carries an
            // arbitrary address, and a producer that staged a real one would
            // otherwise write it past the screen its `Path` twin gets — which
            // is the shape of this bug in the first place (issue #317).
            ObjectRefTarget::Raw(addr) => {
                if let Some(refusal) = invalidated.refusal(*addr) {
                    return Err(Error::EditUnsupported(refusal));
                }
                return Ok(*addr);
            }
            ObjectRefTarget::Path(path) => path,
        };
        let base = superblock.base_address;
        if let Some(&addr) = path_addr.get(path) {
            return Ok(base.relative(addr)?);
        }
        if nodes.contains_key(path)
            || add_targets.iter().any(|t| path.starts_with(t))
            || write_targets.contains(path)
        {
            return Err(Error::EditUnsupported(
                "an object-reference dataset targets a path this commit is still writing; \
                 use separate commits",
            ));
        }
        // After the three above; see this function's doc for why, and for what
        // that ordering does and does not buy.
        if delete_targets.iter().any(|d| path.starts_with(d)) {
            return Err(Error::EditUnsupported(
                "an object-reference dataset targets an object this commit deletes, or one \
                 under it; the reference would be left pointing at storage the delete can \
                 reclaim",
            ));
        }
        Ok(
            match crate::group_v2::resolve_path_any_from_source(
                src,
                AccessMode::ReadWrite,
                superblock,
                &path.as_path(),
            ) {
                Ok(addr) => base.relative(addr)?,
                Err(_) => StoredAddress::new(UNDEF),
            },
        )
    }

    /// Prove, before any byte of this commit is written, that every
    /// object-reference target across every staged dataset will resolve
    /// successfully — either against a pre-existing untouched object or
    /// against something this same commit places. [`resolve_reference_target`]
    /// classifies a target purely from *whether* an [`ObjectPathBuf`] has been placed
    /// yet (`path_addr.get`), never from the address *value*, so replaying the
    /// apply loop's placement order here with a placeholder address standing in
    /// for "already placed" reproduces the exact same verdict the apply loop's
    /// own calls will reach later, without writing anything.
    ///
    /// The placeholder is the file's **base address**, not zero. Any value
    /// reproduces the verdict, but it is also handed to the same `addr - base`
    /// that converts a real address into its stored form: a zero placeholder
    /// underflows that on every file with a userblock, panicking in a debug
    /// build on an otherwise legal edit. At the base address it converts to a
    /// stored `0`, which is exactly the "placed, real address not known yet"
    /// this stands for. If
    /// this preflight pass returns `Ok`, none of the apply loop's own
    /// `resolve_reference_target` calls can fail, so a reference-resolution
    /// error can no longer leave earlier-processed groups' real writes
    /// orphaned in the file (the failure surfaces here instead, before the
    /// apply loop's first `place`/`write_at`).
    ///
    /// [`resolve_reference_target`]: Self::resolve_reference_target
    fn preflight_reference_targets(
        keys: &[ObjectPathBuf],
        flat: &BTreeMap<&ObjectPathBuf, Vec<&FlatDataset>>,
        nodes: &BTreeMap<ObjectPathBuf, Node>,
        add_targets: &[ObjectPathBuf],
        write_targets: &[ObjectPathBuf],
        delete_targets: &[ObjectPathBuf],
        invalidated: &InvalidatedAddresses,
        src: &(impl Source + ?Sized),
        superblock: &Superblock,
    ) -> Result<(), Error> {
        let mut by_depth = keys.to_vec();
        // Stable on purpose, for the same reason as the apply loop's copy: this
        // simulation is only faithful if it walks the groups in that same order.
        by_depth.sort_by_key(|k| std::cmp::Reverse(k.len()));
        let mut sim_addr: BTreeMap<ObjectPathBuf, u64> = BTreeMap::new();
        for key in &by_depth {
            if let Some(datasets) = flat.get(key) {
                // Mirrors the apply loop's `group_datasets.sort_by_key(|fd|
                // fd.reference_targets.is_some())`: non-reference datasets are
                // placed (and so become resolvable) before any reference
                // dataset in the same group.
                let mut ordered: Vec<&FlatDataset> = datasets.to_vec();
                // Stable on purpose, like the loop it mirrors.
                ordered.sort_by_key(|fd| fd.reference_targets.is_some());
                for fd in ordered {
                    if let Some(patches) = &fd.reference_targets {
                        for patch in patches {
                            Self::resolve_reference_target(
                                &patch.target,
                                &sim_addr,
                                nodes,
                                add_targets,
                                write_targets,
                                delete_targets,
                                invalidated,
                                src,
                                superblock,
                            )?;
                        }
                    }
                    sim_addr.insert(key.join(&fd.name), superblock.base_address.get());
                }
            }
            sim_addr.insert(key.clone(), superblock.base_address.get());
        }
        Ok(())
    }

    /// Lay out a chunked / filtered / extensible dataset and return its object
    /// header bytes (which the caller links into the parent group).
    ///
    /// The chunk data and index (fixed-array / extensible-array, with any filter
    /// pipeline applied) are produced as one relocatable blob, whose internal
    /// layout — and therefore total size — is independent of the base address it
    /// is given. That is what lets the dataset be *sized before it is placed*
    /// ([`chunked_data_len`]): the blob goes into a freed region big enough to
    /// hold it, or at end-of-file when there is none, and is then assembled for
    /// whichever address it got, so every absolute address it embeds (chunk
    /// addresses, index-structure addresses, the addresses in the data-layout
    /// message) lands exactly where the bytes are written. The header is then
    /// built with [`build_chunked_dataset_oh`] — the same function the whole-file
    /// writer uses — so the header is byte-identical to one written fresh.
    ///
    /// The dataset's single pass of the filter pipeline happens in
    /// [`compress_chunks`], before either decision; sizing and assembly work from
    /// the compressed set, so choosing an address never costs a recompression
    /// (issue #261).
    fn build_chunked_dataset(&mut self, fd: &FlatDataset) -> Result<Vec<u8>, Error> {
        let chunk_dims = fd.chunk_options.resolve_chunk_dims(&fd.ds.dimensions);
        let ctx = ChunkContext::from_datatype(&chunk_dims, &fd.dt)?;
        // The overhang of a partial edge chunk holds the staged fill value
        // (issue #296).
        let elem = ctx.element_size.narrow::<NonZeroUsize>()?;
        let fill = crate::fill_value::FillPattern::new(fd.fill.as_deref(), elem);
        let set = compress_chunks(
            &fd.raw,
            &fd.ds.dimensions,
            ctx,
            &fd.chunk_options,
            fd.maxshape.as_deref(),
            fill,
            // `flatten_dataset` refuses an unallocated dataset outright, so
            // every dataset this engine builds allocates its storage.
            StorageAllocation::Allocated,
        )?;
        // Chunk data and the index beside it are raw (see `chunked_storage_spans`,
        // which reclaims both as raw).
        let (_addr, (layout_message, pipeline_message)) =
            self.place_relocatable(chunked_data_len(&set)?, PageType::Raw, |stored_base| {
                let result = assemble_chunked_at(&set, stored_base)?;
                Ok((
                    result.data_bytes,
                    (result.layout_message, result.pipeline_message),
                ))
            })?;
        // As in the contiguous branch: a set too large for the header goes to a
        // fractal heap, which the header then names instead of carrying it.
        let attr_info = self.place_dense_attrs_if_needed(fd)?;
        Ok(build_chunked_dataset_oh(
            &fd.dt,
            &DatatypeLocation::Inline,
            &fd.ds,
            &layout_message,
            pipeline_message.as_deref(),
            &fd.attrs,
            attr_info.as_deref(),
            fd.fill.as_deref(),
        )?)
    }

    /// Returns the on-disk byte spans `(addr, len)` of every chunk of the
    /// version 2 object header at `addr`: chunk 0 (signature, prefix, messages,
    /// checksum) plus each continuation (`OCHK`) block.
    ///
    /// Used to reclaim a header's storage when its object is deleted. An error
    /// (propagated from [`oh_region_at`] or a malformed continuation) means the
    /// header is not a plain v2 header this engine can fully account for, and
    /// the caller leaves it as dead bytes of unknown extent.
    fn oh_chunk_spans(&self, addr: u64) -> Result<Vec<(u64, u64)>, Error> {
        Ok(
            read_oh_chunks(&self.image(), addr, self.superblock.base_address)?
                .into_iter()
                .map(|chunk| chunk.span)
                .collect(),
        )
    }

    /// Count, for every object-header address reachable from the root, how many
    /// hard links in the *pre-commit* file point to it. The result drives the
    /// last-hard-link reclaim guard in [`collect_free_spans`](Self::collect_free_spans):
    /// an object is freed only when its count is 1.
    ///
    /// Walks the whole link graph from the root, following hard links through
    /// groups of any on-disk format (v0/v1 symbol-table, v2 compact, v2 dense)
    /// via [`resolve_group_entries_from_source`], tallying each hard-link edge. Datasets and
    /// other leaves contribute no edges. Returns `None` — so the caller reclaims
    /// nothing for the deletions, a safe leak — if the graph cannot be walked in
    /// full: an unparseable header, a group whose links cannot be enumerated, or
    /// more than [`MAX_LINK_GRAPH_NODES`] objects. Cycles are handled by visiting
    /// each object once. Base-aware: stored child addresses are shifted by the
    /// userblock base, so the returned keys are absolute file offsets.
    fn count_incoming_hard_links(&self) -> Option<HashMap<u64, u32>> {
        let os = self.superblock.offset_size;
        let ls = self.superblock.length_size;
        let base = self.superblock.base_address;
        let mut counts: HashMap<u64, u32> = HashMap::new();
        let mut visited: HashSet<u64> = HashSet::new();
        let mut stack: Vec<u64> = vec![self.superblock.root_group_address];
        let mut budget = MAX_LINK_GRAPH_NODES;
        while let Some(addr) = stack.pop() {
            if !visited.insert(addr) {
                continue; // already expanded (also breaks hard-link cycles)
            }
            if budget == 0 {
                return None; // graph larger than we will walk; leak conservatively
            }
            budget -= 1;
            let header = ObjectHeader::parse_from_source(
                &self.image(),
                AccessMode::ReadWrite,
                addr,
                os,
                ls,
                base,
            )
            .ok()?;
            // Datasets and other leaves are not groups and own no links.
            let is_group = header.messages.iter().any(|m| {
                matches!(
                    m.msg_type,
                    MessageType::SYMBOL_TABLE | MessageType::LINK | MessageType::LINK_INFO
                )
            });
            if !is_group {
                continue;
            }
            // A group we cannot enumerate fully would undercount incoming links
            // and risk over-reclaim; bail to the safe-leak fallback instead.
            let entries =
                resolve_group_entries_from_source(&self.image(), &header, os, ls, base).ok()?;
            for e in entries {
                let child = base.absolute(e.object_header_address).ok()?;
                *counts.entry(child).or_insert(0) += 1;
                stack.push(child);
            }
        }
        Some(counts)
    }

    /// Collects, as far as it can account for them, the span and free class of
    /// every on-disk block the object at `addr` owns, and for a group its whole
    /// subtree, into `out` for reclamation after a delete.
    ///
    /// Contiguous datasets (header + data block), chunked datasets (header +
    /// chunk index + chunk data, via [`chunked_storage_spans`](Self::chunked_storage_spans)),
    /// and whole group subtrees are reclaimed. Deliberately conservative: any
    /// object whose layout it cannot fully account for — a non-v2 header, an
    /// unsupported or only-partially-enumerable chunk index, a group holding a
    /// soft/external link, dense attribute storage — contributes nothing and is
    /// not descended into, so `out` never names a region that might still be in
    /// use. Bounded by [`MAX_COPY_DEPTH`] against a hard-link cycle.
    /// Variable-length data in global-heap collections is never reclaimed here (a
    /// collection can be shared between objects), so it is simply left behind.
    ///
    /// `incoming` is the file-wide hard-link count per object-header address
    /// (from [`count_incoming_hard_links`](Self::count_incoming_hard_links)). An
    /// object is reclaimed — and, for a group, descended into — only when its
    /// count is exactly 1, i.e. the link being removed is its last: an object
    /// still reachable through another hard link is live and is left untouched
    /// (so is everything below a surviving group), which is what keeps deleting
    /// one of several hard links from corrupting the survivor.
    fn collect_free_spans(
        &self,
        addr: u64,
        depth: u32,
        incoming: &HashMap<u64, u32>,
        out: &mut Vec<(u64, u64, FreeClass)>,
    ) {
        // `addr` is an absolute file offset (the caller resolves it from the live
        // file, and the group recursion below re-absolutizes each child). `incoming`
        // is keyed by absolute offset, and `oh_chunk_spans`/`chunked_storage_spans`
        // both take an absolute address and return absolute spans, so the whole
        // walk works in absolute file offsets. The one shift this method must apply
        // itself is on the stored addresses `read_object` returns for a contiguous
        // data block and a group's child links: each is converted to an absolute
        // offset (a no-op on a base-0 file) before it is bounds-checked, recorded,
        // or descended into.
        let base = self.superblock.base_address;
        let file_len = self.image().len();
        if depth >= MAX_COPY_DEPTH {
            return;
        }
        // Reclaim only when this delete removes the object's last hard link. A
        // count other than 1 (it has surviving links, or the graph walk could
        // not account for it) means the object — and a group's whole subtree —
        // stays live and must not be freed.
        if incoming.get(&addr) != Some(&1) {
            return;
        }
        // The header's own chunks. If they cannot be mapped, account for nothing.
        let spans = match self.oh_chunk_spans(addr) {
            Ok(s) => s,
            Err(_) => return,
        };
        match Self::read_object(
            &self.image(),
            AccessMode::ReadWrite,
            addr,
            self.superblock.base_address,
        ) {
            Ok(ObjModel::DatasetVerbatim { .. }) => out.extend(meta_spans(spans)),
            Ok(ObjModel::DatasetContiguous {
                data_addr,
                data_size,
                ..
            }) => {
                out.extend(meta_spans(spans));
                // A defined, in-bounds contiguous data block is owned outright;
                // an empty dataset stores the undefined address and owns none. Its
                // absolute file offset is what this bounds-checks and records.
                if !data_addr.is_undefined(OFFSET_SIZE) && data_size > 0 {
                    if let Ok(abs) = base.absolute(data_addr) {
                        if abs.checked_add(data_size).is_some_and(|e| e <= file_len) {
                            // A contiguous data block is raw data.
                            out.push((abs, data_size, FreeClass::Page(PageType::Raw)));
                        }
                    }
                }
            }
            Ok(ObjModel::Group { children, .. }) => {
                out.extend(meta_spans(spans));
                // The recursion works in absolute offsets, matching `incoming`'s
                // keys and `oh_chunk_spans`.
                for (_, _, child) in children {
                    if let Ok(c) = base.absolute(child) {
                        self.collect_free_spans(c, depth + 1, incoming, out);
                    }
                }
            }
            // A chunked dataset: reclaim its chunk index and chunk data blocks
            // alongside its header. `chunked_storage_spans` returns `None` for
            // anything it cannot account for exhaustively (an index type with no
            // walker, an undefined index address, or spans that fail the
            // bounds/overlap check), leaving the whole dataset as dead bytes
            // rather than freeing a region that might still be in use.
            Ok(ObjModel::DatasetChunked { .. }) => {
                if let Some(storage) = self.chunked_storage_spans(addr) {
                    out.extend(meta_spans(spans));
                    // Already page-typed: chunk data raw, index structure metadata.
                    out.extend(storage);
                }
            }
            // A truly unsupported object (one `read_object` cannot model): leave
            // its bytes in place rather than guess its extent.
            //
            // An externally stored dataset lands here as of #336, where before it
            // was modelled as the contiguous dataset it structurally is and had
            // its header chunks reclaimed. Deleting one therefore leaves those
            // chunks behind: measured on the fixture in
            // `crates/crosscheck/tests/main/external_storage.rs`, a commit that deletes it
            // reports 147 B reusable where it reported 431 B. That is a leak and
            // not a hazard — `oh_chunk_spans` never covered the local heap the
            // External Data Files message names either, so neither the old
            // behaviour nor this one accounted for the whole object — and it is
            // the price of stating the refusal once, in the one function both the
            // copy planner and this walk read objects through.
            Err(_) => {}
        }
    }

    /// Returns every on-disk block a *chunked* dataset at `addr` owns: its
    /// chunk index structure (B-tree v1 nodes, or fixed- / extensible-array
    /// header, index, super, and data blocks) plus every allocated chunk data
    /// block.
    ///
    /// The object-header chunks are freed by the caller
    /// ([`collect_free_spans`](Self::collect_free_spans)). This returns only the
    /// storage the data-layout message points at.
    ///
    /// Returns `None` — contribute nothing, leave the object as dead bytes —
    /// whenever the dataset cannot be enumerated *exhaustively* and safely: a
    /// header that does not parse or is not a chunked dataset, a chunk index
    /// with no walker (a version 2 B-tree, index type 5), an undefined index
    /// address (an empty, never-written dataset), or any resulting span that
    /// falls outside the file image or overlaps another. This upholds the
    /// editor's invariant that reclaimed space is never a region still in use:
    /// under-reclaiming only wastes space, while over-reclaiming would corrupt.
    ///
    /// Chunk data addresses and sizes come from the same index walkers the
    /// reader uses, so they match the bytes the writer laid down exactly. The
    /// per-layout enumeration lives in
    /// [`chunked_read::collect_chunked_storage_spans`](crate::chunked_read::collect_chunked_storage_spans);
    /// this method only locates the layout and dataspace messages and validates
    /// the result. Variable-length data in global-heap collections is still
    /// never reclaimed (a collection can be shared between objects); see the
    /// [module docs](self).
    fn chunked_storage_spans(&self, addr: u64) -> Option<Vec<(u64, u64, FreeClass)>> {
        // Locate the data-layout and dataspace messages in the object header.
        let region =
            Self::gather_oh_messages(&self.image(), addr, self.superblock.base_address).ok()?;
        let mut layout_msg: Option<(usize, usize)> = None;
        let mut dataspace_msg: Option<(usize, usize)> = None;
        let mut p = 0;
        loop {
            match region.next_message(p) {
                Ok(Some((msg_type, body, body_end))) => {
                    match msg_type {
                        MessageType::DATA_LAYOUT => layout_msg = Some((body, body_end)),
                        MessageType::DATASPACE => dataspace_msg = Some((body, body_end)),
                        _ => {}
                    }
                    p = body_end;
                }
                Ok(None) => break,
                Err(_) => return None,
            }
        }
        let (lb, le) = layout_msg?;
        let (db, de) = dataspace_msg?;

        let layout = DataLayout::parse(&region[lb..le], OFFSET_SIZE, LENGTH_SIZE).ok()?;
        if !matches!(layout, DataLayout::Chunked { .. }) {
            return None;
        }
        let dataspace = Dataspace::parse(&region[db..de], LENGTH_SIZE).ok()?;

        // Delegate the per-index-type enumeration to the chunked reader (the
        // single owner of chunk-storage layout knowledge), then validate: every
        // span must lie inside the current file image and be pairwise disjoint,
        // or the free list would later hand out live bytes (and a debug build
        // would panic on the double-free). On any error or violation, leave the
        // whole dataset unreclaimed rather than free a region still in use.
        //
        // The layout's stored addresses are relative to the userblock base, so the
        // enumeration runs on a base-relative view of the file and each returned
        // span address is shifted back to an absolute file offset by adding `base`
        // (a no-op on a base-0 file). The free list and the bounds check below both
        // work in absolute file offsets.
        let base = self.superblock.base_address;
        let split = crate::chunked_read::collect_chunked_storage_spans(
            &BaseOffsetSource {
                inner: &self.image(),
                base,
            },
            &layout,
            &dataspace,
            OFFSET_SIZE,
            LENGTH_SIZE,
        )
        .ok()?;
        // Chunk data is raw under every writer, so its spans are raw outright. The
        // index is only raw where this crate placed it — see
        // [`index_is_provably_raw`](Self::index_is_provably_raw) — and is recorded
        // as dead rather than free otherwise.
        let mut data: Vec<(u64, u64)> = Vec::with_capacity(split.data.len());
        for (addr, len) in split.data {
            data.push((base.absolute(StoredAddress::new(addr)).ok()?, len));
        }
        let mut index: Vec<(u64, u64)> = Vec::with_capacity(split.index.len());
        for (addr, len) in split.index {
            index.push((base.absolute(StoredAddress::new(addr)).ok()?, len));
        }
        // Validate both halves together — they must be disjoint from each other as
        // well as internally — before either is trusted, including by the proof.
        let mut plain: Vec<(u64, u64)> = data.iter().chain(index.iter()).copied().collect();
        if !spans_disjoint_in_bounds(&mut plain, self.image.len()) {
            return None;
        }
        let index_class = if self.index_is_provably_raw(&data, &index) {
            FreeClass::Page(PageType::Raw)
        } else {
            FreeClass::Dead
        };
        let mut spans: Vec<(u64, u64, FreeClass)> = data
            .iter()
            .map(|&(a, l)| (a, l, FreeClass::Page(PageType::Raw)))
            .collect();
        spans.extend(index.into_iter().map(|(a, l)| (a, l, index_class)));
        Some(spans)
    }

    /// Whether a chunked dataset's `index` provably occupies raw pages, given the
    /// `data` spans it accompanies. Both are absolute file offsets.
    ///
    /// A chunk index is *metadata* by the format's taxonomy, and the reference C
    /// library allocates one as such — out of metadata pages, nowhere near the
    /// chunk data. This crate instead lays a dataset's index down in the same run
    /// as its chunk data, inside raw pages, so that a reader following the layout
    /// message walks one contiguous blob. Both are valid; what matters on a paged
    /// file is that freeing an index records it under the page type it actually
    /// sits in, since that decides what a later allocation may overwrite there.
    ///
    /// Nothing in the file says which writer produced it, so the index is placed
    /// from the layout itself, by two tests it must pass together: some chunk-data
    /// span must **abut** it — ending exactly where the index begins, or beginning
    /// exactly where it ends — and no byte of it may lie in **page 0**.
    ///
    /// The abutment is this crate's single blob, and is not a layout the reference
    /// library sets out to produce. A chunk index is metadata to that library,
    /// allocated out of metadata pages — and those are allocated from the bottom
    /// of the file, while large raw data goes above, so its index lands far below
    /// the chunk data it indexes rather than against it. Measured across seven
    /// files it wrote (Extensible and Fixed Array indexes, page sizes
    /// 512/4096/8192, 8 to 16 chunks): the index began in page 0 every time, with
    /// the chunk data starting at page 9 or higher, so neither join address was
    /// within reach of it.
    ///
    /// The page-0 screen closes the one way that measured layout can nonetheless
    /// satisfy the abutment. An index block that ends exactly on a page boundary
    /// with a raw page after it — a Fixed Array data block that exactly fills its
    /// page, say — is abutted by the first chunk in that raw page, and the whole
    /// index run would then be filed as raw, its header included, *even though
    /// that header sits in page 0 beside the superblock*. A later raw allocation
    /// would land inside a metadata page, which is the page-mixing defect of
    /// issue #261. Page 0 of a paged file always begins with the superblock, so it
    /// is a metadata page by construction and an index with a byte in it cannot be
    /// raw whatever abuts it. The screen costs this crate nothing: it allocates
    /// chunk data and the indexes beside it out of raw pages, and page 0 — never
    /// wholly free, since the superblock lives there — is never one of them.
    ///
    /// The tighter rule considered instead was to require the join address to fall
    /// *strictly inside* a page, which proves rawness outright: the byte before
    /// the join and the byte after it are then in one page, and a page holding
    /// chunk data is raw. It is sound, and a paged file the reference library
    /// wrote cannot satisfy it at all, since such a file never puts raw and
    /// metadata bytes in one page and so can only ever join them on a boundary.
    /// It was rejected because it also refuses layouts *this* crate produces:
    /// whenever a blob's chunk bytes total a page multiple from a page-aligned
    /// start, its index begins on a boundary too. Measured over 100
    /// `append_staged` commits at a 4096-byte page with 2048-byte chunks, that
    /// stranded about 500 bytes per commit — 62 KB against the 10 KB the abutment
    /// rule left — because the two writers produce the same geometry there and
    /// nothing in the file distinguishes them.
    ///
    /// Both sides have to be admitted, not just the index that follows its data.
    /// A staged append keeps the existing chunks where they are and places the new
    /// ones past the index it is superseding, so from the second append onward the
    /// dataset has chunk data on both sides of that index and the highest data
    /// address is no longer the one beside it. Asking only whether the index
    /// begins where the *last* chunk ends refused every such index, and a
    /// repeatedly appended dataset dropped one per commit (issue #388).
    ///
    /// An index the test does not place is [dead](FreeClass::Dead) to the caller
    /// rather than free: its bytes are no longer in use, but advertising space
    /// inside a possible metadata page for raw reuse would mix the page
    /// (issue #261), so they are reusable only once the whole page around them is
    /// empty.
    ///
    /// A non-paged file has no page types to keep apart, so everything is
    /// reclaimable there; the tag is ignored by its commit tail entirely.
    fn index_is_provably_raw(&self, data: &[(u64, u64)], index: &[(u64, u64)]) -> bool {
        match &self.paged {
            None => true,
            Some(paged) => {
                index_abuts_chunk_data(data, index)
                    && !index_touches_page_zero(index, paged.page_size)
            }
        }
    }

    /// Returns every on-disk byte span of a chunked dataset's *index structure
    /// only* (not its chunk data), for reclaiming the old index after a
    /// relocating append ([`MovingWrite::AppendedChunks`]) that keeps the chunk
    /// data in place.
    ///
    /// This is the index-only counterpart of
    /// [`chunked_storage_spans`](Self::chunked_storage_spans), delegating to
    /// [`chunk_index_spans_from_source`], which enumerates the index structure's
    /// own blocks (B-tree v1 nodes, or fixed- / extensible-array header, index,
    /// super, and data blocks) and never a chunk-data address, so the shared
    /// kept chunk data is never freed. Base-aware and validated
    /// disjoint/in-bounds. Returns `None` (leave unreclaimed) on any error or
    /// violation.
    fn chunked_index_spans(&self, addr: u64) -> Option<Vec<(u64, u64)>> {
        let region =
            Self::gather_oh_messages(&self.image(), addr, self.superblock.base_address).ok()?;
        let mut layout_msg: Option<(usize, usize)> = None;
        let mut p = 0;
        loop {
            match region.next_message(p) {
                Ok(Some((msg_type, body, body_end))) => {
                    if msg_type == MessageType::DATA_LAYOUT {
                        layout_msg = Some((body, body_end));
                    }
                    p = body_end;
                }
                Ok(None) => break,
                Err(_) => return None,
            }
        }
        let (lb, le) = layout_msg?;
        let layout = DataLayout::parse(&region[lb..le], OFFSET_SIZE, LENGTH_SIZE).ok()?;
        if !matches!(layout, DataLayout::Chunked { .. }) {
            return None;
        }
        let base = self.superblock.base_address;
        let mut spans = chunk_index_spans_from_source(
            &BaseOffsetSource {
                inner: &self.image(),
                base,
            },
            &layout,
            OFFSET_SIZE,
            LENGTH_SIZE,
        )
        .ok()?;
        for (a, _) in &mut spans {
            *a = base.absolute(StoredAddress::new(*a)).ok()?;
        }
        if !spans_disjoint_in_bounds(&mut spans, self.image.len()) {
            return None;
        }
        Some(spans)
    }
}

/// A dirty group in the edit plan: its base object-header message region and the
/// additions targeting it.
#[derive(Default)]
struct Node {
    is_new: bool,
    /// Names of links to remove from this group (from `delete`).
    deletes: Vec<LinkNameBuf>,
    /// Copies to add to this group: (new link name, the source subtree read out
    /// for writing). Built at staging time from either this file (an in-file
    /// [`copy`](crate::File::copy)) or another open file (a cross-file
    /// [`copy_from`](crate::File::copy_from)).
    copies: Vec<(LinkNameBuf, CopyTree)>,
    /// New link names this commit adds to this group by cross-file copy. The
    /// subtrees themselves stay in the staged set — which the preflight may
    /// still refuse, and must not empty (issue #316) — and join `copies` at the
    /// point of no return.
    cross_copies: Vec<LinkNameBuf>,
    /// Value overwrites whose dataset header relocates (a resize or compact
    /// rewrite by `write_dataset`, a staged append, an attribute edit), as (child
    /// link name, the pre-commit object-header address, the relocation plan). On
    /// apply, the new data and header are written and this group's existing link
    /// to the moved header is patched to its new address — exactly like an
    /// existing child group's link. The old address is carried rather than
    /// re-resolved: the screen above and the reclaim below both need it, and one
    /// derivation cannot disagree with itself.
    writes: Vec<(LinkNameBuf, u64, MovingWrite)>,
    base_region: OhRegion,
    existing_links: Vec<String>,
    /// What this group's attribute edits left for the apply loop, staged by
    /// [`plan_attr_ops`] and resolved by [`WriteEngine::place_edited_attrs`]
    /// right before this node's header is built. Default (an empty compact set)
    /// for a group with no attribute edits.
    attrs: EditedAttrs,
}

/// A staged compact attribute edit for a group or dataset (shared by
/// [`Group::set_attr`](crate::Group::set_attr)/`remove_group_attr` and
/// [`Dataset::set_attr`](crate::Dataset::set_attr)/`remove_dataset_attr`).
enum AttrOp {
    Set { name: String, value: AttrValue },
    Remove { name: String },
}

/// A source object parsed for copying. Headers are reproduced from their
/// verbatim message bytes; only the contiguous data address and child link
/// targets are repointed to the freshly-written copies.
enum ObjModel {
    /// A compact dataset (data inline in the header): copy the region verbatim.
    /// `dense_attrs` is empty unless the source stored its attributes densely, in
    /// which case the Attribute Info message and inline Attribute messages have
    /// been stripped from `region` and the parsed set is carried here to be
    /// re-emitted into a fresh fractal heap on write.
    DatasetVerbatim {
        region: OhRegion,
        dense_attrs: DenseAttrSet,
    },
    /// A contiguous dataset: copy the region, repointing the data address at
    /// `addr_off` (region-relative) to a fresh copy of `[data_addr, +data_size)`.
    /// See [`DatasetVerbatim`](ObjModel::DatasetVerbatim) for `dense_attrs`.
    DatasetContiguous {
        region: OhRegion,
        addr_off: usize,
        data_addr: StoredAddress,
        data_size: u64,
        dense_attrs: DenseAttrSet,
    },
    /// A chunked (and possibly filtered) dataset: the verbatim header `region`
    /// (datatype, dataspace, fill value, data layout, and filter pipeline kept as
    /// written). The chunk data is not captured here — [`read_copy_subtree`](WriteEngine::read_copy_subtree)
    /// enumerates and reads the chunks (it holds the source buffer), repointing the
    /// rebuilt index on write. See [`DatasetVerbatim`](ObjModel::DatasetVerbatim)
    /// for `dense_attrs`.
    DatasetChunked {
        region: OhRegion,
        dense_attrs: DenseAttrSet,
    },
    /// A group: every non-link message verbatim, plus its hard-link children to
    /// copy and re-link as `(name, creation index, target address)`. See
    /// [`DatasetVerbatim`](ObjModel::DatasetVerbatim) for `dense_attrs`.
    Group {
        non_link_region: OhRegion,
        children: Vec<(String, Option<u64>, StoredAddress)>,
        dense_attrs: DenseAttrSet,
    },
}

/// An object subtree fully read out of a source buffer and owning every byte it
/// will write, the read result of [`WriteEngine::read_copy_subtree`] and the
/// input to [`WriteEngine::write_copy_subtree`]. Unlike [`ObjModel`] (a single
/// object still referencing source addresses) it is recursive and self-contained:
/// a contiguous dataset owns its data bytes, and a group owns its children, so it
/// can be written into the destination without the source buffer still in hand —
/// which is what lets a cross-file copy read the source at staging time and apply
/// it at commit time.
enum CopyTree {
    /// A compact dataset: the header region is written verbatim (data is inline).
    /// `dense_attrs`, when non-empty, is re-emitted into a freshly built fractal
    /// heap appended just before the header, whose Attribute Info message is
    /// spliced into the region on write.
    DatasetVerbatim {
        region: OhRegion,
        dense_attrs: DenseAttrSet,
    },
    /// A contiguous dataset: `data` is written first and its new address patched
    /// into the header `region` at `addr_off` before the header is written. See
    /// [`DatasetVerbatim`](CopyTree::DatasetVerbatim) for `dense_attrs`.
    ///
    /// `data` is `None` when the source allocated no storage at all: nothing is
    /// written and `addr_off` keeps the undefined address the source stored, so
    /// the copy declares the same absent storage rather than a block at a real
    /// address. That is a statement about storage, not about length (issue
    /// #336).
    DatasetContiguous {
        region: OhRegion,
        addr_off: usize,
        data: Option<Vec<u8>>,
        dense_attrs: DenseAttrSet,
    },
    /// A chunked (and possibly filtered) dataset. The header `region` is written
    /// verbatim except its data-layout message, which is swapped for one naming the
    /// freshly rebuilt index; `chunk_bytes` (each chunk's already-compressed bytes,
    /// in dense row-major grid order, with sizes/masks in `meta`) and the source
    /// `pipeline_message` are carried unchanged, so the copy preserves the filter
    /// pipeline and chunk payloads byte-for-byte. The on-disk index *type* is
    /// reselected from `maxshape`/chunk count (single / fixed-array / extensible-
    /// array), so a B-tree-v1 or implicit source is reproduced with a v4 index. See
    /// [`DatasetVerbatim`](CopyTree::DatasetVerbatim) for `dense_attrs`.
    DatasetChunked {
        region: OhRegion,
        /// The dataset's current shape. Held because the index's element
        /// numbering is taken over the *maximum* chunk grid, which needs both
        /// extents (see [`crate::chunk_grid`]).
        shape: Vec<u64>,
        chunk_dims: Vec<u64>,
        element_size: NonZeroUsize,
        maxshape: Option<Vec<MaxExtent>>,
        pipeline_message: Option<Vec<u8>>,
        meta: Vec<ChunkMeta>,
        chunk_bytes: Vec<Vec<u8>>,
        dense_attrs: DenseAttrSet,
    },
    /// A group: every non-link message verbatim, plus the `(name, creation
    /// index, subtree)` children to write first and re-link by name. See
    /// [`DatasetVerbatim`](CopyTree::DatasetVerbatim) for `dense_attrs`.
    Group {
        non_link_region: OhRegion,
        children: Vec<(String, Option<u64>, CopyTree)>,
        dense_attrs: DenseAttrSet,
    },
}

/// The addresses this commit invalidates, against which a reference it writes is
/// screened (issue #317).
///
/// An object reference *is* an object-header address, so a commit can falsify one
/// two ways. It can **remove** the object: a deletion frees its header and
/// storage, and a group's whole subtree with it
/// ([`WriteEngine::collect_free_spans`] is the walk that enumerates it), so the
/// address survives the commit only until the next allocation reuses the span and
/// then reads as whatever landed there. Or it can **move** it: a dirty group and
/// a relocating dataset write are both rebuilt at a fresh address, and the old
/// header is freed once the superblock is repointed.
///
/// Both are screened here because on the *path* side neither can come out as a
/// pre-commit address: [`WriteEngine::resolve_reference_target`] refuses a deleted
/// path by name, and a dirty group or a write target it either refuses as "still
/// writing" or — once this commit has placed the target — resolves to the address
/// it lands on. An address that skipped either check would be the one form that
/// writes the value the commit is vacating, which is the shape of the defect this
/// exists to close.
///
/// The removal spans are the very ones the commit hands to the free-space
/// manager, taken from one walk rather than two, so the screen and the reclaimer
/// cannot come to disagree about what this commit removes.
struct InvalidatedAddresses {
    /// Absolute `(offset, length)` file spans this commit's deletions reclaim, as
    /// [`collect_free_spans`](WriteEngine::collect_free_spans) reports them.
    removed: Vec<(u64, u64)>,
    /// Absolute pre-commit object-header addresses this commit rewrites
    /// elsewhere: every existing group it dirties, and every dataset write that
    /// relocates (a resizing or compact overwrite, a staged append, an attribute
    /// edit). An in-place overwrite does not relocate and is not here. The path
    /// side refuses it anyway, deliberately conservatively — `write_targets`
    /// records that a dataset is written, not which plan it got — while this list
    /// is filled from the plans themselves.
    moved: Vec<u64>,
    /// The superblock base address. A stored object reference is *base-relative*
    /// and both lists above are absolute, so the comparison needs it; it is zero
    /// for every file without a userblock.
    base: BaseAddress,
}

impl InvalidatedAddresses {
    /// Whether this commit invalidates anything at all, so a caller with nothing
    /// cheap to check can skip its own work. Worth testing on the copy screen,
    /// whose `moved` is empty by construction; the supplied screen's never is,
    /// since every commit that builds one rebuilds its root group.
    fn is_empty(&self) -> bool {
        self.removed.is_empty() && self.moved.is_empty()
    }

    /// The refusal `stored` earns — one object-reference element exactly as it is
    /// written to disk — or `None` when it still names what it named.
    ///
    /// The null (`0`) and undefined ([`UNDEF`]) references name no object at all,
    /// so neither is screened: the same two values
    /// [`crate::repack()`] copies through verbatim without
    /// resolving, and the two [`crate::reader`] refuses to dereference.
    fn refusal(&self, stored: StoredAddress) -> Option<&'static str> {
        if stored.get() == 0 || stored.get() == UNDEF {
            return None;
        }
        // An address that cannot even be shifted into the file is not one of
        // ours; leave it to whatever reads it.
        let abs = self.base.absolute(stored).ok()?;
        if self
            .removed
            .iter()
            .any(|&(off, len)| abs >= off && abs - off < len)
        {
            return Some(REFERENCE_INTO_RECLAIMED_SPACE);
        }
        if self.moved.contains(&abs) {
            return Some(REFERENCE_TO_A_MOVED_OBJECT);
        }
        None
    }
}

/// The validated, chunk-collapsed message region and existing link names of a
/// group header.
struct GroupInfo {
    region: OhRegion,
    link_names: Vec<String>,
}

/// Element bytes staged for a value overwrite, together with the
/// variable-length staging (if any) whose global heap collections must be
/// placed — and whose addresses patched into the element references in `raw` —
/// before those bytes reach the file.
///
/// The two halves travel together because resolving them is an **apply-phase**
/// step while planning is a preflight one: placing a collection allocates, and
/// [`commit`](WriteEngine::commit)'s preflight only reads. So a plan carries
/// the bytes unresolved and every consumer resolves them where it writes, with
/// [`resolve_overwrite_bytes`](WriteEngine::resolve_overwrite_bytes) (issue
/// #321).
///
/// Patching never changes `raw`'s length — an element reference is fixed-width
/// whatever string it names — which is what lets the plan be *chosen* from the
/// unresolved bytes. The one place that is not enough is a filtered chunked
/// dataset, whose compressed chunk sizes do depend on the addresses: see
/// [`ChunkPayload::Deferred`].
struct OverwriteBytes {
    raw: Vec<u8>,
    /// `None` for every overwrite that stages no variable-length data, which is
    /// all of them but a `with_vlen_strings` one.
    vlen: Option<VlenOverwrite>,
}

/// The variable-length half of an [`OverwriteBytes`]: the staged collections to
/// place, and the path they are recorded under once placed.
struct VlenOverwrite {
    staging: VlStringStaging,
    /// The overwritten dataset's path, the key its collections are recorded
    /// under in [`vl_overwrite_heaps`](WriteEngine::vl_overwrite_heaps) — and
    /// the key the ones a previous overwrite left are read back from, to be
    /// freed once this commit lands.
    path: ObjectPathBuf,
}

impl OverwriteBytes {
    /// Bytes that need no resolving, and so name nothing to record or free.
    fn ready(raw: Vec<u8>) -> Self {
        Self { raw, vlen: None }
    }
}

/// How a staged value overwrite (`write_dataset`) will be applied, decided by
/// [`WriteEngine::prepare_write`] during the all-or-nothing preflight.
// `allow` rather than `expect`, which is otherwise this module's habit: the lint
// fires on a 64-bit pointer width and not on i686, where the `Vec` and `usize`
// fields inside `MovingWrite` shrink and the variant gap falls back under
// clippy's threshold. An `expect` is therefore unfulfilled on exactly the two
// 32-bit CI jobs, both of which deny `unfulfilled_lint_expectations`.
#[allow(
    clippy::large_enum_variant,
    reason = "boxing the large variant would cost more than it saves: this enum is returned by \
              `prepare_write` and destructured at its one call site, which moves the `MovingWrite` \
              straight into a `Vec<MovingWrite>` that stores it unboxed either way -- so a box \
              here adds an allocation and a free without removing a single copy"
)]
enum WritePlan {
    /// A contiguous dataset whose new data is the same length as its existing,
    /// defined data block: overwrite the bytes straight in place at `data_addr`.
    /// No object header is rewritten.
    ///
    /// The superblock root is not flipped either — *unless* the bytes still
    /// carry variable-length staging, which appends a heap collection and so
    /// moves end-of-file, a figure only the superblock records. `commit`'s
    /// fast path excludes exactly that case (issue #321).
    InPlace {
        data_addr: u64,
        bytes: OverwriteBytes,
    },
    /// A chunked dataset overwritten chunk-by-chunk in place: each `(addr, bytes)`
    /// pair is written straight over an existing chunk slot. Used when every new
    /// (re-encoded) chunk is the same byte length as the slot it replaces — an
    /// unfiltered chunked overwrite (chunk sizes are fixed by the unchanged shape)
    /// or a filtered one whose re-encoded chunks happen to match. Like
    /// [`InPlace`](WritePlan::InPlace) it touches no header and no chunk index, so
    /// the superblock root is not flipped.
    InPlaceChunks { writes: Vec<(u64, Vec<u8>)> },
    /// The dataset's header relocates: a contiguous resize, a compact rewrite, or
    /// a chunked rebuild. The parent group is rebuilt and its link patched. See
    /// [`MovingWrite`].
    Moving(MovingWrite),
}

/// A value overwrite that relocates the dataset's object header — a contiguous
/// dataset whose data length changed (or had no data block) or a compact dataset
/// whose inline bytes are replaced. On apply the new data and a rewritten header
/// are written at end-of-file (or into reusable freed space), and the parent
/// group's link is repointed at the new header address.
enum MovingWrite {
    /// A contiguous dataset: write `raw` elsewhere, patch the data-layout address
    /// at `addr_off` in the verbatim header `region`, rewrite the header, and free
    /// `old_extent` (the prior data block, if any) after the commit lands.
    Contiguous {
        region: OhRegion,
        addr_off: usize,
        bytes: OverwriteBytes,
        old_extent: Option<(u64, u64)>,
    },
    /// A compact dataset: rebuild the header `region` with the bytes inline.
    Compact {
        region: OhRegion,
        bytes: OverwriteBytes,
    },
    /// A chunked dataset whose new (re-encoded) chunks do not all fit their
    /// existing slots, so its whole storage is rebuilt and relocated. A fresh
    /// chunk-data blob and index are placed — in a freed region that fits, else at
    /// end-of-file (via the verbatim
    /// layout path, carrying the chunk bytes and the source filter `pipeline_message`
    /// unchanged — no recompression and no filter-parameter reconstruction), the
    /// data-layout message in the verbatim header `region` is swapped for the new
    /// one (every other header message — datatype, dataspace, fill value, filter
    /// pipeline, and attributes, including a dense attribute heap referenced by an
    /// untouched Attribute Info message — is preserved verbatim), and the old
    /// chunk storage at `old_addr` is freed after the commit lands.
    Chunked {
        region: OhRegion,
        /// See [`CopyTree::DatasetChunked::shape`].
        shape: Vec<u64>,
        chunk_dims: Vec<u64>,
        element_size: NonZeroUsize,
        maxshape: Option<Vec<MaxExtent>>,
        pipeline_message: Option<Vec<u8>>,
        payload: ChunkPayload,
        old_addr: u64,
    },
    /// A relocating **append** to a chunked, unlimited, Extensible-Array-indexed
    /// dataset (`append_dataset`). The dataset's existing chunk *data* stays in
    /// place; only the newly-appended chunks and any rewritten trailing partial
    /// chunk (`new_chunk_bytes`, already compressed through the on-disk pipeline)
    /// are placed (reusing freed raw space where it fits), a fresh Extensible
    /// Array is rebuilt over
    /// `kept_chunks ++ new_chunk_bytes`, the verbatim header `region`'s dataspace
    /// message is grown (`new_dataspace_body`) and its data-layout message
    /// repointed at the new index (every other message — datatype, filter
    /// pipeline, fill value, attributes — preserved verbatim), and the header is
    /// relocated. After the commit lands, only the old index structure at
    /// `old_addr`, the old header, and the relocated old trailing chunk
    /// (`old_tail_extent`) are freed — never the kept chunk data, which both the
    /// old and new index share during the commit.
    AppendedChunks {
        region: OhRegion,
        /// The grown dataspace message body (v2-serialized), current axis-0
        /// dimension increased, maximum dimensions (unlimited) preserved.
        new_dataspace_body: Vec<u8>,
        /// Rank-only spatial chunk dimensions, for the rebuilt v4 layout message.
        chunk_dims_u32: Vec<u32>,
        element_size: NonZeroUsize,
        /// Full (uncompressed) chunk byte size = product(spatial) * element_size.
        has_filters: bool,
        /// Existing complete chunks, in index order, carried by metadata alone —
        /// their base-relative addresses, on-disk stored sizes, and filter masks
        /// preserved exactly (a nonzero mask from a C/h5py-skipped filter is kept).
        kept_chunks: Vec<WrittenChunk>,
        /// The appended chunks in index order: the recompressed trailing partial
        /// chunk first (when present), then the remaining new full chunks.
        new_chunk_bytes: Vec<Vec<u8>>,
        /// The dataset header address, for old-index and old-header reclaim.
        old_addr: u64,
        /// The absolute `(addr, len)` of the old trailing partial chunk's data
        /// block when it was rewritten, freed after the commit lands. `None` when
        /// the append was chunk-aligned (no partial chunk to rewrite).
        old_tail_extent: Option<(u64, u64)>,
    },
    /// A dataset-attribute edit (`set_dataset_attr` / `remove_dataset_attr`).
    /// The verbatim header `region` already carries whatever the preflight could
    /// resolve; `attrs` is what [`WriteEngine::write_moving`] still has to place —
    /// a variable-length attribute's heap collection, or a whole dense set to
    /// rebuild. The rewritten header is relocated and the parent link repointed,
    /// exactly like the other relocating writes — but the data-layout message is
    /// preserved verbatim, so the dataset's chunk data and index stay in place;
    /// only the old header is freed.
    AttrEdit {
        region: OhRegion,
        attrs: EditedAttrs,
    },
}

/// The chunk data a relocating chunked overwrite ([`MovingWrite::Chunked`])
/// will place, either already encoded or still to be.
enum ChunkPayload {
    /// Split and encoded by the preflight, which had to do that anyway to find
    /// out whether the chunks still fit their slots.
    Encoded(Vec<Vec<u8>>),
    /// Element bytes that still carry unresolved variable-length references
    /// ([`OverwriteBytes::vlen`]), so the split has to wait for the apply
    /// phase: a filtered chunk's compressed length depends on the heap
    /// addresses patched into it, which is why the preflight cannot split
    /// first and patch after (issue #321).
    ///
    /// Such an overwrite therefore always relocates, which is what keeps the
    /// single-hard-link refusal (the one every [`MovingWrite`] answers to) in
    /// the preflight, where it belongs: whether it could have stayed in its
    /// slots is not knowable until those lengths exist.
    Deferred {
        bytes: OverwriteBytes,
        /// What the edge overhang of a partial chunk must hold (issue #296).
        padding: crate::fill_value::PaddingFill,
        /// The dataset's datatype, for the filter pipeline's [`ChunkContext`].
        dt: crate::datatype::Datatype,
    },
}

/// A staged dataset reduced to the pieces the writer needs.
struct FlatDataset {
    name: LinkNameBuf,
    dt: crate::datatype::Datatype,
    ds: Dataspace,
    raw: Vec<u8>,
    attrs: Vec<crate::attribute::AttributeMessage>,
    /// Chunked/filtered storage options. When [`ChunkOptions::is_chunked`] is
    /// false and `maxshape` is `None`, the dataset is written as contiguous,
    /// unfiltered storage; otherwise its chunk data and index are built by
    /// [`WriteEngine::build_chunked_dataset`].
    chunk_options: ChunkOptions,
    /// Maximum dimensions for an extensible dataset, mirrored into
    /// `ds.max_dimensions`. `None` for a fixed-shape dataset. A maxshape with an
    /// unlimited dimension selects the extensible-array chunk index, and a
    /// finite maxshape the fixed-array or single-chunk one.
    maxshape: Option<Vec<MaxExtent>>,
    /// Variable-length attributes still carrying a placeholder heap address:
    /// (index into `attrs`, that attribute's global heap collections).
    /// Resolved in the apply loop right before this dataset's header is built.
    vl_attrs: Vec<(usize, Vec<Vec<u8>>)>,
    /// Whether `attrs` goes in a fractal heap rather than the object header,
    /// decided by `file_writer::needs_dense_attrs` where the set was validated
    /// against what that heap can represent.
    ///
    /// Carried rather than asked again in the apply loop, so the set that was
    /// checked and the set that is placed cannot be decided differently — and
    /// because the question costs a serialization of every attribute to answer,
    /// which is most expensive for exactly the oversized attribute it selects
    /// for. Patching a variable-length attribute's references is
    /// length-preserving, so nothing between the two phases can change the
    /// answer.
    attrs_are_dense: bool,
    /// A staged variable-length-string dataset's element references (still
    /// carrying placeholder heap addresses in `raw`) and global heap
    /// collections. Resolved in the apply loop right before `raw` is appended.
    vl_string_staging: Option<VlStringStaging>,
    /// An object-reference dataset's per-element targets, still unresolved.
    /// Resolved (see [`WriteEngine::resolve_reference_target`]) and patched
    /// into `raw` in the apply loop, once every object this commit places has
    /// a known address. `None` for an ordinary dataset.
    reference_targets: Option<Vec<ObjectRefPatch>>,
    /// A user-defined fill value, encoded in the dataset's datatype, or `None`
    /// for the library default. Validated against the datatype element size in
    /// [`flatten_dataset`].
    fill: Option<Vec<u8>>,
    /// The provenance attributes' own inputs, kept so they can be rebuilt when
    /// `raw` changes before the commit writes it. `None` for a dataset staged
    /// without [`DatasetBuilder::with_provenance`](crate::DatasetBuilder::with_provenance).
    #[cfg(feature = "provenance")]
    provenance: Option<StagedProvenance>,
}

/// A staged dataset's provenance metadata, and where in
/// [`FlatDataset::attrs`] the attributes it produced sit.
///
/// The hash is over the dataset's raw bytes, so it is only correct as of the
/// bytes it was taken from; [`FlatDataset::rebuild_provenance`] is what keeps
/// it that way when [`WriteEngine::extend_staged_dataset`] grows them.
#[cfg(feature = "provenance")]
struct StagedProvenance {
    /// What [`crate::provenance::Provenance::build_attrs`] was, and is, called
    /// with. Held rather than re-derived from the attributes it produced, which
    /// a caller may also have set by hand.
    inputs: crate::provenance::Provenance,
    /// The index in `attrs` where its attributes begin. They are appended last
    /// and are the only ones this may replace, so everything below it — the
    /// caller's own attributes, and the `vl_attrs` indices into them — is left
    /// exactly as it was.
    attrs_start: usize,
}

#[cfg(feature = "provenance")]
impl FlatDataset {
    /// Recompute this dataset's provenance attributes over the bytes it now
    /// holds. A no-op for a dataset staged without provenance.
    ///
    /// The rebuilt set is the same attributes in the same order and of the same
    /// serialized length — the digest is 64 hex characters whatever it hashes —
    /// so `attrs_are_dense`, decided once in [`flatten_dataset`] over the
    /// original set, still describes this one.
    fn rebuild_provenance(&mut self) {
        let Some(prov) = &self.provenance else {
            return;
        };
        self.attrs.truncate(prov.attrs_start);
        let rebuilt = prov.inputs.build_attrs(&self.raw);
        self.attrs.extend(rebuilt);
    }
}

/// A borrow adapter that drives the shared Extensible-Array append engine
/// ([`crate::chunk_index_inplace`]) against the engine's *own* image and
/// superblock, so a session runs an immediate O(1) in-place append without
/// constructing a second writable handle (which would take a second exclusive
/// lock and keep a divergent view of the file). It borrows only those two
/// fields, leaving [`WriteEngine::located`] independently borrowable.
///
/// [`Store`] is the append engine's view of a file — an image *plus* the
/// superblock, which the image itself knows nothing about. Pairing them here is
/// all this adapter does; every primitive delegates, so the image's own
/// write-ordering discipline is what applies.
///
/// It carries the session's paged-file state too, so `alloc_raw` keeps a paged
/// file's pages homogeneous through exactly the rule the staged commit uses
/// ([`PagedEdit::begin`]). Before issue #198 there were two copies of that state —
/// one per engine — and the whole-file editor's copy was reachable only from the
/// commit path, so it had to refuse an in-place append to a paged file outright.
struct EditStore<'a> {
    image: &'a mut dyn FileImage,
    superblock: &'a mut Superblock,
    sb_sig_off: u64,
    /// The session's paged state when the file is paged, `None` otherwise. A
    /// borrow rather than a copy: padding recorded here has to reach the manager
    /// rewrite at the next commit or at close.
    paged: Option<&'a mut PagedEdit>,
    /// The list an immediate append may draw from, and `None` when it may draw
    /// from none — [`WriteEngine::immediate_reuse_allowed`] states which. It is
    /// the session's own free list on a file that forgets its holes at close, and
    /// [`WriteEngine::reserved`] — space already taken out of the on-disk
    /// managers — on one that persists them.
    free: Option<&'a mut FreeList>,
    /// The session's `fsync` cadence, carried by value: the append engine's own
    /// ordered barriers ([`apply_ea_append`]) are durability points like the
    /// commit's, and answer to the same policy.
    sync_policy: SyncPolicy,
}

impl EditStore<'_> {
    /// Append `bytes` into a raw page, padding the tail page first when a paged
    /// file's tail holds metadata. A plain append on the common non-paged file.
    ///
    /// Raw is the only page type this adapter allocates: see
    /// [`Store::alloc_raw`] for why an extensible-array index block belongs in a
    /// raw page here.
    fn append_into_raw_page(&mut self, bytes: &[u8]) -> Result<u64, Error> {
        if let Some(pg) = self.paged.as_deref_mut() {
            pg.begin(self.image, PageType::Raw)?;
        }
        self.image.append(bytes)
    }
}

impl crate::source::Source for EditStore<'_> {
    fn len(&self) -> u64 {
        self.image.len()
    }
    fn read_at(&self, offset: u64, buf: &mut [u8]) -> Result<(), crate::error::FormatError> {
        self.image.read_at(offset, buf)
    }
    fn read_metadata_at(
        &self,
        offset: u64,
        len: usize,
    ) -> Result<Vec<u8>, crate::error::FormatError> {
        self.image.read_metadata_at(offset, len)
    }

    // Forwarded because this is a `Source` wrapper over the image and the
    // trait requires it, not because anything asks yet: `File`'s own accessor
    // reaches the image directly. A wrapper that answered `None` for a live
    // cache is the failure that rule exists to prevent.
    fn metadata_cache_stats(&self) -> Option<crate::source::MetadataCacheStats> {
        self.image.metadata_cache_stats()
    }

    fn reset_metadata_cache_stats(&self) {
        self.image.reset_metadata_cache_stats();
    }
}

impl Store for EditStore<'_> {
    fn offset_size(&self) -> u8 {
        self.superblock.offset_size
    }
    fn length_size(&self) -> u8 {
        self.superblock.length_size
    }
    fn alloc_raw(&mut self, bytes: &[u8]) -> Result<u64, Error> {
        if let Some(free) = self.free.as_deref_mut() {
            if let Some(addr) = free.alloc(bytes.len() as u64) {
                self.image.write_at(addr, bytes)?;
                return Ok(addr);
            }
        }
        self.append_into_raw_page(bytes)
    }
    fn write_at(&mut self, offset: u64, bytes: &[u8]) -> Result<(), Error> {
        self.image.write_at(offset, bytes)
    }
    fn patch_superblock_eof(&mut self) -> Result<(), Error> {
        // Advance only the recorded end-of-file and re-serialize the superblock in
        // place. Unlike `WriteEngine::commit`, this deliberately does NOT clear the
        // consistency flags and does NOT repoint the root group: base_address is 0
        // for every in-place-append-eligible file, so the normalized-absolute root
        // address serializes back to the same stored value.
        let eof = self.image.len();
        self.superblock.eof_address = eof;
        let bytes = self.superblock.serialize();
        self.write_at(self.sb_sig_off, &bytes)
    }
    fn sync(&mut self) -> Result<(), Error> {
        barrier_data(self.image, self.sync_policy)
    }
}

/// Re-tags a rejection from the shared append engine (`AppendUnsupported`) as the fast-path
/// [`Error::AppendInPlaceUnsupported`], which a caller catches to fall back to the staged
/// [`stage_dataset_append`](WriteEngine::stage_dataset_append). That path covers the filtered
/// partial-trailing-chunk case and index-geometry limits. A platform-width limit applies on both
/// paths: the engine reports it as `AppendUnsupported`, and the staged path as
/// [`FormatError::ValueTooLargeForPlatform`]. I/O and format errors pass through unchanged.
pub(crate) fn as_inplace_error(e: Error) -> Error {
    match e {
        Error::AppendUnsupported(m) => Error::AppendInPlaceUnsupported(m),
        other => other,
    }
}

/// Validate a gathered append's bytes against a located dataset: the byte
/// length must be a whole number of elements, and the element datatype must
/// match the on-disk datatype (or, for a raw append, be raw-appendable).
/// Returns the appended element count (`0` = nothing to do). Shared by
/// [`WriteEngine::append_inplace_gathered`]'s path and the bounded backend's immediate
/// append so the acceptance rules stay identical.
pub(crate) fn validate_gathered_append(st: &LocatedState, b: &AppendBuilder) -> Result<u64, Error> {
    let raw = b.raw();
    if raw.len() % st.element_size != 0 {
        return Err(Error::AppendInPlaceUnsupported(
            "appended byte length is not a whole number of elements",
        ));
    }
    match b.elem_dt() {
        Some(expected) if *expected != st.datatype => {
            return Err(Error::AppendInPlaceUnsupported(
                "append datatype does not match the on-disk dataset (wrong element \
                 type or byte order)",
            ));
        }
        Some(_) => {}
        None => {
            if !datatype_is_raw_appendable(&st.datatype) {
                return Err(Error::AppendInPlaceUnsupported(
                    "append_raw onto this dataset's datatype (non-little-endian, \
                     variable-length, or reference) could misencode the bytes; use a \
                     typed append",
                ));
            }
        }
    }
    Ok((raw.len() / st.element_size) as u64)
}

/// Locate the dataset at `oh_addr` in `file` and build its [`LocatedState`],
/// validating in-place append eligibility (rank-1 / unlimited / Extensible-Array
/// indexed, a nonzero chunk length, and a re-encodable filter pipeline). A
/// dataset this path cannot append to takes an
/// [`Error::AppendInPlaceUnsupported`]. A datatype or filter pipeline message
/// that does not read or parse returns the [`Error::Format`] the read or the
/// parse produced.
pub(crate) fn locate_dataset_state<F: Store>(
    file: &F,
    oh_addr: u64,
) -> Result<LocatedState, Error> {
    let result = Located::locate_at(file, oh_addr, Error::AppendInPlaceUnsupported)?;
    if result.located.chunk_elems == 0 {
        return Err(Error::AppendInPlaceUnsupported(
            "in-place append requires a nonzero chunk length",
        ));
    }
    let (dt_off, dt_size) = result.spans.datatype;
    let dt_bytes = file.read_metadata_at(dt_off, dt_size)?;
    let (datatype, _) = Datatype::parse(&dt_bytes)?;
    let pipeline = match result.spans.filter {
        Some((fb, fsize)) => {
            let fp_bytes = file.read_metadata_at(fb, fsize)?;
            let parsed = FilterPipeline::parse(&fp_bytes)?;
            if !pipeline_reencodable(&parsed) {
                return Err(Error::AppendInPlaceUnsupported(
                    "dataset uses a filter this engine cannot re-encode",
                ));
            }
            Some(parsed)
        }
        None => None,
    };
    let element_size = result.located.elem_bytes;
    let spatial = vec![result.located.chunk_elems];
    // A fill message that cannot be read leaves `PaddingFill::Unknown`, which
    // fails only where a chunk actually needs padding — an append that lands on
    // a chunk boundary asks nothing of it.
    let fill = match result.spans.fill {
        Some((msg_type, off, size)) => match file.read_metadata_at(off, size) {
            Ok(body) => crate::fill_value::PaddingFill::from_message(msg_type, &body),
            Err(_) => crate::fill_value::PaddingFill::Unknown,
        },
        None => crate::fill_value::PaddingFill::Zero,
    };
    Ok(LocatedState {
        loc: result.located,
        datatype,
        spatial,
        element_size,
        pipeline,
        fill,
    })
}

/// Group `(parent group, item)` pairs by their parent, preserving the input
/// order within each group.
///
/// The one rule by which a commit's staged datasets become per-group batches:
/// the preflight groups borrowed ones to prove its guards, and the apply loop
/// groups the owned ones it places, so the order
/// [`preflight_reference_targets`](WriteEngine::preflight_reference_targets)
/// replays is the order the apply loop uses by construction.
fn group_by_parent<K: Ord, T>(items: impl IntoIterator<Item = (K, T)>) -> BTreeMap<K, Vec<T>> {
    let mut out: BTreeMap<K, Vec<T>> = BTreeMap::new();
    for (parent, item) in items {
        out.entry(parent).or_default().push(item);
    }
    out
}

/// Ensure a node exists for every ancestor prefix of `path` (so each is rebuilt
/// and can re-wire its child link). Does not set `is_new`.
fn ensure_ancestors(nodes: &mut BTreeMap<ObjectPathBuf, Node>, path: &ObjectPathBuf) {
    for len in 0..=path.len() {
        nodes.entry(path.prefix(len)).or_default();
    }
}

/// Validate that every reclaim span `(addr, len)` is non-empty, ends at or
/// before `eof`, and that no two overlap; sorts `spans` by address as a side
/// effect. Returns `false` on any violation so the caller can decline to
/// reclaim the object rather than feed the free list an out-of-bounds or
/// overlapping (double-free) region. Touching spans are allowed — the free list
/// coalesces them.
fn spans_disjoint_in_bounds(spans: &mut [(u64, u64)], eof: u64) -> bool {
    for &(addr, len) in spans.iter() {
        match addr.checked_add(len) {
            Some(end) if len > 0 && end <= eof => {}
            _ => return false,
        }
    }
    spans.sort_unstable_by_key(|&(addr, _)| addr);
    spans.windows(2).all(|w| w[0].0 + w[0].1 <= w[1].0)
}

/// Sanitize the accumulated free spans for a whole commit so the free list never
/// sees an out-of-bounds or overlapping (double-free) region: drop empty or
/// past-`eof` spans, sort by address, then drop any span overlapping one already
/// kept. Dropping only leaks (the bytes stay allocated); it never frees a live
/// region. With the last-hard-link guard in force nothing should be dropped for
/// a well-formed file — this is a backstop, not the primary defense.
fn retain_disjoint_in_bounds(spans: &mut Vec<(u64, u64, FreeClass)>, eof: u64) {
    spans.retain(|&(addr, len, _)| len > 0 && addr.checked_add(len).is_some_and(|e| e <= eof));
    spans.sort_unstable_by_key(|&(addr, _, _)| addr);
    let mut kept_end = 0u64;
    spans.retain(|&(addr, len, _)| {
        if addr >= kept_end {
            kept_end = addr + len;
            true
        } else {
            false // overlaps a span already kept; leak it rather than double-free
        }
    });
}

/// Whether some `data` span abuts the run of `index` blocks — ends exactly where
/// the index begins, or begins exactly where it ends. The geometric half of
/// [`WriteEngine::index_is_provably_raw`], where the reasoning lives; split out so
/// the rule can be stated against spans alone.
///
/// An empty index is vacuously placed: there is nothing to file.
fn index_abuts_chunk_data(data: &[(u64, u64)], index: &[(u64, u64)]) -> bool {
    if index.is_empty() {
        return true;
    }
    // The index as a whole: this crate writes its blocks as one run beside the
    // chunk data, so a data span abutting either end places all of them.
    let (Some(index_start), Some(index_end)) = (
        index.iter().map(|&(a, _)| a).min(),
        index.iter().filter_map(|&(a, l)| a.checked_add(l)).max(),
    ) else {
        return false;
    };
    // A zero-length span touches an address without occupying a byte beside it,
    // so it places nothing.
    data.iter().any(|&(addr, len)| {
        len > 0 && (addr.checked_add(len) == Some(index_start) || addr == index_end)
    })
}

/// Whether any byte of the `index` run lies in page 0 of a paged file, whose
/// first bytes are the superblock and which is therefore a metadata page. The
/// screening half of [`WriteEngine::index_is_provably_raw`], where the reasoning
/// lives.
///
/// A page size of zero is not a paged file's; it cannot be reasoned about, so it
/// screens everything out.
fn index_touches_page_zero(index: &[(u64, u64)], page_size: u64) -> bool {
    if page_size == 0 {
        return !index.is_empty();
    }
    index.iter().any(|&(addr, len)| len > 0 && addr < page_size)
}

/// Tag object-header chunk spans as file metadata. Every span
/// [`oh_chunk_spans`](WriteEngine::oh_chunk_spans) returns is part of an object
/// header, so the page type is the same for all of them.
fn meta_spans(spans: Vec<(u64, u64)>) -> impl Iterator<Item = (u64, u64, FreeClass)> {
    spans
        .into_iter()
        .map(|(a, l)| (a, l, FreeClass::Page(PageType::Meta)))
}

/// Validate a staged dataset and reduce it to a [`FlatDataset`]. `name` is the link name the
/// dataset has in its parent group, taken off the staged path by the caller. Contiguous,
/// unfiltered datasets are emitted as such; chunked, filtered, or extensible
/// datasets carry their [`ChunkOptions`] and maxshape through to the commit,
/// where [`WriteEngine::build_chunked_dataset`] lays out their chunk data and
/// index. An empty (zero-element) shape is allowed under either storage,
/// mirroring the whole-file writer: a contiguous one takes the `HADDR_UNDEF`
/// data address (see the apply loop) and a chunked one an index over zero
/// chunks, which is what an extensible dataset is created as before the first
/// append fills it. The geometry validation below still requires explicit chunk
/// dimensions for it — auto-chunking has no shape to derive them from. A
/// `provenance` dataset has its SHA-256/creator/timestamp/source attributes
/// computed here from `raw`, exactly as the whole-file writer does. A
/// variable-length attribute's global heap collection is built here (it is
/// fully self-contained — no address of its own) but placed and patched later,
/// in the apply loop, once its final address is known; likewise a
/// variable-length-string dataset's staged references and collection
/// (`db.vl_string_staging`) are carried through unresolved. An object-reference
/// dataset's per-element targets (`db.reference_targets`) are likewise carried
/// through unresolved — resolving a path target requires knowing every other
/// object this commit places, which is only known well into the apply loop
/// (see [`WriteEngine::resolve_reference_target`]). Rejects any remaining
/// feature this engine cannot reproduce faithfully: a
/// chunked/extensible variable-length-string or object-reference dataset, or a
/// filter pipeline the build cannot construct.
fn flatten_dataset(db: DatasetBuilder, name: LinkNameBuf) -> Result<FlatDataset, Error> {
    let dt = db
        .datatype
        .ok_or(Error::EditUnsupported("dataset has no datatype/data"))?;
    let shape = db
        .shape
        .ok_or(Error::EditUnsupported("dataset has no shape"))?;
    let is_empty = shape.contains(&0);
    let chunked = db.chunk_options.is_chunked() || db.maxshape.is_some();
    // Storage this engine allocates for every dataset it places: it appends into
    // an existing layout, and the undefined data address the whole-file writer
    // gives an unallocated dataset (issue #293) has no equivalent here. Refused
    // by name rather than left to the "dataset has no data" arm below, which is
    // the same symptom from the opposite cause, and refused rather than ignored,
    // since ignoring it would write out the grid of fill values the caller asked
    // not to have.
    if db.allocation == StorageAllocation::Unallocated {
        return Err(Error::EditUnsupported(
            "a dataset with unallocated storage cannot be added to an existing file in place",
        ));
    }
    // Variable-length string element references live in the global heap, whose
    // address is only known once the apply loop places the collection. For
    // chunked/filtered/resizable storage the references sit inside chunks
    // written before that address exists, so patching them in is impossible.
    //
    // The whole-file writer lifted the same restriction by placing such a
    // dataset's collections ahead of everything else (issue #109); this engine
    // appends into an existing layout, where there is no "ahead" to place them
    // in, so the equivalent fix is a separate piece of work.
    if db.vl_string_staging.is_some() && chunked {
        return Err(Error::EditUnsupported(
            "chunked or extensible variable-length-string datasets cannot be added in place yet",
        ));
    }
    // Object-reference elements are resolved (see `resolve_reference_target`)
    // and patched into `raw` right before it is appended; for chunked storage
    // that patch would need to reach inside already-built chunk data, which
    // this engine does not support (mirrors the variable-length-string
    // refusal above — untested and unneeded combination for v1).
    if db.reference_targets.is_some() && chunked {
        return Err(Error::EditUnsupported(
            "chunked or extensible object-reference datasets cannot be added in place yet",
        ));
    }
    let raw = if is_empty {
        db.data.unwrap_or_default()
    } else {
        db.data
            .ok_or(Error::EditUnsupported("dataset has no data"))?
    };

    // Refused for the same reason the whole-file writer refuses it: nothing
    // occupies zero bytes per element, the writers divide by the element size,
    // and a caller-built `Datatype` never passes through `Datatype::parse`.
    // Taking it as a `NonZeroUsize` hands the proof to the staging below rather
    // than leaving each step to re-derive it.
    let elem_size = dt.element_size_usize()?;

    let elem = elem_size.get() as u64;
    // Multiply with checked arithmetic: an absurd shape whose element count
    // (or byte size) overflows `u64` is refused rather than panicking in a
    // debug build or silently wrapping in release (which could let a wrapped
    // product spuriously match `raw.len()`). For a zero-element shape this
    // expected length is always 0 (a `0` dimension makes every checked
    // multiplication `Some(0)` regardless of the other dimensions), so this
    // also catches data mistakenly supplied for a shape that holds nothing.
    let expected = shape
        .iter()
        .try_fold(1u64, |acc, &d| acc.checked_mul(d))
        .and_then(|n| n.checked_mul(elem));
    match expected {
        Some(expected) if raw.len() as u64 == expected => {}
        Some(_) => {
            return Err(Error::EditUnsupported(
                "dataset data length does not match its shape",
            ));
        }
        None => {
            return Err(Error::EditUnsupported(
                "dataset shape is too large to address on this platform",
            ));
        }
    }

    if chunked {
        // Refuse malformed chunk geometry up front (the same validation the
        // whole-file writer applies), so a bad request — chunk dimensions of the
        // wrong rank, a zero chunk dimension, an inconsistent maximum shape, or
        // chunking a scalar — never reaches and panics the chunk splitter, nor
        // yields a dataset the reader cannot decode.
        Extent::new(&shape, db.maxshape.as_deref())
            .and_then(|extent| db.chunk_options.validate_geometry(extent))
            .map_err(Error::EditUnsupported)?;
        // A filter this build cannot apply is refused up front rather than
        // failing mid-apply when a chunk is compressed.
        db.chunk_options
            .refuse_unavailable_filters()
            .map_err(Error::EditUnsupported)?;
        // Validate the requested filter pipeline now — before any file bytes are
        // written — so an unsupported filter, an incompatible datatype, or a
        // fill value the filter cannot record is refused up front; the chunk
        // data itself is laid out in the commit's apply phase. A filter the
        // build compiled out is not among them: `build_pipeline` emits its
        // descriptor regardless, which is why the check above exists. Chunked/filtered
        // storage flows through the very builder the normal writer uses
        // ([`compress_chunks`] + [`assemble_chunked_at`] + [`build_chunked_dataset_oh`]),
        // so the
        // resulting object header is byte-identical to a freshly written one.
        //
        // The fill value goes in for the same reason the apply phase passes it:
        // scale-offset records it in the filter's parameters, so a validation
        // that left it out would be checking a pipeline the commit does not
        // build — and would pass a fill value the encoder later refuses.
        let chunk_dims = db.chunk_options.resolve_chunk_dims(&shape);
        let ctx = ChunkContext::from_datatype(&chunk_dims, &dt)?;
        db.chunk_options.build_pipeline(
            &ctx,
            crate::fill_value::FillPattern::new(
                db.fill.as_deref(),
                ctx.element_size.narrow::<NonZeroUsize>()?,
            ),
        )?;
    }

    // The link message body (whose length is independent of the address) must
    // fit the object-header message's u16 size field; a pathologically long
    // name would otherwise overflow it into silent corruption. Measured with a
    // creation index present — the widest form, written into a group that tracks
    // link creation order — since the parent group is not known here.
    let mut sized = make_link(name.as_str(), StoredAddress::new(0));
    sized.creation_order = Some(0);
    if sized.serialize(OFFSET_SIZE).len() > OBJECT_HEADER_MESSAGE_MAX {
        return Err(Error::EditUnsupported(
            "dataset name is too long to encode as a link message",
        ));
    }

    let ds = Dataspace {
        space_type: if shape.is_empty() {
            DataspaceType::Scalar
        } else {
            DataspaceType::Simple
        },
        #[expect(
            clippy::cast_possible_truncation,
            reason = "dataspace rank fits the 1-byte dimensionality field (HDF5 caps rank at 32)"
        )]
        rank: shape.len() as u8,
        dimensions: shape,
        // A chunked, extensible dataset records its maximum dimensions. A
        // fixed-shape dataset has none.
        max_dimensions: db.maxshape.clone(),
    };
    let mut attrs: Vec<crate::attribute::AttributeMessage> = Vec::with_capacity(db.attrs.len());
    for (n, v) in &db.attrs {
        attrs.push(v.to_message(n));
    }
    // The message above already carries a placeholder (heap address 0) for each
    // element of a variable-length string attribute; stage its self-contained
    // global heap collections here (no address of their own to resolve yet) and
    // record which `attrs` slot they patch once the apply loop places them.
    let mut vl_attrs: Vec<(usize, Vec<Vec<u8>>)> = Vec::new();
    for (i, (_, v)) in db.attrs.iter().enumerate() {
        if let Some(strings) = v.var_len_strings() {
            vl_attrs.push((i, build_global_heap_collections(strings)));
        }
    }
    // Appended last, and recorded as such: `raw` can still grow before the
    // commit writes it (`WriteEngine::extend_staged_dataset`), and the digest
    // has to follow it.
    #[cfg(feature = "provenance")]
    let provenance = db.provenance.as_ref().map(|prov| {
        let inputs = crate::provenance::Provenance {
            creator: prov.creator.clone(),
            timestamp: prov.timestamp.clone(),
            source: prov.source.clone(),
        };
        let attrs_start = attrs.len();
        attrs.extend(inputs.build_attrs(&raw));
        StagedProvenance {
            inputs,
            attrs_start,
        }
    });
    // More attributes than an object header keeps compactly, or one whose message
    // overflows its 2-byte message-size field, sends the set to a fractal heap —
    // the same disjunction, and the same heap, the whole-file writer uses. What
    // that heap cannot represent is refused here, in the preflight, so a staged
    // dataset that cannot be written is refused before the commit places a byte.
    let attrs_are_dense = crate::file_writer::needs_dense_attrs(&attrs);
    if attrs_are_dense {
        crate::file_writer::dense_attrs_check(&attrs).map_err(Error::Format)?;
    }

    // A committed datatype is an object of its own, which an in-place edit has no
    // way to place: it appends into an existing file rather than laying one out,
    // so there is nothing to resolve the named path against. Writing the type
    // inline instead would produce a dataset that reads correctly but no longer
    // shares the named type, so refuse by name. The whole-file writer places
    // them; [`crate::repack`] is the route from an edited file to one.
    if db.datatype_location.is_committed()
        || attrs.iter().any(|a| a.datatype_location.is_committed())
    {
        return Err(Error::EditUnsupported(
            "a dataset or attribute naming a committed (shared) datatype cannot be added in \
             place; write the file with FileBuilder instead",
        ));
    }

    // A user-defined fill value is one element wide, so its byte length must
    // equal the datatype's element size (mirrors the whole-file writer's check).
    if let Some(fill) = &db.fill {
        let expected = elem.to_usize()?;
        if fill.len() != expected {
            return Err(Error::Format(FormatError::FillValueSizeMismatch {
                expected,
                actual: fill.len(),
            }));
        }
    }

    Ok(FlatDataset {
        name,
        dt,
        ds,
        raw,
        attrs,
        chunk_options: db.chunk_options,
        maxshape: db.maxshape,
        vl_attrs,
        attrs_are_dense,
        vl_string_staging: db.vl_string_staging,
        reference_targets: db.reference_targets,
        fill: db.fill,
        #[cfg(feature = "provenance")]
        provenance,
    })
}

/// A minimal Group Info message body (type 0x000A): version 0 with neither the
/// link-phase-change nor the estimated-entry fields stored. With both absent the
/// HDF5 C library fills `max_compact`/`min_dense` from its own defaults (8 and
/// 6). See [`ensure_group_info`] for why every group needs this message.
const GROUP_INFO_BODY: [u8; 2] = [0, 0];

/// Whether every filter in `pipeline` is one this crate can *apply* (re-encode a
/// chunk through) — not merely decode. A pipeline with any other filter cannot be
/// re-encoded for an in-place overwrite, so the caller refuses with a typed error
/// rather than letting `compress_chunk` surface a raw `UnsupportedFilter`.
pub(crate) fn pipeline_reencodable(pipeline: &FilterPipeline) -> bool {
    pipeline.filters.iter().all(|f| match f.filter_id {
        FILTER_DEFLATE | FILTER_SHUFFLE | FILTER_FLETCHER32 | FILTER_SCALEOFFSET | FILTER_LZF => {
            true
        }
        #[cfg(feature = "zfp")]
        crate::filter_pipeline::FILTER_ZFP => true,
        _ => false,
    })
}

/// Whether re-encoding a chunk through `pipeline` reproduces the values it was
/// decoded from — the condition for rewriting a chunk that is **already
/// committed**, as growing a partial trailing chunk does.
///
/// Stricter than [`pipeline_reencodable`], and for a different question. That
/// one asks whether this crate can *apply* the filters at all, which is what a
/// brand-new chunk needs. This one asks whether decode-then-encode is the
/// identity, which is what a chunk somebody has already read needs. Deflate,
/// shuffle, fletcher32 and LZF are lossless by construction; scale-offset only
/// in its integer mode; ZFP fixed-rate quantizes every block to a bit budget, so
/// re-encoding it against a *different* set of neighbours in the same block
/// re-quantizes the values that were already there.
///
/// This is the line [`repack`](crate::repack())'s `check_pipeline` already draws
/// for its two re-encoding paths, stated once more here because the append
/// engine reaches it by a different route.
pub(crate) fn pipeline_lossless(pipeline: &FilterPipeline) -> bool {
    pipeline.filters.iter().all(|f| match f.filter_id {
        FILTER_DEFLATE | FILTER_SHUFFLE | FILTER_FLETCHER32 | FILTER_LZF => true,
        // The integer mode subtracts a per-chunk minimum and packs the residuals
        // whole; float D-scale rounds to a decimal count and is documented lossy.
        // Anything else (float E-scale) this crate neither writes nor decodes.
        FILTER_SCALEOFFSET => matches!(
            crate::scaleoffset::scale_offset_mode(&f.client_data),
            Some((crate::scaleoffset::ScaleOffset::Integer(_), _))
        ),
        // Unknown ids included: a filter whose semantics are unknown is not one
        // to assume round-trips.
        _ => false,
    })
}

/// The refusal both append paths raise for a lossy pipeline sitting on a partial
/// trailing chunk. Named here beside the predicate it goes with, and used from
/// [`chunk_index_inplace`](crate::chunk_index_inplace) as well as the staged
/// rebuild above.
pub(crate) const LOSSY_TAIL_REFUSAL: &str = "this dataset's filter pipeline is lossy (ZFP, or float D-scale scale-offset), and its \
     length is not a whole multiple of the chunk length: growing that trailing chunk would \
     decode and re-encode values that are already committed, changing them. Append whole \
     chunks from a chunk-aligned length instead";

/// Rebuild a header message `region`, replacing the single Data Layout message's
/// record with one carrying `new_layout_body` and leaving every other message
/// (datatype, dataspace, fill value, filter pipeline, attributes, attribute info)
/// byte-for-byte. The replacement may differ in length from the original — a
/// chunked rebuild can change the index type and thus the layout message size — so
/// the record is rebuilt via [`OhRecordLayout::record`]. The chunked overwrite
/// and copy paths use this to relocate a dataset's chunk storage while preserving
/// the rest of its header exactly.
fn replace_layout_message(region: &OhRegion, new_layout_body: &[u8]) -> Result<OhRegion, Error> {
    let mut out = Vec::with_capacity(region.len());
    let mut p = 0;
    let mut replaced = false;
    while let Some((msg_type, _body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::DATA_LAYOUT && !replaced {
            out.extend_from_slice(
                &region
                    .layout()
                    .record(MessageType::DATA_LAYOUT, new_layout_body),
            );
            replaced = true;
        } else {
            out.extend_from_slice(&region[p..body_end]);
        }
        p = body_end;
    }
    if !replaced {
        return Err(Error::EditUnsupported(
            "chunked dataset header has no data-layout message to relocate",
        ));
    }
    Ok(region.with_bytes(out))
}

/// Rebuild a header message `region`, replacing the single Dataspace message's
/// record with one carrying `new_dataspace_body` (the grown current dimensions,
/// v2-serialized, maximum dimensions preserved) and leaving every other message
/// byte-for-byte. Used by the append path to grow a dataset's axis-0 dimension.
/// The replacement may differ in length from the original (a v1 on-disk
/// dataspace is normalized to v2 in the rebuilt header), so the record is rebuilt
/// via [`OhRecordLayout::record`].
fn replace_dataspace_message(
    region: &OhRegion,
    new_dataspace_body: &[u8],
) -> Result<OhRegion, Error> {
    let mut out = Vec::with_capacity(region.len());
    let mut p = 0;
    let mut replaced = false;
    while let Some((msg_type, _body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::DATASPACE && !replaced {
            out.extend_from_slice(
                &region
                    .layout()
                    .record(MessageType::DATASPACE, new_dataspace_body),
            );
            replaced = true;
        } else {
            out.extend_from_slice(&region[p..body_end]);
        }
        p = body_end;
    }
    if !replaced {
        return Err(Error::AppendUnsupported(
            "dataset header has no dataspace message to grow",
        ));
    }
    Ok(region.with_bytes(out))
}

/// Whether a datatype's raw on-disk bytes can be appended verbatim from a caller
/// via [`AppendBuilder::append_raw`]. True only when every scalar leaf is safe to
/// write as flat little-endian bytes:
///
/// - numeric leaves (fixed-point, floating-point, time, bit field) must be
///   little-endian, or the caller's little-endian bytes would silently misencode
///   into a big-endian (or VAX) field;
/// - string and opaque leaves are byte arrays with no numeric byte order, so they
///   are order-agnostic and safe;
/// - aggregates (enumeration, array, compound) are appendable iff every leaf is;
/// - variable-length and reference leaves embed global-heap or object addresses
///   that a flat byte append cannot reproduce, so they are never raw-appendable.
///
/// A typed `append_*` bypasses this: it checks full datatype equality instead, so
/// it already refuses every non-little-endian and non-scalar dataset.
pub(crate) fn datatype_is_raw_appendable(dt: &Datatype) -> bool {
    match dt {
        Datatype::FixedPoint { byte_order, .. }
        | Datatype::FloatingPoint { byte_order, .. }
        | Datatype::Time { byte_order, .. }
        | Datatype::BitField { byte_order, .. } => *byte_order == DatatypeByteOrder::LittleEndian,
        Datatype::String { .. } | Datatype::Opaque { .. } => true,
        Datatype::Enumeration { base_type, .. } | Datatype::Array { base_type, .. } => {
            datatype_is_raw_appendable(base_type)
        }
        Datatype::Compound { members, .. } => members
            .iter()
            .all(|m| datatype_is_raw_appendable(&m.datatype)),
        Datatype::VariableLength { .. } | Datatype::Reference { .. } => false,
    }
}

/// The datatype, dataspace, parsed chunked data layout, and verbatim filter-
/// pipeline message bytes (if any) of a chunked dataset header, parsed by
/// [`parse_chunked_header`].
struct ChunkedHeaderParts {
    dt: crate::datatype::Datatype,
    ds: Dataspace,
    layout: DataLayout,
    pipeline_message: Option<Vec<u8>>,
}

/// Parse the datatype, dataspace, chunked data layout, and verbatim filter-
/// pipeline message bytes (if any) from a chunked dataset header `region`. Used by
/// the chunked copy path to derive chunk geometry and the on-disk filter pipeline.
/// Errors if any required message is missing or the layout is not chunked.
fn parse_chunked_header(region: &OhRegion) -> Result<ChunkedHeaderParts, Error> {
    let mut datatype: Option<(usize, usize)> = None;
    let mut dataspace: Option<(usize, usize)> = None;
    let mut layout: Option<(usize, usize)> = None;
    let mut pipeline: Option<(usize, usize)> = None;
    let mut p = 0;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        match msg_type {
            MessageType::DATATYPE => datatype = Some((body, body_end)),
            MessageType::DATASPACE => dataspace = Some((body, body_end)),
            MessageType::DATA_LAYOUT => layout = Some((body, body_end)),
            MessageType::FILTER_PIPELINE => pipeline = Some((body, body_end)),
            _ => {}
        }
        p = body_end;
    }
    let (dt_b, dt_e) = datatype.ok_or(Error::EditUnsupported("dataset header has no datatype"))?;
    let (ds_b, ds_e) =
        dataspace.ok_or(Error::EditUnsupported("dataset header has no dataspace"))?;
    let (lb, le) = layout.ok_or(Error::EditUnsupported("dataset header has no data layout"))?;
    let (dt, _) = Datatype::parse(&region[dt_b..dt_e])?;
    let ds = Dataspace::parse(&region[ds_b..ds_e], LENGTH_SIZE)?;
    let dl = DataLayout::parse(&region[lb..le], OFFSET_SIZE, LENGTH_SIZE)?;
    if !matches!(dl, DataLayout::Chunked { .. }) {
        return Err(Error::EditUnsupported("dataset is not chunked"));
    }
    let pipeline_message = pipeline.map(|(b, e)| region[b..e].to_vec());
    Ok(ChunkedHeaderParts {
        dt,
        ds,
        layout: dl,
        pipeline_message,
    })
}

/// The chunk geometry a verbatim chunked rebuild needs, derived by
/// [`chunked_geometry`] from a chunked dataset's datatype, dataspace, and parsed
/// [`DataLayout::Chunked`].
struct ChunkedGeometry {
    /// Rank-only spatial chunk dimensions.
    spatial: Vec<u64>,
    /// Element size in bytes, proven non-zero: the chunk splitter divides by
    /// it, and so does the append path's element-count arithmetic.
    element_size: NonZeroUsize,
    /// Full (uncompressed) chunk byte size, `product(spatial) * element_size`.
    /// This is what the chunk index's element width is derived from, so it has
    /// to be the geometry's product and not any written chunk's size.
    raw_size: u64,
    /// The on-disk maximum dimensions, when they differ from the current shape.
    /// An unlimited dimension selects the extensible-array index and a finite
    /// one the fixed-array index. `None` keeps the fixed-array or single-chunk
    /// index.
    maxshape: Option<Vec<MaxExtent>>,
}

/// Derive the [`ChunkedGeometry`] for a chunked dataset from its datatype,
/// dataspace, and parsed [`DataLayout::Chunked`].
fn chunked_geometry(
    dt: &crate::datatype::Datatype,
    ds: &Dataspace,
    layout: &DataLayout,
) -> Result<ChunkedGeometry, Error> {
    let DataLayout::Chunked {
        chunk_dimensions, ..
    } = layout
    else {
        return Err(Error::EditUnsupported("dataset is not chunked"));
    };
    let rank = ds.dimensions.len();
    if chunk_dimensions.len() <= rank {
        return Err(Error::EditUnsupported(
            "chunked layout has malformed dimensions",
        ));
    }
    let spatial: Vec<u64> = chunk_dimensions[..rank].to_vec();
    let element_size = dt.element_size_usize()?;
    let raw_size = spatial
        .iter()
        .copied()
        .product::<u64>()
        .saturating_mul(element_size.get() as u64);
    let maxshape = ds.extensible_max_dimensions().map(<[MaxExtent]>::to_vec);
    Ok(ChunkedGeometry {
        spatial,
        element_size,
        raw_size,
        maxshape,
    })
}

/// The element bytes a staged value overwrite will write, paired with the
/// variable-length staging that has still to be resolved into them and the
/// `path` they belong to.
///
/// Cloned rather than moved because the staged set must survive a refused
/// commit whole (issue #316), and this runs in the preflight that may refuse.
fn staged_bytes(fd: &FlatDataset, path: &ObjectPathBuf) -> OverwriteBytes {
    OverwriteBytes {
        raw: fd.raw.clone(),
        vlen: fd.vl_string_staging.clone().map(|staging| VlenOverwrite {
            staging,
            path: path.clone(),
        }),
    }
}

/// Split `raw` into full-size chunk buffers in dense row-major grid order and
/// re-encode each through `pipeline_message`, the dataset's on-disk filter
/// pipeline.
///
/// The overhang past the dataset's edge holds the dataset's own fill value, not
/// zeros: an allocated chunk is expected to carry it wherever nothing was
/// written, and those slots are what a reader returns once the dataset is
/// extended into them (issue #296).
///
/// The caller has already refused a pipeline
/// [`pipeline_reencodable`] rejects, so the only errors here are a malformed
/// pipeline message (unreachable for the same reason) and the encoder's own.
fn split_and_encode_chunks(
    raw: &[u8],
    shape: &[u64],
    chunk_dims: &[u64],
    element_size: NonZeroUsize,
    padding: &crate::fill_value::PaddingFill,
    pipeline_message: Option<&[u8]>,
    dt: &crate::datatype::Datatype,
) -> Result<Vec<Vec<u8>>, Error> {
    let split = split_into_chunks(
        raw,
        shape,
        chunk_dims,
        element_size,
        padding.pattern(element_size),
    )
    .map_err(Error::Format)?;
    let Some(pm) = pipeline_message else {
        return Ok(split);
    };
    let pipeline = FilterPipeline::parse(pm)?;
    let ctx = ChunkContext::from_datatype(chunk_dims, dt)?;
    let mut encoded = Vec::with_capacity(split.len());
    // One encoder across the rewrite; see `FilterScratch`.
    let mut scratch = FilterScratch::new();
    for buf in &split {
        encoded.push(compress_chunk_with(&mut scratch, buf, &pipeline, ctx)?);
    }
    Ok(encoded)
}

/// Tries to overwrite a chunked dataset's chunks in place. When the dataset's
/// on-disk chunks form a dense grid aligned with `new_bytes` (dense row-major
/// order), every slot is unmasked (`filter_mask == 0`), and every new chunk
/// **fits** the slot it replaces (`new_len <= slot`), returns the in-place
/// `(address, bytes)` writes:
///
/// - When every new chunk is **exactly** its slot's size, only the chunk data is
///   written; the index is untouched (so any enumerable index type works, and a
///   crash can tear at most a chunk's value bytes, not the structure).
/// - When some new chunks are **smaller** (fit with slack), the chunk index
///   records each chunk's stored size, so the index is rebuilt in place to record
///   the new sizes (see [`try_rebuild_index_in_place`]). This is supported only
///   for a v4 fixed-array or extensible-array index occupying a single contiguous
///   on-disk region; any other case returns `None` to relocate.
///
/// Returns `None` — so the caller relocates the dataset instead — when the index
/// cannot be enumerated, the grid is sparse, a slot is masked, a new chunk does
/// not fit, the index cannot be rebuilt in place, or any write would be out of
/// bounds or overlap another.
fn try_inplace_chunk_writes<S: Source + ?Sized>(
    src: &S,
    layout: &DataLayout,
    ds: &Dataspace,
    spatial: &[u64],
    raw_size: u64,
    new_bytes: &[Vec<u8>],
) -> Option<Vec<(u64, Vec<u8>)>> {
    let infos = enumerate_chunks_from_source(src, layout, ds, OFFSET_SIZE, LENGTH_SIZE).ok()?;
    let grid = plan_dense_grid(infos, &ds.dimensions, spatial)?;
    if grid.grid_order.len() != new_bytes.len() {
        return None;
    }
    let mut writes = Vec::with_capacity(new_bytes.len() + 1);
    let mut spans: Vec<(u64, u64)> = Vec::with_capacity(new_bytes.len() + 1);
    let mut any_shrunk = false;
    for (ci, bytes) in grid.grid_order.iter().zip(new_bytes.iter()) {
        // A nonzero filter mask means the source left some filter unapplied for
        // this chunk; re-encoding always applies every filter (mask 0), so an
        // in-place overwrite would desync the index-recorded mask. Relocate.
        if ci.filter_mask != 0 {
            return None;
        }
        let new_len = bytes.len() as u64;
        let slot = ci.chunk_size.get();
        // A chunk that no longer fits its slot must relocate.
        if new_len > slot {
            return None;
        }
        if new_len < slot {
            any_shrunk = true;
        }
        ci.address
            .get()
            .checked_add(new_len)
            .filter(|&e| e <= src.len())?;
        writes.push((ci.address.get(), bytes.clone()));
        spans.push((ci.address.get(), new_len));
    }

    // A shrinking overwrite changes the index-recorded chunk sizes, so the index
    // must be rebuilt in place to match; an equal-size one leaves it untouched.
    if any_shrunk {
        let (index_addr, index_bytes) = try_rebuild_index_in_place(
            src,
            layout,
            ds,
            spatial,
            raw_size,
            &grid.grid_order,
            new_bytes,
        )?;
        spans.push((index_addr, index_bytes.len() as u64));
        writes.push((index_addr, index_bytes));
    }

    // Refuse to perform overlapping in-place writes (a malformed source index, or
    // an index region that overlaps a chunk slot); relocate instead so two writes
    // never clobber each other.
    if !spans_disjoint_in_bounds(&mut spans, src.len()) {
        return None;
    }
    Some(writes)
}

/// Rebuilds a chunked dataset's index **in place** so it records the new
/// (smaller) per-chunk stored sizes after a fits-with-slack overwrite, returning
/// the `(address, bytes)` write that replaces it. The chunks keep their existing
/// addresses (only their stored bytes shrank), so the rebuilt index points at the
/// same slots with the new sizes.
///
/// Supported only for a v4 **fixed-array** or **extensible-array** index whose
/// on-disk structure is a single contiguous region starting at the index address
/// — the layout this crate's own writer produces. The element width derives from
/// the unchanged raw chunk size, so the rebuilt structure is byte-for-byte the
/// same length as the original; this is required to match exactly, which rejects a
/// scattered or differently-laid-out (e.g. C-written) index, leaving the caller
/// to relocate. Single-chunk (size in the layout message) and B-tree-v1 (no
/// writer) indexes are not rebuilt here.
///
/// Like any in-place value overwrite (the HDF5 `H5Dwrite` model) this is not
/// atomic: a crash mid-write can tear the index and leave the dataset needing a
/// rewrite. It is used only on the in-place path, whose linearization point is the
/// synced data write.
#[allow(clippy::too_many_arguments)]
fn try_rebuild_index_in_place<S: Source + ?Sized>(
    src: &S,
    layout: &DataLayout,
    ds: &Dataspace,
    spatial: &[u64],
    raw_size: u64,
    grid_order: &[crate::chunked_read::ChunkInfo],
    new_bytes: &[Vec<u8>],
) -> Option<(u64, Vec<u8>)> {
    let DataLayout::Chunked { index, .. } = layout else {
        return None;
    };
    let index_addr = index.address()?;
    let written: Vec<crate::chunked_write::WrittenChunk> = grid_order
        .iter()
        .zip(new_bytes)
        .map(|(ci, b)| crate::chunked_write::WrittenChunk {
            address: ci.address,
            compressed_size: b.len() as u64,
            filter_mask: 0,
        })
        .collect();
    // `grid_order` is dense row-major over the *current* shape; where each of
    // those chunks sits in the index is the maximum grid's business, and the
    // rebuild has to reproduce the numbering the original index was written
    // with. Same rule, same function as the writer.
    let (_, slot_of_chunk, index_slots) = crate::chunked_write::plan_index_slots(
        &ds.dimensions,
        spatial,
        ds.max_dimensions.as_deref(),
        raw_size,
        true,
        // Rebuilding the index of a dataset whose chunks are in hand.
        crate::chunked_write::StorageAllocation::Allocated,
    )
    .ok()?;
    let slots =
        crate::chunked_write::IndexSlots::new(&written, &slot_of_chunk, index_slots).ok()?;
    let new_index = match index {
        // `raw_size` is the whole-chunk byte size, which is what the element
        // width derives from — the same value the original index was built with,
        // so the rebuilt structure matches its length. (An index written by a
        // version that derived the width from the written chunks instead can
        // disagree; the length check below then rejects it and the caller
        // relocates, which is the safe direction.)
        ChunkIndexLayout::FixedArray { .. } => crate::chunked_write::build_fixed_array_at(
            &slots,
            raw_size,
            OFFSET_SIZE,
            LENGTH_SIZE,
            true,
            index_addr,
        ),
        ChunkIndexLayout::ExtensibleArray { .. } => {
            crate::chunked_write::build_extensible_array_at(
                &slots,
                raw_size,
                OFFSET_SIZE,
                LENGTH_SIZE,
                true,
                index_addr,
            )
            .ok()?
        }
        // A single chunk's stored size sits in the layout message, so rebuilding
        // it would rewrite the header, an implicit index has no structure to
        // rebuild, and neither B-tree has a writer. All four relocate instead.
        ChunkIndexLayout::SingleChunk { .. }
        | ChunkIndexLayout::Implicit { .. }
        | ChunkIndexLayout::BTreeV1 { .. }
        | ChunkIndexLayout::BTreeV2 { .. } => return None,
    };

    // The on-disk index must be a single contiguous region starting at the index
    // address, and the rebuilt structure must be exactly the same length (true for
    // an index this crate wrote). A scattered or different on-disk layout fails
    // the check and the caller relocates.
    let mut spans =
        crate::chunked_read::chunk_index_spans_from_source(src, layout, OFFSET_SIZE, LENGTH_SIZE)
            .ok()?;
    if spans.is_empty() {
        return None;
    }
    spans.sort_unstable_by_key(|&(a, _)| a);
    if spans[0].0 != index_addr.get() {
        return None;
    }
    let mut end = index_addr.get();
    for &(a, l) in &spans {
        if a != end {
            return None; // a gap means the index is not contiguous
        }
        end = a.checked_add(l)?;
    }
    if new_index.len() as u64 != end - index_addr.get() {
        return None;
    }
    index_addr
        .get()
        .checked_add(new_index.len() as u64)
        .filter(|&e| e <= src.len())?;
    Some((index_addr.get(), new_index))
}

/// A [`ChunkProvider`] over chunk bytes already held in memory, in dense
/// row-major grid order. Used by the editor's chunked copy and relocating
/// overwrite, which own each chunk's bytes (a [`CopyTree`] or [`MovingWrite`]
/// captured them) rather than streaming from a source file like repack.
struct SliceChunkProvider<'a> {
    chunks: &'a [Vec<u8>],
}

impl ChunkProvider for SliceChunkProvider<'_> {
    fn chunk_bytes(&self, index: usize, out: &mut Vec<u8>) -> Result<(), FormatError> {
        let chunk = self.chunks.get(index).ok_or_else(|| {
            FormatError::ChunkedReadError("chunk index out of range for in-memory provider".into())
        })?;
        out.extend_from_slice(chunk);
        Ok(())
    }
}

/// The chunk-0 message region of a fresh, empty compact-link group: a LinkInfo
/// message advertising no dense storage, followed by a GroupInfo message.
/// Mirrors `build_group_oh`.
fn fresh_group_region() -> OhRegion {
    let mut li = Vec::with_capacity(18);
    li.push(0); // version
    li.push(0); // flags
    li.extend_from_slice(&u64::MAX.to_le_bytes()); // fractal heap addr = UNDEF
    li.extend_from_slice(&u64::MAX.to_le_bytes()); // btree name index addr = UNDEF
    let mut region = OhRegion::empty(OhHeaderProps::PLAIN);
    region.push(MessageType::LINK_INFO, &li);
    region.push(MessageType::GROUP_INFO, &GROUP_INFO_BODY);
    region
}

/// Ensure a group's chunk-0 message `region` carries a Group Info message,
/// appending a minimal one when absent.
///
/// The HDF5 C library refuses to insert a link into a group whose object header
/// has a Link Info message but no Group Info message: on the new-format path
/// `H5G_obj_insert` reads the Group Info message unconditionally and fails with
/// "message type not found". Such a group round-trips for *reading* but cannot
/// be *modified* by the C library. Earlier hdf5-pure releases wrote groups that
/// way, so heal any such header whenever we rewrite one in place.
fn ensure_group_info(region: &mut OhRegion) -> Result<(), Error> {
    let mut p = 0;
    while let Some((msg_type, _body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::GROUP_INFO {
            return Ok(());
        }
        p = body_end;
    }
    region.push(MessageType::GROUP_INFO, &GROUP_INFO_BODY);
    Ok(())
}

/// Ensure a chunk-0 message `region` that carries inline Attribute messages also
/// carries an Attribute Info message, appending the compact-storage one when
/// absent.
///
/// On a version 2 object header the reference C library never counts attribute
/// messages: `H5O__attr_count_real` reads the count out of the Attribute Info
/// message, and reports zero when there is none, even though `H5Aiterate` and
/// `H5Aopen_by_name` still find every attribute. Tools that size their work by
/// that count then skip the attributes silently — `h5repack` copies such an
/// object with none of them. Releases through 0.33.0 wrote every compact
/// attribute set that way, so heal any such header whenever we rewrite one, the
/// same reason [`ensure_group_info`] exists.
///
/// A region already carrying an Attribute Info message is left alone, whichever
/// storage it names: a defined heap address means dense storage, whose count the
/// C library takes from the heap's B-tree instead.
fn ensure_attribute_info(region: &mut OhRegion) -> Result<(), Error> {
    let mut has_attrs = false;
    let mut p = 0;
    while let Some((msg_type, _body, body_end)) = region.next_message(p)? {
        match msg_type {
            MessageType::ATTRIBUTE_INFO => return Ok(()),
            MessageType::ATTRIBUTE => has_attrs = true,
            _ => {}
        }
        p = body_end;
    }
    if has_attrs {
        let body = compact_attribute_info_body(region)?;
        region.push(MessageType::ATTRIBUTE_INFO, &body);
    }
    Ok(())
}

/// Encode a complete object-header Link message (4-byte record header + body)
/// for a hard link `name -> addr`. The caller must have validated that the body
/// fits the u16 size field (see [`flatten_dataset`]); group names are short.
///
/// `creation_order` is the link's creation index, which a group tracking **link**
/// creation order carries on every link and no other group carries at all
/// ([`LinkCreationOrder`] is what hands one out). It is a flagged field of the
/// Link message *body*, quite separate from the object header's own per-message
/// creation index: that one records an *attribute*'s creation order, and the
/// reference C library writes it as the zero [`OhRecordLayout::record`] passes
/// for every other message type.
fn encode_link_message(
    layout: OhRecordLayout,
    name: &str,
    addr: StoredAddress,
    creation_order: Option<u64>,
) -> Vec<u8> {
    let mut link = make_link(name, addr);
    link.creation_order = creation_order;
    let body = link.serialize(OFFSET_SIZE);
    layout.record(MessageType::LINK, &body)
}

/// Patch an existing hard Link message in a chunk-0 message `region`, retargeting
/// the link named `name` to `new_addr` (used to repoint a parent at a relocated
/// child group). The target address is the trailing `OFFSET_SIZE` bytes of the
/// link body for a hard link.
fn patch_link_target(
    region: &mut OhRegion,
    name: &str,
    new_addr: StoredAddress,
) -> Result<(), Error> {
    let mut p = 0;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::LINK {
            if let Ok(link) = LinkMessage::parse(&region[body..body_end], OFFSET_SIZE) {
                if link.name == name {
                    return match link.link_target {
                        LinkTarget::Hard { .. } => {
                            let ofs = body_end - OFFSET_SIZE as usize;
                            region.bytes_mut()[ofs..body_end]
                                .copy_from_slice(&new_addr.get().to_le_bytes());
                            Ok(())
                        }
                        _ => Err(Error::EditUnsupported(
                            "a group on the edited path is reached by a soft/external link",
                        )),
                    };
                }
            }
        }
        p = body_end;
    }
    Err(Error::EditUnsupported(
        "expected child link not found in parent group",
    ))
}

/// Bytes a compact Data Layout message carries ahead of its inline data:
/// version(1) + class(1) + the 2-byte inline size.
const COMPACT_LAYOUT_PREAMBLE: usize = 4;

/// Copy a chunk-0 message `region`, replacing the single (compact) Data Layout
/// message's inline data with `raw` and preserving every other message verbatim.
/// Used by `write_dataset` to overwrite a compact dataset's values. The message
/// header (type and flags) and version byte are kept; only the inline data — and
/// the message size and 2-byte inline-size fields — change. `raw` must fit both
/// the compact layout's own 2-byte size field (HDF5's 64 KiB compact-storage
/// limit) and, once the 4-byte layout preamble is added, the object header's
/// 2-byte message-size field — the tighter of the two, which an overwrite of an
/// existing compact dataset always satisfies.
fn rebuild_compact_layout_region(region: &OhRegion, raw: &[u8]) -> Result<OhRegion, Error> {
    // The bound is on the *message body* the layout becomes — version, class,
    // and the 2-byte inline size ahead of the data — not on `raw` alone, or the
    // last four lengths below the limit would truncate the size field written
    // for them.
    if raw.len() > OBJECT_HEADER_MESSAGE_MAX - COMPACT_LAYOUT_PREAMBLE {
        return Err(Error::EditUnsupported(
            "compact dataset data is too large to overwrite in place",
        ));
    }
    let mut out = Vec::with_capacity(region.len() + raw.len());
    let mut p = 0;
    let mut replaced = false;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::DATA_LAYOUT {
            if body_end - body < 2 || region[body + 1] != 0 {
                return Err(Error::EditUnsupported(
                    "compact-layout overwrite found a non-compact data layout",
                ));
            }
            // New compact layout body: version (kept), class=0, 2-byte inline
            // size, then the data.
            let mut layout = Vec::with_capacity(COMPACT_LAYOUT_PREAMBLE + raw.len());
            layout.push(region[body]); // version (3 or 4)
            layout.push(0); // class = compact
            #[expect(
                clippy::cast_possible_truncation,
                reason = "raw.len() bounded below the u16 inline-size field above"
            )]
            layout.extend_from_slice(&(raw.len() as u16).to_le_bytes());
            layout.extend_from_slice(raw);
            // Message record: type byte, 2-byte size (LE), then the rest of
            // the prefix — flags, and a creation index where the header has one
            // — kept verbatim.
            out.push(region[p]);
            #[expect(
                clippy::cast_possible_truncation,
                reason = "the guard above bounds COMPACT_LAYOUT_PREAMBLE + raw.len(), this \
                          body's exact length, to the 2-byte message-size field"
            )]
            out.extend_from_slice(&(layout.len() as u16).to_le_bytes());
            out.extend_from_slice(&region[p + 3..p + region.layout().prefix_len()]);
            out.extend_from_slice(&layout);
            replaced = true;
        } else {
            out.extend_from_slice(&region[p..body_end]);
        }
        p = body_end;
    }
    if p < region.len() {
        out.extend_from_slice(&region[p..]);
    }
    if !replaced {
        return Err(Error::EditUnsupported(
            "compact dataset header has no data-layout message",
        ));
    }
    Ok(region.with_bytes(out))
}

/// The link creation indexes one commit hands out to the links it adds to one
/// group, and the running maximum its Link Info message ends up recording.
///
/// Link creation order is a separate mechanism from the attribute creation order
/// the object header's own flags describe: a group that tracks it — h5py's
/// `track_order=True`, `H5Pset_link_creation_order`, and every group netCDF-4
/// writes — carries a creation index on every Link message and, in its Link Info
/// message, the maximum it has ever assigned. The reference C library hands out
/// that maximum and increments it (`H5G_obj_insert`), so the recorded value is
/// the *next* index rather than the highest in use, and deleting a link never
/// lowers it. This is the link-side counterpart of [`next_attr_creation_index`],
/// and it works the same way.
///
/// A group that does not track the order has no maximum recorded, hands out no
/// index, and records nothing back — which is every group this crate writes
/// itself.
struct LinkCreationOrder {
    /// The next index to assign, on a group that tracks link creation order.
    next: Option<u64>,
    /// Whether any index has been handed out. Nothing is recorded back
    /// otherwise, so a commit that adds no link to the group leaves its Link
    /// Info message byte-identical.
    assigned: bool,
}

impl LinkCreationOrder {
    /// Read a group's counter out of its object-header message `region`.
    fn for_region(region: &OhRegion) -> Result<Self, Error> {
        let next = find_link_info(region)?.and_then(|(_, _, info)| info.max_creation_order);
        Ok(Self {
            next,
            assigned: false,
        })
    }

    /// The creation index for one link being added, or `None` on a group that
    /// does not track the order. Consecutive calls return consecutive indexes,
    /// so links added in one commit are ordered by the order they are placed in.
    fn take(&mut self) -> Result<Option<u64>, Error> {
        let Some(index) = self.next else {
            return Ok(None);
        };
        self.next = Some(index.checked_add(1).ok_or(Error::EditUnsupported(
            "a group has assigned every link creation index its link-info message can record",
        ))?);
        self.assigned = true;
        Ok(Some(index))
    }

    /// Write the running maximum back into the group's Link Info message, once
    /// every link this commit adds to that group has taken an index.
    ///
    /// The maximum is patched in place rather than re-encoded: it is present
    /// (that is what [`Self::for_region`] read) and it is the eight bytes
    /// following the message's version and flags whatever the file's offset
    /// size, so the record's length — and every offset into the region — is
    /// unchanged.
    fn record(&self, region: &mut OhRegion) -> Result<(), Error> {
        if !self.assigned {
            return Ok(());
        }
        let Some(next) = self.next else {
            return Ok(());
        };
        let (_, body, _) = find_link_info(region)?.ok_or(Error::EditUnsupported(
            "a group's link-info message went missing while its links were being added",
        ))?;
        let max = body.start + 2..body.start + 10;
        if max.end > body.end {
            return Err(Error::EditUnsupported(
                "a group's link-info message is too short to record its link creation order",
            ));
        }
        region.bytes_mut()[max].copy_from_slice(&next.to_le_bytes());
        Ok(())
    }
}

/// The Link Info message in `region`, as `(record start, body range, parsed
/// message)`. An unparseable message is reported as absent, the same reading
/// [`find_attribute_info`] gives an unparseable Attribute Info message — safe
/// here because a group whose link storage this editor cannot account for is
/// refused by [`inspect_group`](WriteEngine::inspect_group) before it reaches
/// this.
fn find_link_info(
    region: &OhRegion,
) -> Result<Option<(usize, core::ops::Range<usize>, LinkInfoMessage)>, Error> {
    let mut p = 0;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::LINK_INFO
            && let Ok(info) = LinkInfoMessage::parse(&region[body..body_end], OFFSET_SIZE)
        {
            return Ok(Some((p, body..body_end, info)));
        }
        p = body_end;
    }
    Ok(None)
}

/// The number of links a group keeps in its object header before they move into
/// a fractal heap: the "maximum compact value" its Group Info message (type
/// 0x000A) declares, or the reference C library's default of 8 where that
/// message stores no link-phase-change values.
///
/// Body: version(1), flags(1), then the maximum compact (2) and minimum dense
/// (2) values if bit 0 of the flags is set, then the estimated entry count (2)
/// and name length (2) if bit 1 is. An absent or truncated message reads as the
/// default, which is the value the C library itself would use for it.
///
/// Not to be confused with [`AttrPhaseChange`], the *attribute* thresholds
/// (`H5Pset_attr_phase_change`) that live in the object header's own prefix:
/// this is the *link* phase change, it lives in a message, and it is what
/// [`reject_dense_link_creation_order`] measures an addition against.
fn max_compact_links(region: &OhRegion) -> Result<u16, Error> {
    /// `H5G_CRT_GINFO_MAX_COMPACT`, the C library's default.
    const DEFAULT_MAX_COMPACT: u16 = 8;
    let mut p = 0;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::GROUP_INFO {
            let stores_phase_change = body_end - body >= 2 && region[body + 1] & 0x01 != 0;
            if stores_phase_change && body + 4 <= body_end {
                return Ok(u16::from_le_bytes([region[body + 2], region[body + 3]]));
            }
            return Ok(DEFAULT_MAX_COMPACT);
        }
        p = body_end;
    }
    Ok(DEFAULT_MAX_COMPACT)
}

/// Refuse to add a link to a group that tracks **link** creation order when the
/// addition would move that group's links into *dense* (fractal-heap) storage.
///
/// `links_after_commit` is how many links the group would hold once this commit's
/// deletions and additions are applied. Past the threshold its Group Info message
/// declares, the reference C library moves a tracked group's links into a fractal
/// heap indexed by *two* B-trees: one on name, and a type 6 one on creation
/// order. This crate emits neither, so the point where the group would stop
/// being compact is the point where an addition stops being reproducible — that
/// work belongs with dense link storage as a whole (issue #102).
///
/// Below the threshold an addition is written rather than refused
/// ([`LinkCreationOrder`] numbers it), as are the two things that write no
/// creation order at all: **removing** a link, which leaves a gap in the order
/// exactly as dropping an Attribute message does and copies the running maximum
/// through untouched (the C library does not lower it on a deletion either), and
/// **retargeting** one ([`patch_link_target`], how a relocated child is rewired),
/// which rewrites an address inside a Link message and leaves every other field
/// of it, creation index included, where it was.
///
/// A group whose links are *already* dense never reaches here at all:
/// [`inspect_group`](WriteEngine::inspect_group) refuses every dense-link group,
/// tracked or not, as it reads the header.
fn reject_dense_link_creation_order(
    region: &OhRegion,
    links_after_commit: usize,
) -> Result<(), Error> {
    let tracked =
        find_link_info(region)?.is_some_and(|(_, _, info)| info.max_creation_order.is_some());
    if tracked && links_after_commit > usize::from(max_compact_links(region)?) {
        return Err(Error::EditUnsupported(
            "a group on the edited path tracks link creation order, and the links this commit \
             adds to it would take it past the compact storage its group-info message allows; \
             dense (fractal-heap) link storage cannot be written in place yet",
        ));
    }
    Ok(())
}

/// Copy a chunk-0 message `region`, dropping the single Link message named
/// `name` and preserving every other message verbatim (used by `delete`). Errors
/// if no such link is present.
fn remove_link_from_region(region: &OhRegion, name: &str) -> Result<OhRegion, Error> {
    let mut out = Vec::with_capacity(region.len());
    let mut p = 0;
    let mut removed = false;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        let mut skip = false;
        if msg_type == MessageType::LINK {
            if let Ok(link) = LinkMessage::parse(&region[body..body_end], OFFSET_SIZE) {
                if link.name == name {
                    skip = true;
                    removed = true;
                }
            }
        }
        if !skip {
            out.extend_from_slice(&region[p..body_end]);
        }
        p = body_end;
    }
    if p < region.len() {
        out.extend_from_slice(&region[p..]);
    }
    if !removed {
        return Err(Error::EditUnsupported(
            "link to delete not found in its parent group",
        ));
    }
    Ok(region.with_bytes(out))
}

/// Apply attribute edits to an object's header `region` *compactly*, preserving
/// every non-attribute message verbatim. A fixed-size `Set`/`Remove` is resolved
/// into `region` directly; a variable-length `Set` (one whose value reports
/// [`AttrValue::var_len_strings`]) is
/// instead collected into the returned `pending_vl_attrs` — its placeholder
/// heap address is only patched, and the message appended to the object's
/// header, by the apply loop once its global heap collection's real address
/// is known (see [`WriteEngine::place_vl_collections`]). A later op for the
/// same name (another `Set`, fixed-size or not, or a `Remove`) replaces or
/// cancels an earlier still-pending variable-length entry, keeping the net
/// effect the same regardless of op order within one commit.
///
/// Only [`plan_attr_ops`] calls this, and only for an object whose attributes
/// are stored compactly: appending an inline Attribute message to an object
/// whose set lives in a fractal heap would leave it carrying two storage forms,
/// so a dense object is routed to a heap rebuild there instead.
fn apply_compact_attr_ops(
    region: &OhRegion,
    ops: &[&AttrOp],
) -> Result<(OhRegion, PendingVlAttrs), Error> {
    let mut out = region.clone();
    let mut pending_vl: PendingVlAttrs = Vec::new();
    for op in ops {
        match op {
            AttrOp::Set { name, value } => {
                pending_vl.retain(|a| &a.msg.name != name);
                if let Some(strings) = value.var_len_strings() {
                    // The message this replaces is dropped now and re-appended
                    // in the apply phase, so the creation index it carried has
                    // to be read before it goes: an overwrite keeps the index
                    // the attribute already had.
                    let creation_index = attr_creation_index(&out, name)?;
                    // Nothing yet to remove from `region` if this name has
                    // never been set as a fixed-size attribute.
                    out = remove_attr_from_region(&out, name, false)?;
                    let msg = build_attr_message(name, value);
                    if msg.serialize(LENGTH_SIZE).len() > OBJECT_HEADER_MESSAGE_MAX {
                        return Err(Error::EditUnsupported(
                            "attribute is too large to encode in place",
                        ));
                    }
                    pending_vl.push(PendingVlAttr {
                        msg,
                        collections: build_global_heap_collections(strings),
                        creation_index,
                    });
                } else {
                    out = set_attr_in_region(&out, name, value)?;
                }
            }
            AttrOp::Remove { name } => {
                let before = pending_vl.len();
                pending_vl.retain(|a| &a.msg.name != name);
                if pending_vl.len() == before {
                    out = remove_attr_from_region(&out, name, true)?;
                }
            }
        }
    }
    Ok((out, pending_vl))
}

/// The attribute storage an edit resolves to, and the header region that carries
/// it. Produced by [`plan_attr_ops`] in the commit preflight, so the apply phase
/// only places bytes.
struct AttrEdits {
    /// The object's chunk-0 message region with the edit applied. Compact: it
    /// already carries every fixed-size attribute. Dense: it carries no attribute
    /// message at all, and the Attribute Info message naming the heap is appended
    /// once that heap is placed.
    region: OhRegion,
    /// What the apply phase still has to place, and where the result is stored.
    attrs: EditedAttrs,
}

/// What an attribute edit leaves for the apply phase, in the storage the result
/// belongs in. The two are alternatives rather than a pair of fields, because
/// every object has exactly one attribute storage: a compact result's
/// variable-length attributes are appended to the header as messages, a dense
/// one's are patched into the set the heap is then built over.
#[derive(Clone)]
enum EditedAttrs {
    /// Variable-length attributes still to be placed and appended as messages.
    /// Empty for an edit that resolved entirely in the preflight, and for one
    /// that left the object with no attributes at all.
    Compact(PendingVlAttrs),
    /// A set to rebuild into a fresh fractal heap.
    Dense(DenseAttrEdit),
}

impl Default for EditedAttrs {
    /// An object with nothing left to place: what a group carries before an
    /// attribute edit reaches it, and what one without attribute edits keeps.
    fn default() -> Self {
        Self::Compact(Vec::new())
    }
}

/// A dense (fractal-heap) attribute set and what its object records about
/// attribute creation order.
///
/// The two travel together because a dense set is never copied as bytes: the
/// heap is rebuilt from the parsed attributes wherever it lands, so the creation
/// indexes the old heap's indexes carried have to be carried alongside or they
/// are lost. An object that does not track the order leaves this
/// [`DenseAttrCreationOrder::Untracked`], which is every set this crate's
/// whole-file writer produces.
#[derive(Clone, Default)]
struct DenseAttrSet {
    attrs: Vec<crate::attribute::AttributeMessage>,
    creation: DenseAttrCreationOrder,
}

impl DenseAttrSet {
    fn is_empty(&self) -> bool {
        self.attrs.is_empty()
    }
}

/// A dense (fractal-heap) attribute set an edit rebuilds, staged by
/// [`plan_attr_ops`] and placed by the apply phase.
#[derive(Clone)]
struct DenseAttrEdit {
    /// Every attribute the object will carry: the ones it already had, in the
    /// order it had them, with this edit's applied over them by name — and what
    /// the object records about their creation order.
    set: DenseAttrSet,
    /// `(index into `attrs`, its global heap collections)` for each
    /// variable-length attribute this edit *sets*. The heap is built over the
    /// attribute message bytes, so each of these has to be patched with its
    /// collection's real address before the build rather than after it, which is
    /// why they are carried to the apply phase instead of resolved here.
    ///
    /// A variable-length attribute the object already had is not in this list and
    /// needs no patch: it was read back carrying the addresses it is stored with,
    /// and they name collections in this same file.
    vl: Vec<(usize, Vec<Vec<u8>>)>,
}

/// Apply attribute edits to an object's header `region`, choosing the storage
/// the result belongs in (issue #102).
///
/// `addr` is the object's pre-commit object-header address, or `None` for a
/// group this commit creates, which has no stored attributes to read.
///
/// Three outcomes, in the order they are tried:
///
/// - **Compact.** The region is rewritten message by message, so every attribute
///   the edit does not name keeps its exact bytes. An ordinary `set_attr` on an
///   ordinary object takes this route.
/// - **Dense.** The object already stores its attributes in a fractal heap, or
///   this edit takes it past what an object header holds compactly — more than
///   [`MAX_COMPACT_ATTRS`] attributes, or one whose message overflows the
///   header's 2-byte message-size field. The object's whole attribute set is read
///   back, the edits are applied to it, and the apply phase builds a fresh heap
///   for the result. Those are the two halves of the disjunction
///   `file_writer::needs_dense_attrs` selects on, so an object an edit sends to a
///   heap is one the whole-file writer would have written to a heap — measured
///   here over the ops as staged, which is why setting an oversized attribute and
///   removing it again in one commit still sends the object to a heap.
/// - **Neither.** The edit removes the object's last attribute, so the rebuilt
///   header carries no attribute message and no Attribute Info message — what
///   this crate and the reference C library both write for an object with none.
///
/// A dense rebuild is a re-encoding rather than a byte copy: the attributes are
/// read into messages and re-serialized, as [`WriteEngine::read_object`] already
/// does for a copied object's heap. Name, datatype, dataspace and value survive
/// it; a version 1 or 2 message becomes the version 3 one dense storage holds,
/// and attribute creation order — which this crate indexes for neither storage
/// form — does not.
///
/// An object that is dense stays dense, even where the edit takes its set back
/// under the threshold. The reference C library keeps one dense down to its
/// `min_dense` (6 by default) as well, and re-encoding a set that is only losing
/// members buys nothing.
fn plan_attr_ops<S: Source + ?Sized>(
    src: &S,
    base: BaseAddress,
    addr: Option<u64>,
    region: &OhRegion,
    ops: &[&AttrOp],
) -> Result<AttrEdits, Error> {
    let dense_now = region_uses_dense_attrs(region)?;
    if !dense_now {
        // An attribute past the header's message-size field has no compact form
        // at all, so the compact pass would refuse it rather than report a count
        // this could act on. Ask before running it, not after.
        let oversized = ops.iter().any(|op| match op {
            AttrOp::Set { name, value } => {
                build_attr_message(name, value).serialize(LENGTH_SIZE).len()
                    > OBJECT_HEADER_MESSAGE_MAX
            }
            AttrOp::Remove { .. } => false,
        });
        if !oversized {
            let (out, pending_vl) = apply_compact_attr_ops(region, ops)?;
            // An edit that only *removes* stays compact whatever the count is.
            // A region already holding more attributes than this writer would
            // emit came from another writer — the C library's `max_compact` is a
            // property list setting — and taking one away is no reason to
            // re-encode the ones that remain.
            let only_removes = !ops.iter().any(|op| matches!(op, AttrOp::Set { .. }));
            if only_removes || compact_attr_count(&out)? + pending_vl.len() <= MAX_COMPACT_ATTRS {
                return Ok(AttrEdits {
                    region: out,
                    attrs: EditedAttrs::Compact(pending_vl),
                });
            }
        }
    }

    // The compact pass refuses a shared attribute message as it walks the
    // messages it copies; this path never walks them, so it asks here — before
    // the read below resolves one into a message indistinguishable from a private
    // attribute.
    if region_has_shared_attr(region)? {
        return Err(Error::EditUnsupported(SHARED_ATTRIBUTE_MESSAGE));
    }

    // A heap is rebuilt from the object's whole attribute set — half of which may
    // live in a heap the header only names — so read that set back before
    // applying the edits to it. A group this commit creates has none.
    let existing = match addr {
        Some(addr) => read_object_attrs(src, AccessMode::ReadWrite, addr, base)?,
        None => Vec::new(),
    };
    // Each attribute travels with the global heap collections it still needs
    // placed (`Some` only for a variable-length attribute this edit sets) and
    // with the creation index its object records for it, so no removal can shift
    // one away from the bytes — or the index — it belongs to.
    let existing = dense_attr_set(region, existing)?;
    let tracked = region.layout().tracks_creation_order();
    // The next index the object would hand out: what a *new* attribute takes.
    // `dense_attr_set` has already raised it past every index in use.
    let mut next_index = match &existing.creation {
        DenseAttrCreationOrder::Tracked { max, .. } => *max,
        DenseAttrCreationOrder::Untracked => 0,
    };
    let mut set: Vec<DenseAttrSlot> = existing
        .attrs
        .into_iter()
        .enumerate()
        .map(|(i, msg)| DenseAttrSlot {
            msg,
            collections: None,
            creation_index: existing.creation.index_at(i),
        })
        .collect();
    for op in ops {
        match op {
            AttrOp::Set { name, value } => {
                let msg = build_attr_message(name, value);
                let collections = value.var_len_strings().map(build_global_heap_collections);
                match set.iter_mut().find(|slot| &slot.msg.name == name) {
                    // Setting an attribute the object already has replaces it
                    // where it stands, so a repeated `set_attr` does not reorder
                    // the set — and it keeps the creation index it had, so an
                    // iteration by creation order does not reorder it either.
                    Some(slot) => {
                        slot.msg = msg;
                        slot.collections = collections;
                    }
                    None => {
                        let creation_index = tracked.then_some(next_index);
                        if tracked {
                            next_index = bump_creation_index(next_index)?;
                        }
                        set.push(DenseAttrSlot {
                            msg,
                            collections,
                            creation_index,
                        });
                    }
                }
            }
            AttrOp::Remove { name } => {
                let before = set.len();
                // A deletion leaves a gap in the creation order rather than
                // renumbering what is left, and does not lower the maximum:
                // the reference C library hands out indexes from a counter that
                // only ever rises.
                set.retain(|slot| &slot.msg.name != name);
                if set.len() == before {
                    return Err(Error::EditUnsupported("attribute to remove was not found"));
                }
            }
        }
    }

    // Every attribute message is rebuilt from the set above, so the region keeps
    // none of them, nor the Attribute Info message naming the storage they were
    // in.
    let region = strip_attr_messages(region)?;
    // No attributes left: the header carries neither an Attribute message nor an
    // Attribute Info one. `append_dense_attrs` would decline to build a heap for
    // an empty set anyway; saying `None` here is what makes that a property of
    // this decision rather than of the emitter that carries it out.
    if set.is_empty() {
        return Ok(AttrEdits {
            region,
            attrs: EditedAttrs::default(),
        });
    }
    let mut attrs = Vec::with_capacity(set.len());
    let mut indices = Vec::with_capacity(set.len());
    let mut vl = Vec::new();
    for (i, slot) in set.into_iter().enumerate() {
        if let Some(collections) = slot.collections {
            vl.push((i, collections));
        }
        // Every slot of a tracked object carries an index: one read from its
        // storage, or one this edit just handed out. `indices` is discarded for
        // an untracked object, where none of them do.
        debug_assert!(
            !tracked || slot.creation_index.is_some(),
            "a tracked object's attribute must carry a creation index"
        );
        indices.push(slot.creation_index.unwrap_or_default());
        attrs.push(slot.msg);
    }
    let creation = if tracked {
        DenseAttrCreationOrder::Tracked {
            indices,
            max: next_index,
            indexed: region.layout().indexes_creation_order(),
        }
    } else {
        DenseAttrCreationOrder::Untracked
    };
    // Moving a set into a heap takes it out of reach of the reference repointing
    // a commit does as its last act: `reference_patch::scan_object` reads an
    // object whose attributes are dense as unproven and collects no edit from it.
    // An object *already* dense is not this refusal's business — its attributes
    // stood outside that walk before this edit and still do — but converting one
    // would strand a reference the header was keeping reachable, and permanently,
    // since an object that goes dense stays dense (issue #324).
    if !dense_now {
        for attr in &attrs {
            if crate::reference_patch::attribute_references_are_repointable(&attr.datatype) {
                return Err(Error::EditUnsupported(
                    REFERENCE_ATTRIBUTE_WOULD_LEAVE_THE_HEADER,
                ));
            }
        }
    }
    // What the heap itself cannot represent is the last refusal, and it belongs
    // in the preflight: past here the commit is placing bytes.
    crate::file_writer::dense_attrs_check(&attrs).map_err(Error::Format)?;
    Ok(AttrEdits {
        region,
        attrs: EditedAttrs::Dense(DenseAttrEdit {
            set: DenseAttrSet { attrs, creation },
            vl,
        }),
    })
}

/// One attribute of the set a dense rebuild is assembled from.
struct DenseAttrSlot {
    msg: crate::attribute::AttributeMessage,
    /// Global heap collections still to be placed, for a variable-length
    /// attribute *this edit* sets.
    collections: Option<Vec<Vec<u8>>>,
    /// The creation index the object records for it, where it tracks the order.
    creation_index: Option<u16>,
}

/// Whether an object's chunk-0 message `region` stores its attributes densely —
/// an Attribute Info message naming a fractal heap. See
/// [`attribute_info_is_dense`] for why the message's mere presence is not that.
fn region_uses_dense_attrs(region: &OhRegion) -> Result<bool, Error> {
    let mut p = 0;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::ATTRIBUTE_INFO
            && attribute_info_is_dense(&region[body..body_end])
        {
            return Ok(true);
        }
        p = body_end;
    }
    Ok(false)
}

/// Whether any Attribute message in `region` carries a message flag — which for
/// an attribute means the body is a shared (SOHM) record rather than the
/// attribute itself. Reads the same byte [`parse_compact_attr_name`] does, so the
/// two attribute paths refuse exactly the same messages.
fn region_has_shared_attr(region: &OhRegion) -> Result<bool, Error> {
    let mut p = 0;
    while let Some((msg_type, _body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::ATTRIBUTE && region[p + 3] != 0 {
            return Ok(true);
        }
        p = body_end;
    }
    Ok(false)
}

/// Copy a chunk-0 message `region`, dropping every Attribute message and the
/// Attribute Info message that names their storage. What the object carries in
/// their place is [`plan_attr_ops`]'s decision, appended by the apply phase.
fn strip_attr_messages(region: &OhRegion) -> Result<OhRegion, Error> {
    let mut out = Vec::with_capacity(region.len());
    let mut p = 0;
    while let Some((msg_type, _body, body_end)) = region.next_message(p)? {
        if !matches!(
            msg_type,
            MessageType::ATTRIBUTE | MessageType::ATTRIBUTE_INFO
        ) {
            out.extend_from_slice(&region[p..body_end]);
        }
        p = body_end;
    }
    if p < region.len() {
        out.extend_from_slice(&region[p..]);
    }
    Ok(region.with_bytes(out))
}

/// Every attribute an object carries, compact or dense, parsed into messages.
///
/// This read is what makes a dense edit possible at all: the set an edit works
/// from is the object's whole attribute set, and for a dense object none of it
/// is in the header — `extract_attributes_full_from_source` walks the fractal
/// heap the Attribute Info message names. It reads both storage forms, so no
/// caller has to ask which one an object uses.
///
/// Shared with [`WriteEngine::read_object`], which needs the same set for the
/// same reason when it copies a dense object. That caller checks
/// [`crate::file_writer::dense_attrs_check`] on what it reads; the attribute
/// editor checks it after applying its edits, on the set it will actually write.
fn read_object_attrs<S: Source + ?Sized>(
    src: &S,
    access_mode: AccessMode,
    addr: u64,
    base: BaseAddress,
) -> Result<Vec<crate::attribute::StoredAttribute>, Error> {
    let header =
        ObjectHeader::parse_from_source(src, access_mode, addr, OFFSET_SIZE, LENGTH_SIZE, base)?;
    if base.get() > src.len() {
        return Err(Error::EditUnsupported(
            "this file's userblock is larger than the file itself",
        ));
    }
    // Heap addresses in the Attribute Info message are stored relative to the
    // base address, so the walk gets the file framed past its userblock — the
    // same view the reader uses. `base` is 0 for a plain file, where this is
    // `src` itself.
    let framed = BaseOffsetSource { inner: src, base };
    crate::attribute::extract_stored_attributes_from_source(
        &framed,
        access_mode,
        &header,
        OFFSET_SIZE,
        LENGTH_SIZE,
        // No shared-message table: this engine refuses an object carrying a
        // shared attribute message before it gets here ([`SHARED_ATTRIBUTE_MESSAGE`]),
        // and resolving one silently is exactly what that refusal prevents — the
        // resolved copy is indistinguishable from a private attribute, and would
        // be re-emitted as one, leaving the file's reference count naming an
        // attribute that no longer exists.
        None,
    )
    .map_err(Error::Format)
}

/// Assemble a [`DenseAttrSet`] from an object's stored attributes and what its
/// header `region` says about attribute creation order.
///
/// An object that does not track the order gets
/// [`DenseAttrCreationOrder::Untracked`], which is the whole story for every
/// file this crate writes itself. One that does keeps each attribute's stored
/// index, and the maximum its Attribute Info message records — the next index it
/// would hand out, which a deletion does not lower, so it cannot be recomputed
/// from the attributes that remain. An attribute whose storage recorded no index
/// (a header that claims to track the order and then does not) is given the next
/// unused one rather than silently colliding with a real index.
fn dense_attr_set(
    region: &OhRegion,
    stored: Vec<crate::attribute::StoredAttribute>,
) -> Result<DenseAttrSet, Error> {
    let layout = region.layout();
    if !layout.tracks_creation_order() {
        return Ok(DenseAttrSet {
            attrs: stored.into_iter().map(|a| a.message).collect(),
            creation: DenseAttrCreationOrder::Untracked,
        });
    }
    let mut next = next_attr_creation_index(region)?;
    let mut attrs = Vec::with_capacity(stored.len());
    let mut indices = Vec::with_capacity(stored.len());
    for attr in stored {
        let index = match attr.creation_index {
            Some(index) => index,
            None => {
                let index = next;
                next = bump_creation_index(next)?;
                index
            }
        };
        next = next.max(bump_creation_index(index)?);
        attrs.push(attr.message);
        indices.push(index);
    }
    Ok(DenseAttrSet {
        attrs,
        creation: DenseAttrCreationOrder::Tracked {
            indices,
            max: next,
            indexed: layout.indexes_creation_order(),
        },
    })
}

/// One past `index`, refusing the object that has exhausted the 2-byte creation
/// index its Attribute Info message records.
fn bump_creation_index(index: u16) -> Result<u16, Error> {
    index.checked_add(1).ok_or(Error::EditUnsupported(
        "an object has assigned every attribute creation index its header can record",
    ))
}

/// Whether an Attribute Info (0x0015) message body denotes *dense* (fractal-heap)
/// attribute storage — a *defined* heap address. The reference C library and h5py
/// emit an Attribute Info message with an *undefined* heap address even for
/// compact, inline attributes in the latest format (to carry creation-order
/// metadata), so its mere presence is not dense storage; only a defined heap
/// address is. An unparseable message is treated as dense (refused conservatively).
/// Mirrors the copy path's dense detection so the compact-attribute editors accept
/// the undefined-address message that nearly every real-world object carries.
pub(crate) fn attribute_info_is_dense(body: &[u8]) -> bool {
    match crate::attribute_info::AttributeInfoMessage::parse(body, OFFSET_SIZE) {
        Ok(ai) => ai.fractal_heap_address.is_some(),
        Err(_) => true,
    }
}

/// Copy a message region, dropping all Attribute messages named `name` and then
/// appending a fresh compact Attribute message for `value`.
fn set_attr_in_region(region: &OhRegion, name: &str, value: &AttrValue) -> Result<OhRegion, Error> {
    let body = encode_attr_body(name, value)?;
    let keep = attr_creation_index(region, name)?;
    put_attr_message(region, name, &body, keep)
}

/// The creation index the Attribute message named `name` carries, or `None`
/// where the object has no such attribute — or does not track the order at all.
fn attr_creation_index(region: &OhRegion, name: &str) -> Result<Option<u16>, Error> {
    if !region.layout().tracks_creation_order() {
        return Ok(None);
    }
    let mut p = 0;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::ATTRIBUTE
            && parse_compact_attr_name(region, p, body, body_end)? == name
        {
            return Ok(region.creation_index(p));
        }
        p = body_end;
    }
    Ok(None)
}

/// The highest creation index any Attribute record in `region` carries.
fn highest_attr_creation_index(region: &OhRegion) -> Result<Option<u16>, Error> {
    let mut highest = None;
    let mut p = 0;
    while let Some((msg_type, _body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::ATTRIBUTE {
            highest = highest.max(region.creation_index(p));
        }
        p = body_end;
    }
    Ok(highest)
}

/// The next unused attribute creation index for an object whose messages are
/// `region`.
///
/// The reference C library hands out `ainfo.max_crt_idx` and increments it
/// (`H5O__attr_create`), so the recorded maximum is the *next* index rather
/// than the highest in use, and deleting an attribute never lowers it. Read it
/// from the Attribute Info message where there is one, and otherwise derive one
/// past the highest index the records carry — the best evidence a header with no
/// such message leaves.
fn next_attr_creation_index(region: &OhRegion) -> Result<u16, Error> {
    let recorded = find_attribute_info(region)?
        .and_then(|(_, _, info)| info.max_creation_index)
        .unwrap_or(0);
    let derived = highest_attr_creation_index(region)?.map_or(0, |i| u32::from(i) + 1);
    let next = u32::from(recorded).max(derived);
    next.narrow_or_else(|| {
        Error::EditUnsupported(
            "an object has assigned every attribute creation index its header can record",
        )
    })
}

/// The Attribute Info message in `region`, as `(record start, body range, parsed
/// message)`. An unparseable message is reported as absent, which leaves it
/// untouched: `attribute_info_is_dense` reads the same message and refuses the
/// object rather than letting an edit rewrite one it did not understand.
fn find_attribute_info(
    region: &OhRegion,
) -> Result<Option<(usize, core::ops::Range<usize>, AttributeInfoMessage)>, Error> {
    let mut p = 0;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::ATTRIBUTE_INFO
            && let Ok(info) = AttributeInfoMessage::parse(&region[body..body_end], OFFSET_SIZE)
        {
            return Ok(Some((p, body..body_end, info)));
        }
        p = body_end;
    }
    Ok(None)
}

/// The Attribute Info message body an object storing its attributes compactly
/// needs, in `region`'s layout.
///
/// A header that tracks attribute creation order records the maximum index ever
/// assigned here, and nowhere else, so a header carrying no such message leaves
/// only the records to derive it from: one past the highest index in use. That
/// under-counts an object whose most recently created attributes have since been
/// deleted, which is why an edit *patches* an existing message
/// ([`put_attr_message`]) rather than rebuilding one.
fn compact_attribute_info_body(region: &OhRegion) -> Result<Vec<u8>, Error> {
    let layout = region.layout();
    let info = AttributeInfoMessage {
        max_creation_index: layout
            .tracks_creation_order()
            .then(|| next_attr_creation_index(region))
            .transpose()?,
        indexes_creation_order: layout.indexes_creation_order(),
        fractal_heap_address: None,
        btree_name_index_address: None,
        btree_creation_order_address: None,
    };
    Ok(info.serialize(OFFSET_SIZE))
}

/// Copy a message region, dropping every Attribute message named `name` and
/// appending one carrying `body`, then keep the object's Attribute Info message
/// in step with the creation index the new record takes.
///
/// `keep` is the index an *overwrite* preserves: the reference C library modifies
/// an attribute in place, so writing over one does not move it in an iteration
/// by creation order. `None` asks for the object's next unused index, which then
/// becomes the maximum the Attribute Info message records. Both are inert on a
/// header that does not track the order, where no record has the field.
fn put_attr_message(
    region: &OhRegion,
    name: &str,
    body: &[u8],
    keep: Option<u16>,
) -> Result<OhRegion, Error> {
    let layout = region.layout();
    let (creation_index, new_max) = if layout.tracks_creation_order() {
        match keep {
            Some(index) => (index, None),
            None => {
                let index = next_attr_creation_index(region)?;
                let next = index.checked_add(1).ok_or(Error::EditUnsupported(
                    "an object has assigned every attribute creation index its header can record",
                ))?;
                (index, Some(next))
            }
        }
    } else {
        (0, None)
    };
    let new_msg = layout.record_with_creation_index(MessageType::ATTRIBUTE, body, creation_index);

    let mut out = Vec::with_capacity(region.len() + new_msg.len());
    let mut p = 0;
    while let Some((msg_type, msg_body, body_end)) = region.next_message(p)? {
        match msg_type {
            MessageType::ATTRIBUTE
                if parse_compact_attr_name(region, p, msg_body, body_end)? == name =>
            {
                p = body_end;
                continue;
            }
            // Bumping the recorded maximum re-encodes the message, since a
            // header that was not recording one at all needs the field added.
            MessageType::ATTRIBUTE_INFO if new_max.is_some() => {
                if let Ok(mut info) =
                    AttributeInfoMessage::parse(&region[msg_body..body_end], OFFSET_SIZE)
                {
                    info.max_creation_index = new_max;
                    info.indexes_creation_order |= layout.indexes_creation_order();
                    let encoded = info.serialize(OFFSET_SIZE);
                    let len: u16 = encoded.len().narrow_or_else(|| {
                        Error::EditUnsupported("an Attribute Info message is too large to record")
                    })?;
                    // Only the body and its length change: the record keeps its
                    // own flags — the reference C library marks this message
                    // "don't share" — and its creation index.
                    out.push(region[p]);
                    out.extend_from_slice(&len.to_le_bytes());
                    out.extend_from_slice(&region[p + 3..msg_body]);
                    out.extend_from_slice(&encoded);
                    p = body_end;
                    continue;
                }
                out.extend_from_slice(&region[p..body_end]);
            }
            _ => out.extend_from_slice(&region[p..body_end]),
        }
        p = body_end;
    }
    out.extend_from_slice(&new_msg);
    if p < region.len() {
        out.extend_from_slice(&region[p..]);
    }
    let mut out = region.with_bytes(out);
    // An object that had no Attribute Info message to bump gets the one
    // `build_v2_object_header` would add anyway, now, so the maximum this edit
    // just assigned is recorded rather than re-derived from the records.
    if new_max.is_some() && find_attribute_info(&out)?.is_none() {
        let body = compact_attribute_info_body(&out)?;
        out.push(MessageType::ATTRIBUTE_INFO, &body);
    }
    Ok(out)
}

/// Copy a message region, dropping all Attribute messages named `name`. When
/// `required` is true, an absent `name` is an [`Error::EditUnsupported`] (a
/// `Remove` of a nonexistent attribute); when false, it is not an error (a
/// `Set` of a fresh variable-length attribute may have no fixed-size message
/// to remove from the region yet).
fn remove_attr_from_region(
    region: &OhRegion,
    name: &str,
    required: bool,
) -> Result<OhRegion, Error> {
    let mut out = Vec::with_capacity(region.len());
    let mut p = 0;
    let mut removed = false;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        let mut skip = false;
        if msg_type == MessageType::ATTRIBUTE {
            let attr_name = parse_compact_attr_name(region, p, body, body_end)?;
            if attr_name == name {
                skip = true;
                removed = true;
            }
        }
        if !skip {
            out.extend_from_slice(&region[p..body_end]);
        }
        p = body_end;
    }
    if p < region.len() {
        out.extend_from_slice(&region[p..]);
    }
    if !removed && required {
        return Err(Error::EditUnsupported("attribute to remove was not found"));
    }
    Ok(region.with_bytes(out))
}

/// How many attributes an object's message `region` carries inline. Only a
/// compact region is ever counted this way: a dense object carries no inline
/// Attribute message at all, and would count zero — which is why
/// [`plan_attr_ops`] decides an object's storage before it reaches here.
fn compact_attr_count(region: &OhRegion) -> Result<usize, Error> {
    let mut count = 0usize;
    let mut p = 0;
    while let Some((msg_type, _body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::ATTRIBUTE {
            count += 1;
        }
        p = body_end;
    }
    Ok(count)
}

fn parse_compact_attr_name(
    region: &[u8],
    msg_start: usize,
    body: usize,
    body_end: usize,
) -> Result<String, Error> {
    if region[msg_start + 3] != 0 {
        return Err(Error::EditUnsupported(SHARED_ATTRIBUTE_MESSAGE));
    }
    // Only the name is wanted here, and it is the one field that never depends on
    // the datatype or dataspace — either of which may be a reference to a
    // committed message this walk has no file context to follow. Reading the name
    // alone lets an edit pass over such an attribute instead of refusing the
    // whole object because one of its neighbours is committed.
    crate::attribute::message_name(&region[body..body_end]).map_err(Error::Format)
}

fn encode_attr_body(name: &str, value: &AttrValue) -> Result<Vec<u8>, Error> {
    // `apply_compact_attr_ops`'s `Set` branch — this function's only caller —
    // handles a value that needs the global heap itself (staging it into
    // `pending_vl` instead of calling `set_attr_in_region`/here), so this value
    // is always inline by construction, not by a check made at this call site.
    debug_assert!(
        value.var_len_strings().is_none(),
        "a variable-length attribute must be intercepted by apply_compact_attr_ops before reaching encode_attr_message"
    );
    let body = build_attr_message(name, value).serialize(LENGTH_SIZE);
    if body.len() > OBJECT_HEADER_MESSAGE_MAX {
        return Err(Error::EditUnsupported(
            "group attribute is too large to encode in place",
        ));
    }
    Ok(body)
}

/// Parse the version-2 object-header message record at `p` within a chunk-0
/// message region, returning `(message type, body start, body end)`; the next
/// record begins at `body end`. Returns `Ok(None)` once fewer than 4 bytes
/// remain (a clean end of the region), and `Err` if a record's declared body
/// runs past the region. Centralizes the bounds check shared by every walker.
/// Rebuild a superblock-extension object header's message region (as collapsed by
/// [`WriteEngine::gather_oh_messages`]) with
/// its File Space Info message replaced by `info`, preserving every other message
/// verbatim. The persisting message is fixed-size, so the region length is stable.
/// Shared by the whole-file mirror commit and the bounded finalize so both write
/// the same extension bytes.
pub(crate) fn rewrite_extension_region_bytes(
    region: &OhRegion,
    info: &FileSpaceInfo,
) -> Result<OhRegion, Error> {
    let new_body = info.serialize();
    // The message body is the fixed-size File Space Info record (≤ 125 bytes),
    // so it always fits the u16 size field.
    let new_len: u16 = new_body
        .len()
        .narrow_or_else(|| Error::EditUnsupported("File Space Info message too large"))?;
    let mut out = Vec::with_capacity(region.len());
    let mut p = 0;
    let mut replaced = false;
    while let Some((msg_type, _body, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::FILE_SPACE_INFO {
            out.push(region[p]); // message type byte
            out.extend_from_slice(&new_len.to_le_bytes());
            // Preserve the message flags (0x14) and, where the header carries
            // one, the record's creation index.
            out.extend_from_slice(&region[p + 3..p + region.layout().prefix_len()]);
            out.extend_from_slice(&new_body);
            replaced = true;
        } else {
            out.extend_from_slice(&region[p..body_end]);
        }
        p = body_end;
    }
    if !replaced {
        // Persistence is armed only when the extension already carries a File
        // Space Info message, so this is unreachable; refuse rather than
        // silently restructure an extension we did not understand.
        return Err(Error::EditUnsupported(
            "a persisting file's superblock extension has no File Space Info message",
        ));
    }
    Ok(region.with_bytes(out))
}

/// Parse and validate a version 2 object header's prefix, returning the absolute
/// `[start, end)` byte range of its chunk-0 message region.
///
/// `prefix` holds the bytes at `[addr, addr + prefix.len())` — up to
/// [`OH_PREFIX_MAX`], fewer when the header sits near the end of the image — and
/// `file_len` is the length of the image the header lives in, which bounds the
/// region. Rejects headers that are not OHDR v2. Everything the prefix declares
/// ([`OhHeaderProps`]) is returned with the region: the record layout, because a
/// header that tracks attribute creation order carries 6-byte message records
/// and every walk of this region has to step by that width, and the two optional
/// blocks, because nothing below the prefix records them and a rebuild has to
/// put them back.
fn oh_region_at(
    prefix: &[u8],
    addr: u64,
    file_len: u64,
) -> Result<(u64, u64, OhHeaderProps), Error> {
    if prefix.len() < 6 || &prefix[..4] != b"OHDR" || prefix[4] != 2 {
        return Err(Error::EditUnsupported(
            "an object does not use a version 2 object header",
        ));
    }
    let flags = prefix[5];
    let layout = OhRecordLayout::from_header_flags(flags);
    let mut pos = 6usize;
    let mut take = |len: usize| -> Result<usize, Error> {
        let at = pos;
        pos += len;
        if prefix.len() < pos {
            return Err(Error::EditUnsupported("truncated object header"));
        }
        Ok(at)
    };
    let times = if flags & OH_FLAG_STORE_TIMES != 0 {
        let at = take(ObjectTimes::LEN)?;
        Some(ObjectTimes::parse(prefix, at))
    } else {
        None
    };
    let attr_phase_change = if flags & OH_FLAG_ATTR_PHASE_CHANGE != 0 {
        let at = take(AttrPhaseChange::LEN)?;
        Some(AttrPhaseChange::parse(prefix, at))
    } else {
        None
    };
    let props = OhHeaderProps {
        layout,
        times,
        attr_phase_change,
    };
    let size_width = usize::from(UintWidth::from_flags(flags).get());
    if prefix.len() < pos + size_width {
        return Err(Error::EditUnsupported("truncated object header"));
    }
    let chunk0_size = read_le(&prefix[pos..pos + size_width]) as u64;
    pos += size_width;
    let region_start = addr
        .checked_add(pos as u64)
        .ok_or(Error::EditUnsupported("truncated object header"))?;
    // The region is followed by a 4-byte checksum, which must also be present.
    let region_end = region_start
        .checked_add(chunk0_size)
        .filter(|e| e.checked_add(4).is_some_and(|end| end <= file_len))
        .ok_or(Error::EditUnsupported("truncated object header"))?;
    Ok((region_start, region_end, props))
}

/// One chunk of a version 2 object header, read out of a file image.
///
/// The buffer stops at the end of the chunk's message region: the trailing
/// checksum is never walked, and it has already been confirmed present. `span`
/// covers the *whole* on-disk chunk including that checksum, so it can be handed
/// to the free list when the header is reclaimed.
pub(crate) struct OhChunk {
    /// Absolute file address and full on-disk length of the chunk.
    pub(crate) span: (u64, u64),
    /// The chunk's bytes, from `span.0` through the end of its message region.
    buf: Vec<u8>,
    /// Offset of the first message within [`buf`](Self::buf).
    messages_start: usize,
    /// What chunk 0 of this header declared in its prefix, which every chunk of
    /// the header shares.
    props: OhHeaderProps,
}

impl OhChunk {
    /// The slice to walk messages in, and the offset to start at. The two are
    /// returned together because [`OhRecordLayout::next_message`] must not read
    /// past the end of the message region into the checksum.
    pub(crate) fn message_region(&self) -> (&[u8], usize) {
        (&self.buf, self.messages_start)
    }

    /// The record layout to walk [`message_region`](Self::message_region) in.
    pub(crate) fn layout(&self) -> OhRecordLayout {
        self.props.layout
    }

    /// Everything this header's chunk-0 prefix declared.
    pub(crate) fn props(&self) -> OhHeaderProps {
        self.props
    }
}

/// Read chunk 0 of the version 2 object header at `addr` out of `src`.
fn read_oh_chunk0<S: Source + ?Sized>(src: &S, addr: u64) -> Result<OhChunk, Error> {
    let file_len = src.len();
    let window = file_len
        .saturating_sub(addr)
        .min(OH_PREFIX_MAX as u64)
        .to_usize()?;
    let prefix = src.read_metadata_at(addr, window)?;
    let (rs, re, props) = oh_region_at(&prefix, addr, file_len)?;
    // `re >= rs > addr`, so both differences are non-negative, and `oh_region_at`
    // has checked that the 4-byte checksum past `re` is present.
    let len = (re - addr).to_usize()?;
    Ok(OhChunk {
        span: (addr, len as u64 + 4),
        buf: src.read_metadata_at(addr, len)?,
        messages_start: (rs - addr).to_usize()?,
        props,
    })
}

/// Read every chunk of the version 2 object header at `addr`, chunk 0 first,
/// following each `Continuation` message to its `OCHK` block.
///
/// This is the one traversal of a header's chunk chain: [`WriteEngine::gather_oh_messages`]
/// collects the messages out of the result and
/// [`oh_chunk_spans`](WriteEngine::oh_chunk_spans) collects the extents, so the
/// two cannot disagree about what a header occupies.
pub(crate) fn read_oh_chunks<S: Source + ?Sized>(
    src: &S,
    addr: u64,
    base: BaseAddress,
) -> Result<Vec<OhChunk>, Error> {
    let mut chunks = vec![read_oh_chunk0(src, addr)?];
    let mut i = 0;
    while i < chunks.len() {
        if chunks.len() > MAX_OH_CHUNKS {
            return Err(Error::EditUnsupported(
                "object header has too many continuation chunks",
            ));
        }
        // Collect this chunk's continuations before extending the worklist, so the
        // borrow of `chunks[i]` ends first.
        let mut found = Vec::new();
        let props = chunks[i].props();
        let layout = props.layout;
        let (region, mut p) = chunks[i].message_region();
        while let Some((msg_type, body, body_end)) = layout.next_message(region, p)? {
            if msg_type == MessageType::OBJECT_HEADER_CONTINUATION {
                found.push(read_oh_continuation(
                    src, region, body, body_end, base, props,
                )?);
            }
            p = body_end;
        }
        i += 1;
        chunks.extend(found);
    }
    Ok(chunks)
}

/// Read the `OCHK` continuation block a continuation message points at.
///
/// `region[body..body_end]` is the continuation message's body: the block's
/// base-relative address followed by its length.
fn read_oh_continuation<S: Source + ?Sized>(
    src: &S,
    region: &[u8],
    body: usize,
    body_end: usize,
    base: BaseAddress,
    props: OhHeaderProps,
) -> Result<OhChunk, Error> {
    if body_end - body < (OFFSET_SIZE + LENGTH_SIZE) as usize {
        return Err(Error::EditUnsupported("malformed continuation message"));
    }
    let off = u64::from_le_bytes(region[body..body + 8].try_into().unwrap());
    let len = u64::from_le_bytes(region[body + 8..body + 16].try_into().unwrap());
    // The block address is stored relative to the base address; shift it to an
    // absolute file offset before reading.
    let off = base.absolute(StoredAddress::new(off))?;
    // An OCHK block is signature(4) + messages + checksum(4).
    let end = off
        .checked_add(len)
        .filter(|&e| e <= src.len() && len >= 8)
        .ok_or(Error::EditUnsupported("continuation block out of bounds"))?;
    let want = (end - off).to_usize()?;
    let mut buf = src.read_metadata_at(off, want)?;
    if buf[..4] != *b"OCHK" {
        return Err(Error::EditUnsupported(
            "invalid continuation block signature",
        ));
    }
    // Trim the trailing checksum so the message walk stops at the last message.
    buf.truncate(want - 4);
    Ok(OhChunk {
        span: (off, len),
        buf,
        messages_start: 4,
        props,
    })
}

/// How a version 2 object header's message records are laid out, and what its
/// flags say about attribute creation order.
///
/// Every record opens with a type byte, a 2-byte body size and a flags byte. A
/// header that *tracks* attribute creation order — bit 2 of the object header's
/// own flags, what `H5Pset_attr_creation_order` and h5py's `track_order=True`
/// turn on, and what netCDF-4 sets on every object it writes — follows those
/// with a 2-byte creation index, so its records are 6 bytes wide rather than 4.
/// A header that also *indexes* that order (bit 3) carries a creation-order
/// B-tree beside the name index once its attributes go dense; the reference C
/// library reads that bit back out of the header when it builds an Attribute
/// Info message, so a rewrite that dropped it would quietly stop indexing.
///
/// Both bits are properties of the whole header, so chunk 0 and every
/// continuation block of one header share a layout. Carrying it beside the bytes
/// ([`OhRegion`]) is what keeps the two dozen walkers and the emitters in this
/// module from having to agree about it one by one.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub(crate) struct OhRecordLayout {
    /// Records carry a 2-byte creation index (object header flags bit 2).
    tracked: bool,
    /// Dense attribute storage indexes that order (object header flags bit 3).
    indexed: bool,
}

/// Object header flags bit 2: message creation order is tracked, so every
/// message record carries a creation index.
const OH_FLAG_CREATION_ORDER_TRACKED: u8 = 0x04;

/// Object header flags bit 3: attribute creation order is *indexed* as well as
/// tracked, so dense attribute storage carries a creation-order B-tree beside
/// its name index.
const OH_FLAG_CREATION_ORDER_INDEXED: u8 = 0x08;

/// Object header flags bit 4: the header prefix carries the attribute
/// phase-change thresholds (`H5O_HDR_ATTR_STORE_PHASE_CHANGE`).
const OH_FLAG_ATTR_PHASE_CHANGE: u8 = 0x10;

/// Object header flags bit 5: the header prefix carries the four access,
/// modification, change and birth timestamps (`H5O_HDR_STORE_TIMES`).
const OH_FLAG_STORE_TIMES: u8 = 0x20;

/// The four timestamps a version 2 object header stores when
/// [`OH_FLAG_STORE_TIMES`] is set, each 4 bytes of seconds since the Unix epoch
/// and stored in this order.
///
/// The reference C library stores them on **every** version 2 header it writes:
/// `H5O_CRT_OHDR_FLAGS_DEF` is `H5O_HDR_STORE_TIMES`, so a header from libhdf5,
/// h5py or netCDF-4 carries all four, and `H5Oget_info` reads them straight out
/// of this block.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) struct ObjectTimes {
    access: u32,
    modification: u32,
    change: u32,
    birth: u32,
}

impl ObjectTimes {
    /// Bytes this block occupies in a header prefix.
    const LEN: usize = 16;

    /// Parse the block at `at`. The caller must have checked that 16 bytes are
    /// available there.
    fn parse(prefix: &[u8], at: usize) -> Self {
        let field =
            |i: usize| u32::from_le_bytes(prefix[at + 4 * i..at + 4 * i + 4].try_into().unwrap());
        Self {
            access: field(0),
            modification: field(1),
            change: field(2),
            birth: field(3),
        }
    }

    /// The block's on-disk bytes.
    fn to_bytes(self) -> [u8; Self::LEN] {
        let mut out = [0u8; Self::LEN];
        out[0..4].copy_from_slice(&self.access.to_le_bytes());
        out[4..8].copy_from_slice(&self.modification.to_le_bytes());
        out[8..12].copy_from_slice(&self.change.to_le_bytes());
        out[12..16].copy_from_slice(&self.birth.to_le_bytes());
        out
    }

    /// The same times with the modification and change times moved to `now`.
    /// The access and birth times are the object's own history and are left
    /// where they were.
    ///
    /// The reference C library's `H5O_touch_oh` reaches the same two-of-four
    /// shape by a different pair: on a version 2 header it writes
    /// `oh->atime = oh->ctime = now` and carries a source comment saying the
    /// modification time still needs code to update. A rewrite is a
    /// modification, and it is not an *access*, so this writes the field that
    /// says so; both agree on the change time, and neither disturbs the birth
    /// time.
    fn touched(self, now: u32) -> Self {
        Self {
            modification: now,
            change: now,
            ..self
        }
    }
}

/// The attribute phase-change thresholds a version 2 object header stores when
/// [`OH_FLAG_ATTR_PHASE_CHANGE`] is set: `H5Pset_attr_phase_change`'s maximum
/// number of attributes kept compact (in the header) and minimum kept dense (in
/// a fractal heap).
///
/// The reference C library writes this block only when the pair differs from its
/// defaults of 8 and 6, so most headers carry no such block at all. Preserved
/// verbatim: this editor's own compact/dense decision still uses
/// [`MAX_COMPACT_ATTRS`], so a non-default pair survives a rewrite without yet
/// steering it.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) struct AttrPhaseChange {
    max_compact: u16,
    min_dense: u16,
}

impl AttrPhaseChange {
    /// Bytes this block occupies in a header prefix.
    const LEN: usize = 4;

    /// Parse the block at `at`. The caller must have checked that 4 bytes are
    /// available there.
    fn parse(prefix: &[u8], at: usize) -> Self {
        Self {
            max_compact: u16::from_le_bytes(prefix[at..at + 2].try_into().unwrap()),
            min_dense: u16::from_le_bytes(prefix[at + 2..at + 4].try_into().unwrap()),
        }
    }

    /// The block's on-disk bytes.
    fn to_bytes(self) -> [u8; Self::LEN] {
        let mut out = [0u8; Self::LEN];
        out[0..2].copy_from_slice(&self.max_compact.to_le_bytes());
        out[2..4].copy_from_slice(&self.min_dense.to_le_bytes());
        out
    }
}

/// Everything chunk 0's prefix declares about a version 2 object header: the
/// record layout its messages are written in, and the two optional blocks the
/// prefix itself may carry.
///
/// All of it is a property of the *header*, shared by chunk 0 and every
/// continuation block, and none of it can be re-derived from the message bytes —
/// so it travels beside them ([`OhRegion`]) from the parse right through to the
/// rebuild. A rewrite that dropped the optional blocks would silently zero every
/// timestamp `H5Oget_info` reports on a file the C library wrote, and reset the
/// phase-change thresholds a caller set with `H5Pset_attr_phase_change`.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub(crate) struct OhHeaderProps {
    /// How wide a message record prefix is, and whether creation order is
    /// tracked and indexed.
    layout: OhRecordLayout,
    /// The access/modification/change/birth block, where the header stores one.
    times: Option<ObjectTimes>,
    /// The compact/dense attribute thresholds, where the header stores them.
    attr_phase_change: Option<AttrPhaseChange>,
}

impl OhHeaderProps {
    /// 4-byte records, no creation order, and neither optional block: what this
    /// crate's whole-file writer emits, and what a header this editor creates
    /// from nothing uses.
    pub(crate) const PLAIN: Self = Self::with_layout(OhRecordLayout::PLAIN);

    /// A header in `layout` carrying neither optional block.
    pub(crate) const fn with_layout(layout: OhRecordLayout) -> Self {
        Self {
            layout,
            times: None,
            attr_phase_change: None,
        }
    }

    /// The object-header flag bits these properties imply, above the two size
    /// bits the emitter chooses from the region's length.
    const fn header_flags(self) -> u8 {
        let times = if self.times.is_some() {
            OH_FLAG_STORE_TIMES
        } else {
            0
        };
        let phase = if self.attr_phase_change.is_some() {
            OH_FLAG_ATTR_PHASE_CHANGE
        } else {
            0
        };
        self.layout.header_flags() | times | phase
    }

    /// Bytes the optional blocks add to the header prefix.
    const fn optional_len(self) -> usize {
        let times = if self.times.is_some() {
            ObjectTimes::LEN
        } else {
            0
        };
        let phase = if self.attr_phase_change.is_some() {
            AttrPhaseChange::LEN
        } else {
            0
        };
        times + phase
    }
}

impl OhRecordLayout {
    /// 4-byte record prefixes and no creation order at all: what this crate's
    /// whole-file writer emits, and what a header this editor creates from
    /// nothing uses.
    pub(crate) const PLAIN: Self = Self {
        tracked: false,
        indexed: false,
    };

    /// The layout a version 2 object header's flags byte declares.
    pub(crate) const fn from_header_flags(flags: u8) -> Self {
        Self {
            tracked: flags & OH_FLAG_CREATION_ORDER_TRACKED != 0,
            indexed: flags & OH_FLAG_CREATION_ORDER_INDEXED != 0,
        }
    }

    /// Bytes a message record spends before its body.
    pub(crate) const fn prefix_len(self) -> usize {
        if self.tracked { 6 } else { 4 }
    }

    /// The object-header flag bits this layout implies.
    const fn header_flags(self) -> u8 {
        let tracked = if self.tracked {
            OH_FLAG_CREATION_ORDER_TRACKED
        } else {
            0
        };
        let indexed = if self.indexed {
            OH_FLAG_CREATION_ORDER_INDEXED
        } else {
            0
        };
        tracked | indexed
    }

    /// Whether records carry a creation index at all.
    pub(crate) const fn tracks_creation_order(self) -> bool {
        self.tracked
    }

    /// Whether dense attribute storage on this object indexes that order, with
    /// a creation-order B-tree beside the name index.
    pub(crate) const fn indexes_creation_order(self) -> bool {
        self.indexed
    }

    /// Parse the message record at `p` within a chunk's message region,
    /// returning `(message type, body start, body end)`; the next record begins
    /// at `body end`. Returns `Ok(None)` once fewer bytes remain than a record
    /// prefix takes (a clean end of the region, or the gap the reference C
    /// library leaves when a chunk's free space is too small to hold a message),
    /// and `Err` if a record's declared body runs past the region. Centralizes
    /// the bounds check shared by every walker.
    pub(crate) fn next_message(
        self,
        region: &[u8],
        p: usize,
    ) -> Result<Option<(MessageType, usize, usize)>, Error> {
        if p + self.prefix_len() > region.len() {
            return Ok(None);
        }
        let msg_type = MessageType::from_u16(region[p] as u16);
        let msg_size = u16::from_le_bytes([region[p + 1], region[p + 2]]) as usize;
        let body = p + self.prefix_len();
        let body_end = body + msg_size;
        if body_end > region.len() {
            return Err(Error::EditUnsupported("malformed object header message"));
        }
        Ok(Some((msg_type, body, body_end)))
    }

    /// The creation index the record at `msg_start` carries, or `None` where the
    /// layout has no such field. The caller must have located `msg_start` with
    /// [`next_message`](Self::next_message), which bounds the read.
    fn creation_index(self, region: &[u8], msg_start: usize) -> Option<u16> {
        self.tracked
            .then(|| u16::from_le_bytes([region[msg_start + 4], region[msg_start + 5]]))
    }

    /// Encode one message record: this layout's prefix, then `body`.
    ///
    /// `creation_index` is written only where the layout carries one. It is
    /// meaningful for an Attribute message, whose creation index *is* the
    /// attribute's creation order; the reference C library writes zero on every
    /// other message type, which is what [`Self::record`] passes.
    fn record_with_creation_index(
        self,
        msg_type: MessageType,
        body: &[u8],
        creation_index: u16,
    ) -> Vec<u8> {
        let mut m = Vec::with_capacity(self.prefix_len() + body.len());
        #[expect(
            clippy::cast_possible_truncation,
            reason = "message type ids are a small enum that fits the 1-byte v2 type field"
        )]
        m.push(msg_type.to_u16() as u8);
        #[expect(
            clippy::cast_possible_truncation,
            reason = "callers pass bodies that fit the 2-byte message-size field (see doc comment)"
        )]
        m.extend_from_slice(&(body.len() as u16).to_le_bytes());
        m.push(MessageFlags::NONE.get()); // message flags
        if self.tracks_creation_order() {
            m.extend_from_slice(&creation_index.to_le_bytes());
        }
        m.extend_from_slice(body);
        m
    }

    /// Encode one message record whose creation index, if the layout has one, is
    /// the zero the reference C library writes for every non-attribute message.
    fn record(self, msg_type: MessageType, body: &[u8]) -> Vec<u8> {
        self.record_with_creation_index(msg_type, body, 0)
    }
}

/// A version 2 object header's message records, in the layout the header
/// declares them in.
///
/// The editor's model of a header is one contiguous run of message records:
/// [`WriteEngine::gather_oh_messages`] collapses a multi-chunk header into it,
/// the rewriters copy it message by message, and
/// [`build_v2_object_header`] wraps it back up. Every one of those has to agree
/// with the header's own flags about how wide a record prefix is, so the layout
/// travels with the bytes rather than being re-derived — or assumed — at each
/// step.
#[derive(Clone, Debug, Default)]
pub(crate) struct OhRegion {
    bytes: Vec<u8>,
    props: OhHeaderProps,
}

impl core::ops::Deref for OhRegion {
    type Target = [u8];
    fn deref(&self) -> &[u8] {
        &self.bytes
    }
}

impl OhRegion {
    /// A region of `bytes` belonging to a header with these `props`.
    pub(crate) fn new(bytes: Vec<u8>, props: OhHeaderProps) -> Self {
        Self { bytes, props }
    }

    /// An empty region for a header with these `props`, to be filled record by
    /// record.
    fn empty(props: OhHeaderProps) -> Self {
        Self::new(Vec::new(), props)
    }

    /// The same header's properties over different bytes: what every rewriter
    /// that copies a region message by message returns.
    fn with_bytes(&self, bytes: Vec<u8>) -> Self {
        Self::new(bytes, self.props)
    }

    pub(crate) fn layout(&self) -> OhRecordLayout {
        self.props.layout
    }

    /// Everything the header's prefix declares, which a rebuild re-emits.
    pub(crate) fn props(&self) -> OhHeaderProps {
        self.props
    }

    /// [`OhRecordLayout::next_message`] over this region's own bytes.
    pub(crate) fn next_message(
        &self,
        p: usize,
    ) -> Result<Option<(MessageType, usize, usize)>, Error> {
        self.props.layout.next_message(&self.bytes, p)
    }

    /// The creation index of the record at `msg_start`, or `None` for a header
    /// that does not track it.
    fn creation_index(&self, msg_start: usize) -> Option<u16> {
        self.props.layout.creation_index(&self.bytes, msg_start)
    }

    /// Append one already-encoded record (or run of records) written in this
    /// region's layout.
    fn push_bytes(&mut self, record: &[u8]) {
        self.bytes.extend_from_slice(record);
    }

    /// The bytes, for a patch that changes a field inside a record without
    /// changing any record's length.
    fn bytes_mut(&mut self) -> &mut [u8] {
        &mut self.bytes
    }

    /// Append a hard Link message record for `name -> addr`, carrying
    /// `creation_order` where the group tracks link creation order and `None`
    /// where it does not (see [`LinkCreationOrder`], which is what decides
    /// which).
    fn push_link(&mut self, name: &str, addr: StoredAddress, creation_order: Option<u64>) {
        let record = encode_link_message(self.props.layout, name, addr, creation_order);
        self.push_bytes(&record);
    }

    /// Append a message record for `body`, with the zero creation index the
    /// reference C library writes for every non-attribute message.
    fn push(&mut self, msg_type: MessageType, body: &[u8]) {
        let record = self.props.layout.record(msg_type, body);
        self.push_bytes(&record);
    }

    /// Append a message whose body is a *reference* to the one copy of it stored
    /// elsewhere, rather than the message itself.
    ///
    /// The flag is what tells every reader to follow the body instead of
    /// decoding it, so it travels with the bytes: a rewrite that dropped it
    /// would leave a reference to be read as content.
    fn push_shared(&mut self, msg_type: MessageType, reference: &[u8]) {
        let mut record = self.props.layout.record(msg_type, reference);
        record[3] = MessageFlags::SHARED.get();
        self.push_bytes(&record);
    }
}

/// Refuse to copy a dataset whose element bytes live in files outside this one
/// (`H5Pset_external`, the External Data Files header message, type 7), which
/// this crate does not follow (issue #336).
///
/// Such a dataset carries a *contiguous* layout message with an undefined data
/// address, the same encoding a never-written dataset uses — and
/// [`read_copy_subtree`](WriteEngine::read_copy_subtree) copies that encoding as
/// the storage it does not have. The layout alone cannot tell the two apart, so
/// naming this message is the only thing standing between a dataset that holds
/// data and a copy carrying its schema and none of it. [`crate::repack()`]
/// rejects the same shape for the same reason.
///
/// Applied on both copy paths rather than only the cross-file one: an in-file
/// copy would reproduce the header without its data just as readily. It is also
/// the only screen that reads this message at all — [`reject_foreign_addresses`]
/// inspects shared messages, datatypes, and attributes, not this body, which
/// carries a local-heap address that would dangle in another file.
fn reject_external_storage(region: &OhRegion) -> Result<(), Error> {
    let mut p = 0;
    while let Some((msg_type, _, body_end)) = region.next_message(p)? {
        if msg_type == MessageType::EXTERNAL_DATA_FILES {
            return Err(Error::EditUnsupported(
                "a dataset stores its elements in external files (H5Pset_external), \
                 which a copy cannot reproduce -- its data lives in files this crate \
                 does not read",
            ));
        }
        p = body_end;
    }
    Ok(())
}

/// Refuse to copy an object whose header embeds a *source-file* absolute address
/// that a verbatim copy into another file cannot translate. An in-file copy keeps
/// these valid by sharing the source file's heaps and objects; a cross-file copy
/// cannot. Three things qualify:
///
/// - a **variable-length** datatype, whose element bytes are global-heap
///   references (collection address + index) into the source file's heap;
/// - a **reference** datatype (object or dataset-region), whose element bytes are
///   absolute object addresses in the source file;
/// - any **shared message** ([`MessageFlags::SHARED`] set): a committed datatype, but also a shared
///   dataspace, fill value, or filter-pipeline message, whose body is a reference into the source
///   file's shared-message storage.
///
/// The scan covers a copied object's whole message region (a dataset's or a
/// group's): it refuses any shared message outright, and inspects Datatype
/// messages (the element type) and Attribute messages (their own datatype),
/// recursing through compound members, array elements, and enumeration bases so a
/// nested variable-length or reference occurrence is caught too. It is applied
/// only on the cross-file path; the same-file [`copy`](crate::File::copy)
/// deliberately keeps these forms (their addresses stay valid in one file).
fn reject_foreign_addresses(region: &OhRegion) -> Result<(), Error> {
    let mut p = 0;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        // A *shared* message stores, in place of its real body, a reference into
        // the source file's shared-message storage — an object-header address or a
        // fractal-heap (SOHM) id — which means nothing in another file. This
        // catches committed (shared) datatypes and shared attributes as well as a
        // shared dataspace, fill value, or filter-pipeline message, all of which
        // HDF5 may place in the shared-message table. Refuse any of them, whatever
        // the message type. The flags byte is the 4th of the record header (type,
        // size, flags); `next_message` returning `Some` guarantees
        // `p + 4 <= region.len()`.
        if MessageFlags::new(region[p + 3]).is_shared() {
            return Err(Error::EditUnsupported(
                "a shared (committed/SOHM) object-header message cannot be copied to another file yet",
            ));
        }
        match msg_type {
            MessageType::DATATYPE => {
                let (dt, _) = Datatype::parse(&region[body..body_end])?;
                if datatype_holds_file_address(&dt) {
                    return Err(Error::EditUnsupported(
                        "variable-length or reference datasets cannot be copied to another file yet",
                    ));
                }
            }
            MessageType::ATTRIBUTE => {
                // An attribute's *own* datatype or dataspace field can be a
                // reference to a committed message, which the record's shared
                // flag above does not report: that flag describes the attribute
                // message, not the fields inside it. The reference addresses the
                // source file, so it cannot travel any more than a shared record
                // can — and the parse below cannot resolve it here in any case.
                if crate::attribute::message_shares_a_field(&region[body..body_end]) {
                    return Err(Error::EditUnsupported(
                        "an attribute with a committed (shared) datatype cannot be copied to another file yet",
                    ));
                }
                let attr = crate::attribute::AttributeMessage::parse(
                    &region[body..body_end],
                    LENGTH_SIZE,
                )?;
                if datatype_holds_file_address(&attr.datatype) {
                    return Err(Error::EditUnsupported(
                        "variable-length or reference attributes cannot be copied to another file yet",
                    ));
                }
            }
            _ => {}
        }
        p = body_end;
    }
    Ok(())
}

/// Cross-file screen for a dense (fractal-heap) attribute set. The bytes parsed
/// out of the source heap can embed source-file absolute addresses just as inline
/// attribute messages can — variable-length (global-heap) or reference attribute
/// data — which would dangle in another file. [`reject_foreign_addresses`] screens
/// the verbatim object-header region but not heap-resident attribute bytes, so a
/// dense attribute set is screened here instead. Same-file copies skip this (their
/// addresses stay valid); the fresh heap built on write is same-file by
/// construction, so only the source datatypes matter.
fn reject_foreign_dense_attrs(attrs: &[crate::attribute::AttributeMessage]) -> Result<(), Error> {
    for attr in attrs {
        if datatype_holds_file_address(&attr.datatype) {
            return Err(Error::EditUnsupported(
                "variable-length or reference dense (fractal-heap) attributes cannot be copied to another file yet",
            ));
        }
    }
    Ok(())
}

/// Re-encode a shared-message reference in the modern form, for a message being
/// rewrapped from a version 1 object header into a version 2 one.
///
/// A version 1 header carries the oldest reference encoding: a symbol-table
/// entry, with the object-header address buried behind a local-heap address.
/// Copying those bytes into a version 2 header would put an encoding there that
/// no writer of such a header has produced since the format gained one; parsing
/// and re-encoding gives the shape every current reader and writer uses.
///
/// Which shape that is depends on where the message lives. An object header is
/// named by the version 2 form this crate already writes for a committed
/// datatype. The shared-message heap has no encoding before version 3, so a heap
/// reference gets that one. Both name exactly what the version 1 reference named,
/// so the message keeps its single copy and its reference count is unchanged.
fn modernize_shared_reference(
    body: &[u8],
    offset_size: u8,
    length_size: u8,
) -> Result<Vec<u8>, Error> {
    let reference = crate::shared_message::parse_shared_ref(body, offset_size, length_size)?;
    Ok(match reference.location {
        crate::shared_message::SharedLocation::ObjectHeader(addr) => {
            crate::shared_message::encode_committed_ref(addr, offset_size)
        }
        crate::shared_message::SharedLocation::SohmHeap(id) => {
            crate::shared_message::encode_sohm_ref(&id)
        }
    })
}

/// Refuse a commit that removes or moves an object header the file's
/// shared-message (SOHM) index names.
///
/// A heap-stored record names no object, so it is not screened here: what a
/// commit does to it is a *reference count* that goes stale, which the engine
/// refuses at the point it would change one (see [`SHARED_ATTRIBUTE_MESSAGE`]).
/// This screens the other record shape, whose address would dangle.
fn screen_shared_message_records(
    records: &[crate::sohm::SohmRecord],
    invalidated: &InvalidatedAddresses,
) -> Result<(), Error> {
    for record in records {
        if let crate::sohm::SohmLocation::ObjectHeader { address, .. } = record.location
            && invalidated.refusal(address).is_some()
        {
            return Err(Error::EditUnsupported(
                SHARED_MESSAGE_INDEX_NAMES_A_MOVED_OBJECT,
            ));
        }
    }
    Ok(())
}

/// Refuse a staged dataset whose element bytes already hold *resolved* object
/// references naming space this commit reclaims (issue #317).
///
/// This is the address-side half of the rule
/// [`WriteEngine::resolve_reference_target`] enforces on the path side. A target
/// named as a path is screened there, by name; a target supplied as an address —
/// `DatasetBuilder::with_reference_data`, or `with_raw_data` over a datatype that
/// holds a reference — never reaches that function at all, and before this screen
/// existed it was written straight through to disk.
///
/// Element bytes still carrying placeholders are unaffected: an unresolved slot
/// holds zero, which [`InvalidatedAddresses::refusal`] passes, and the target
/// that replaces it is screened by `resolve_reference_target` instead — by name
/// for a path, by address for a raw one. So this runs over every staged
/// dataset's `raw` without asking which builder filled it: the rule is about the
/// bytes, not the door they came through.
fn screen_resolved_references(
    dt: &Datatype,
    raw: &[u8],
    invalidated: &InvalidatedAddresses,
) -> Result<(), Error> {
    if !datatype_holds_object_address(dt) {
        return Ok(());
    }
    // The datatype declares an object reference, so the walker must find one.
    // Two ways it does not: slots that do not fit the element size (`None`), and
    // a reference `embedded_reference_slots` does not map — it locates the
    // 8-byte object reference only, and reports any other width as no slots at
    // all. Both mean the addresses cannot be read, and an empty list taken for
    // "nothing to check" would wave through exactly the datatype
    // `datatype_holds_object_address` had just recognised. Neither shape is
    // one this crate builds; refuse rather than write references past a screen
    // that could not see them.
    let Some(slots) = embedded_reference_slots(dt).filter(|slots| !slots.is_empty()) else {
        // Unconditional, unlike the per-address checks below. Those refuse an
        // address that lands somewhere this commit vacates; this one cannot read
        // the addresses at all, so there is nothing to compare against and no
        // amount of screening would help. Every commit that reaches here rebuilds
        // at least the root group, so `for_supplied.moved` is never empty and
        // gating this would have changed nothing anyway. A commit whose only
        // staged edit is a same-length in-place overwrite never reaches here at
        // all — it takes the fast path above, which vacates nothing. An in-file
        // copy reaches this only beside a deletion, because it is screened
        // against `for_copied`, whose `moved` is empty by construction.
        //
        // The `filter` is belt-and-braces: `embedded_reference_slots` already
        // reports an unaddressable reference as `None` rather than as an empty
        // list, and reading an empty list as "nothing to check" is exactly the
        // hole this screen was found to have.
        return Err(Error::EditUnsupported(
            "a reference this commit writes sits in a datatype whose addresses this screen \
             cannot read — a width other than eight, a dataset-region reference, a \
             variable length of them, or a compound holding one beside a reference it can \
             read — so it cannot be checked against what this commit vacates; supply an \
             8-byte object reference (`with_reference_data`)",
        ));
    };
    // Non-empty slots mean the datatype has room for at least one 8-byte
    // address, so the element size is at least 8.
    for (_, stored) in stored_object_references(raw, dt.type_size() as usize, &slots) {
        if let Some(refusal) = invalidated.refusal(StoredAddress::new(stored)) {
            return Err(Error::EditUnsupported(refusal));
        }
    }
    Ok(())
}

/// Screen an in-file copy's subtree against the space this commit reclaims
/// (issue #317).
///
/// An in-file copy re-emits its source's element bytes verbatim, which is what
/// keeps a copied variable-length or reference dataset valid: the addresses
/// still name the same file, so nothing has to be rewritten. A deletion in the
/// same commit takes that away for object references alone — a global heap
/// collection is never reclaimed by a delete, so variable-length elements keep
/// pointing at data that is still there.
///
/// Every element that can be read is screened by address, through the same
/// [`screen_resolved_references`] a staged dataset's bytes go through: a
/// contiguous dataset's data block, a compact dataset's inline data, and every
/// attribute, inline or dense. A committed (shared) datatype is resolved through
/// `src` first, so a copy of an object with a named type is screened like any
/// other rather than refused for carrying a type this could not read.
///
/// One form cannot be read at all and so is refused by *datatype*, and only when
/// it holds an object reference: a **chunked** dataset, whose addresses sit
/// inside chunks this path carries compressed and never decodes — the same
/// obstacle that makes [`crate::repack()`] reject a chunked object-reference
/// dataset outright.
///
/// `src` is the session's image framed at its base address, the view a stored
/// (base-relative) shared-message address indexes directly.
fn screen_copied_references(
    tree: &CopyTree,
    invalidated: &InvalidatedAddresses,
    src: &(impl Source + ?Sized),
) -> Result<(), Error> {
    if invalidated.is_empty() {
        return Ok(());
    }
    use crate::shared_message::SharedResolver as _;
    // No shared-message table: a heap-stored message is refused rather than
    // followed here, and this screen treats an unreadable datatype as one it
    // cannot clear, which is the conservative answer.
    let resolver = crate::shared_message::SourceResolver::new(
        src,
        AccessMode::ReadWrite,
        OFFSET_SIZE,
        LENGTH_SIZE,
        None,
    );
    let (region, dense_attrs) = match tree {
        CopyTree::DatasetVerbatim {
            region,
            dense_attrs,
        }
        | CopyTree::DatasetContiguous {
            region,
            dense_attrs,
            ..
        }
        | CopyTree::DatasetChunked {
            region,
            dense_attrs,
            ..
        }
        | CopyTree::Group {
            non_link_region: region,
            dense_attrs,
            ..
        } => (region, dense_attrs),
    };
    for attr in &dense_attrs.attrs {
        screen_resolved_references(&attr.datatype, &attr.raw_data, invalidated)?;
    }

    // One walk of the header: the object's own element datatype, the compact
    // data the layout message may carry, and every inline attribute.
    let mut element_dt: Option<Datatype> = None;
    let mut compact: Option<Vec<u8>> = None;
    let mut p = 0;
    while let Some((msg_type, body, body_end)) = region.next_message(p)? {
        // The flags byte is the 4th of the record header (type, size, flags);
        // `next_message` returning `Some` guarantees it is in bounds.
        let shared = MessageFlags::new(region[p + 3]).is_shared();
        match msg_type {
            MessageType::DATATYPE => {
                // A committed datatype's message body is a pointer into the
                // file's shared-message storage rather than an encoded type, so
                // read the type it names before parsing.
                let committed;
                let encoded = if shared {
                    committed = resolver
                        .resolve(&region[body..body_end], MessageType::DATATYPE)
                        .map_err(|source| Error::CopyScreenUnreadable {
                            message: MessageType::DATATYPE,
                            source,
                        })?;
                    &committed[..]
                } else {
                    &region[body..body_end]
                };
                let (dt, _) = Datatype::parse(encoded)?;
                element_dt = Some(dt);
            }
            MessageType::DATA_LAYOUT => {
                if let Ok(DataLayout::Compact { data }) =
                    DataLayout::parse(&region[body..body_end], OFFSET_SIZE, LENGTH_SIZE)
                {
                    compact = Some(data);
                }
            }
            MessageType::ATTRIBUTE => {
                // A *shared record* is the whole attribute message held in the
                // file's shared-message table, which is a different indirection
                // from the committed datatype `parse_resolving` follows inside
                // the fields — and a rare one this path has never modelled. Its
                // elements cannot be reached here, so it is refused.
                if shared {
                    return Err(Error::EditUnsupported(
                        "a copy in this commit carries a shared (SOHM) attribute message, whose \
                         elements cannot be screened against the same commit's deletions; use \
                         separate commits",
                    ));
                }
                // `parse_resolving` rather than `parse`: an attribute's own
                // datatype field can name a committed message, which the record's
                // shared flag does not report — that flag describes the attribute
                // message, not the fields inside it — and `parse` refuses one.
                let attr = crate::attribute::AttributeMessage::parse_resolving(
                    &region[body..body_end],
                    LENGTH_SIZE,
                    &resolver,
                )
                .map_err(|source| Error::CopyScreenUnreadable {
                    message: MessageType::ATTRIBUTE,
                    source,
                })?;
                screen_resolved_references(&attr.datatype, &attr.raw_data, invalidated)?;
            }
            _ => {}
        }
        p = body_end;
    }

    match tree {
        // Compact: the elements are inline in the data-layout message. A layout
        // that did not yield them leaves a reference datatype unscreened, so it
        // is refused for the same reason a chunked one is.
        CopyTree::DatasetVerbatim { .. } => match (&element_dt, &compact) {
            (Some(dt), Some(data)) => screen_resolved_references(dt, data, invalidated)?,
            (Some(dt), None) if datatype_holds_object_address(dt) => {
                return Err(Error::EditUnsupported(
                    "a compact object-reference dataset's elements could not be read to screen \
                     them against this commit's deletions; use separate commits",
                ));
            }
            _ => {}
        },
        // No Datatype message means no declared reference, here and in the
        // chunked arm below: such a header is not a dataset any reader can
        // interpret, so there is nothing in it to dangle. That is why only the
        // compact arm above refuses on a missing piece — there the datatype is
        // present and says a reference is in bytes it could not reach.
        CopyTree::DatasetContiguous { data, .. } => {
            // A dataset whose storage the source never allocated (`None`) stores
            // no elements, so it holds no address that could dangle. Skipped
            // rather than screened as an empty run of element bytes: when
            // `screen_resolved_references` cannot map a datatype's addresses — a
            // dataset-region reference, a variable length of references, one
            // wider than eight bytes — it refuses *unconditionally*, before it
            // reads a byte, so an empty run would refuse this dataset for
            // elements it does not have.
            if let (Some(dt), Some(data)) = (&element_dt, data) {
                screen_resolved_references(dt, data, invalidated)?;
            }
        }
        // Chunked: `chunk_bytes` are carried exactly as the source stored them,
        // filters and all, so there is nothing here to decode addresses out of.
        CopyTree::DatasetChunked { .. } => {
            if element_dt
                .as_ref()
                .is_some_and(datatype_holds_object_address)
            {
                return Err(Error::EditUnsupported(
                    "a chunked object-reference dataset cannot be copied in a commit that also \
                     deletes objects: its addresses live inside chunks this path does not \
                     decode; use separate commits",
                ));
            }
        }
        CopyTree::Group { children, .. } => {
            for (_, _, child) in children {
                screen_copied_references(child, invalidated, src)?;
            }
        }
    }
    Ok(())
}

/// Wrap a chunk-0 message region in a fresh single-chunk version 2 object header
/// (`OHDR` prefix + region + Jenkins checksum), first normalizing the region's
/// attribute storage with [`ensure_attribute_info`]. Mirrors the encoding in
/// [`crate::object_header_writer::ObjectHeaderWriter::serialize`].
///
/// The normalization belongs here rather than at the fifteen call sites because
/// carrying an Attribute Info message is a property of a version 2 header holding
/// inline attributes, not of any one edit operation — and a site that forgot it
/// would reintroduce the zero-count defect silently.
///
/// A region that cannot be walked is reported, not asserted away. Every region
/// reaching here should have been built by this crate or already walked
/// message-by-message on the way in, but "should" is a claim about a file this
/// session did not write: a header whose message size field overruns the region
/// is a malformed *file*, which is the caller's input and so takes the
/// [`Error::EditUnsupported`] the region walk reports. A debug assertion here
/// would make the two build profiles disagree about whether such a file is
/// writable at all: a panic in a test build, and in a release build a header
/// silently missing its Attribute Info message, which is the zero-count defect
/// this function exists to prevent.
pub(crate) fn build_v2_object_header(region: &OhRegion) -> Result<Vec<u8>, Error> {
    let mut owned = region.clone();
    ensure_attribute_info(&mut owned)?;
    Ok(build_v2_object_header_verbatim(&owned))
}

/// [`build_v2_object_header`] without the attribute-storage normalization, for
/// the region that function has already normalized.
///
/// **The header's optional prefix blocks are re-emitted, and its timestamps are
/// stamped.** A region carries whatever chunk 0's prefix declared
/// ([`OhHeaderProps`]), so a rewrite of a header from the reference C library,
/// h5py or netCDF-4 puts its four timestamps and any attribute phase-change
/// thresholds back where it found them, with the flag bits that announce them.
///
/// Of those, the **modification and change times are moved to now**: this
/// function runs once per rebuilt header, and every rebuild is a modification.
/// Access and birth times are the object's own history and are copied verbatim
/// ([`ObjectTimes::touched`] says how that compares with `H5O_touch_oh`).
/// Under `no_std` there is no clock to read, so all four are preserved as they
/// were rather than zeroed — a stale modification time being the honest reading
/// of "this build cannot tell the time", where a zero would claim the epoch.
/// A header this crate creates from nothing stores no times at all, so nothing
/// here applies to it.
fn build_v2_object_header_verbatim(region: &OhRegion) -> Vec<u8> {
    let total = region.len();
    let width = UintWidth::smallest_for_len(total);
    let props = region.props();
    // The creation-order bits and the two optional-block bits are the header's
    // own claim about what follows, so they come from the properties the region
    // was parsed with.
    let flags = width.flag_bits() | props.header_flags();
    let mut buf = Vec::with_capacity(8 + props.optional_len() + total + 4);
    buf.extend_from_slice(b"OHDR");
    buf.push(2); // version
    buf.push(flags);
    if let Some(times) = props.times {
        let stamped = match unix_time_now() {
            Some(now) => times.touched(now),
            None => times,
        };
        buf.extend_from_slice(&stamped.to_bytes());
    }
    if let Some(phase) = props.attr_phase_change {
        buf.extend_from_slice(&phase.to_bytes());
    }
    buf.extend_from_slice(&(total as u64).to_le_bytes()[..usize::from(width.get())]);
    buf.extend_from_slice(region);
    let checksum = jenkins_lookup3(&buf);
    buf.extend_from_slice(&checksum.to_le_bytes());
    buf
}

/// Seconds since the Unix epoch, for the object-header timestamps a rebuild
/// stamps, or `None` where this build has no clock.
///
/// The crate carries no other wall-clock reader: every other date it writes
/// comes from its caller. `no_std` builds have no `SystemTime` at all and return
/// `None`, and so does a `std` build whose clock is set before 1970 — neither is
/// a reason to fail a commit, so both leave the header's stored times alone (see
/// [`build_v2_object_header_verbatim`]). The 4-byte field saturates rather than
/// wrapping, which matters only past 2106.
#[cfg(feature = "std")]
fn unix_time_now() -> Option<u32> {
    std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .ok()
        .map(|d| u32::try_from(d.as_secs()).unwrap_or(u32::MAX))
}

/// No wall clock outside `std`; see the `std` definition.
#[cfg(not(feature = "std"))]
fn unix_time_now() -> Option<u32> {
    None
}

/// Read a little-endian unsigned integer of `bytes.len()` (≤ 8) bytes.
#[expect(
    clippy::cast_possible_truncation,
    reason = "callers parse in-file sizes/offsets bounded by the in-memory image; downstream \
              slicing is length-checked, so a malformed oversized field errors rather than reads OOB"
)]
fn read_le(bytes: &[u8]) -> usize {
    let mut v = 0u64;
    for (i, &b) in bytes.iter().enumerate() {
        v |= (b as u64) << (8 * i);
    }
    v as usize
}

/// The engine as the target a [`reference_patch::Plan`] is applied to
/// (issue #324).
///
/// Deliberately the whole engine rather than the image alone: an object header
/// is republished as one checksummed write, so applying a plan reads as well as
/// writes, and both have to go through the same image the rest of the commit
/// used — the same write-gathering, the same pending-write overlay, the same
/// end-of-file bound.
///
/// [`reference_patch::Plan`]: crate::reference_patch::Plan
impl crate::reference_patch::PatchTarget for WriteEngine {
    fn read(&self, at: u64, len: usize) -> Result<Vec<u8>, Error> {
        self.image().read_exact_at(at, len).map_err(Error::Format)
    }
    fn write(&mut self, at: u64, bytes: &[u8]) -> Result<(), Error> {
        self.image.write_at(at, bytes)
    }
}

#[cfg(test)]
mod tests {
    use rstest::rstest;

    use super::*;
    use crate::{CompoundMember, datatype::layout::FloatingPointLayout, object_path::ObjectPath};

    /// The rule that places a chunk index on a paged file: some chunk-data span
    /// abuts it. Both sides count, which is what a repeatedly appended dataset
    /// needs — its blobs leave chunk data above the index as well as below it, so
    /// the highest data address is not the one beside the index (issue #388).
    #[test]
    fn a_chunk_index_is_placed_by_the_chunk_data_that_abuts_it() {
        // Data below, index above: the from-scratch layout.
        assert!(index_abuts_chunk_data(&[(100, 60)], &[(160, 40)]));
        // Index below, data above: a blob placed in a hole under existing chunks.
        assert!(index_abuts_chunk_data(&[(200, 60)], &[(160, 40)]));
        // Data on both sides, with the abutting span not the highest one — the
        // shape every staged append after the first produces.
        assert!(index_abuts_chunk_data(
            &[(100, 60), (900, 60)],
            &[(160, 40)]
        ));
        // A multi-block index is placed as one run by the span beside its edge.
        assert!(index_abuts_chunk_data(
            &[(100, 60)],
            &[(180, 20), (160, 20)]
        ));

        // Neither edge is touched: the reference library's layout, where the index
        // sits in a metadata page far below the chunk data.
        assert!(!index_abuts_chunk_data(&[(9000, 60)], &[(160, 40)]));
        // A gap of one byte is not an abutment.
        assert!(!index_abuts_chunk_data(&[(100, 59)], &[(160, 40)]));
        // No chunk data at all — an empty resizable dataset's eagerly built index
        // — places nothing.
        assert!(!index_abuts_chunk_data(&[], &[(160, 40)]));
        // Nothing to place.
        assert!(index_abuts_chunk_data(&[], &[]));
        // A zero-length data span touches the index without occupying a byte
        // beside it.
        assert!(!index_abuts_chunk_data(&[(160, 0)], &[(160, 40)]));
    }

    /// The screen that keeps the abutment from admitting a reference-library
    /// layout: an index with a byte in page 0 sits beside the superblock, so it is
    /// in a metadata page however its far end lines up with chunk data.
    #[test]
    fn a_chunk_index_in_page_zero_is_never_raw() {
        const PAGE: u64 = 512;
        // The C-written shape this exists to refuse: the index header sits in
        // page 0 behind the superblock, its last block fills page 1 exactly, and
        // the first raw page begins where that block ends. The abutment alone
        // called the whole run raw.
        let c_written = [(48u64, 80u64), (512, 512)];
        assert!(index_abuts_chunk_data(&[(1024, 512)], &c_written));
        assert!(index_touches_page_zero(&c_written, PAGE));
        // An index wholly above page 0 — every one this crate writes, since it
        // places chunk data and the index beside it out of raw pages — is not
        // screened out.
        assert!(!index_touches_page_zero(&[(1024, 200)], PAGE));
        // The boundary: the last byte of page 0 is still page 0.
        assert!(index_touches_page_zero(&[(511, 200)], PAGE));
        assert!(!index_touches_page_zero(&[(512, 200)], PAGE));
        // A zero-length span occupies nothing, in page 0 as anywhere else.
        assert!(!index_touches_page_zero(&[(0, 0)], PAGE));
        // Nothing to screen.
        assert!(!index_touches_page_zero(&[], PAGE));
        // A page size a paged file never has cannot place anything.
        assert!(index_touches_page_zero(&[(1024, 200)], 0));
    }

    /// A page every byte of which is free or dead belongs to no page type, so it
    /// is promoted whole into the raw list — the free-page convention the rest of
    /// the paged allocator already follows. The partial edges keep their type,
    /// since the pages they sit in may still hold something live.
    #[test]
    fn a_page_that_is_wholly_free_or_dead_is_promoted_whole() {
        const PAGE: u64 = 4096;
        let (mut meta, mut raw, mut dead) = (FreeList::new(), FreeList::new(), FreeList::new());
        // Page 1 is half free metadata and half dead, so neither list can show it
        // empty on its own.
        meta.free(PAGE, 2048);
        dead.free(PAGE + 2048, 2048);
        // Page 2 is wholly dead.
        dead.free(2 * PAGE, PAGE);
        // Page 3 keeps something live at each end, so only its middle is free —
        // and not adjacent to page 2, so the promotion is visible on its own.
        raw.free(3 * PAGE + 1024, 1024);
        PagedEdit::promote_whole_free_pages(&mut meta, &mut raw, &mut dead, PAGE);

        assert_eq!(
            raw.sections(),
            [(PAGE, 2 * PAGE), (3 * PAGE + 1024, 1024)],
            "the two empty pages join the raw list as one run; the partial page stays"
        );
        assert!(
            meta.sections().is_empty(),
            "the promoted page left the metadata list"
        );
        assert!(
            dead.sections().is_empty(),
            "every dead byte was inside a promoted page"
        );
    }

    /// Dead space that does not complete a page stays dead: it is not free space,
    /// and handing it out as either page type would mix the page it sits in.
    #[test]
    fn dead_space_short_of_a_whole_page_is_not_promoted() {
        const PAGE: u64 = 4096;
        let (mut meta, mut raw, mut dead) = (FreeList::new(), FreeList::new(), FreeList::new());
        dead.free(PAGE, 512);
        raw.free(PAGE + 512, 1024);
        PagedEdit::promote_whole_free_pages(&mut meta, &mut raw, &mut dead, PAGE);
        assert_eq!(dead.sections(), [(PAGE, 512)]);
        assert_eq!(raw.sections(), [(PAGE + 512, 1024)]);
    }

    /// The shared-message flag is read from the message header's fourth byte, and
    /// the two attribute paths have to read the same one: the compact pass
    /// refuses such a message as it copies, the dense pass before it reads a set
    /// that would resolve it into something indistinguishable from a private
    /// attribute. A wrong offset here reads a *size* byte and refuses or accepts
    /// by accident.
    #[test]
    fn a_shared_attribute_message_is_told_from_a_private_one_by_its_flags() {
        let body = crate::type_builders::build_attr_message("a", &AttrValue::I64(1))
            .serialize(LENGTH_SIZE);
        let private = message_record(MessageType::ATTRIBUTE, &body);
        assert!(!region_has_shared_attr(&plain_region(private.clone())).unwrap());

        let mut shared = private.clone();
        shared[3] = MessageFlags::SHARED.get();
        assert!(region_has_shared_attr(&plain_region(shared)).unwrap());

        // The flag is only read on an Attribute message: the same byte set on a
        // neighbouring message says nothing about attribute storage.
        let mut other = message_record(MessageType::DATASPACE, &body);
        other[3] = MessageFlags::SHARED.get();
        assert!(!region_has_shared_attr(&plain_region(other)).unwrap());
    }

    /// An object-reference attribute named `name` pointing at `address`.
    ///
    /// Built by taking a `u64` attribute — whose value is already the 8 little-
    /// endian bytes an object reference is stored as — and relabelling its
    /// datatype, because no public API stages one: [`AttrValue`] has no
    /// reference variant, so a file carrying such an attribute was written by
    /// the reference C library, and the copy path re-emits its bytes verbatim
    /// like any other attribute's.
    fn reference_attr(name: &str, address: u64) -> crate::attribute::AttributeMessage {
        let mut attr = crate::type_builders::build_attr_message(name, &AttrValue::U64(address));
        attr.datatype = Datatype::Reference {
            size: 8,
            ref_type: crate::datatype::ReferenceType::Object,
        };
        assert_eq!(
            attr.raw_data,
            address.to_le_bytes(),
            "the value is the address"
        );
        attr
    }

    /// Wrap a message body in the object-header record a header region holds it
    /// in: type, body size, flags, body.
    fn message_record(msg_type: MessageType, body: &[u8]) -> Vec<u8> {
        OhRecordLayout::PLAIN.record(msg_type, body)
    }

    /// A region of plain (4-byte-record) messages, the layout every writer in
    /// this crate emits.
    fn plain_region(bytes: Vec<u8>) -> OhRegion {
        OhRegion::new(bytes, OhHeaderProps::PLAIN)
    }

    /// A header region holding one attribute inline.
    fn inline_attr_region(attr: &crate::attribute::AttributeMessage) -> OhRegion {
        plain_region(message_record(
            MessageType::ATTRIBUTE,
            &attr.serialize_v3(LENGTH_SIZE),
        ))
    }

    /// The header region of a one-element *compact* object-reference dataset:
    /// its datatype, and a version 3 compact data-layout message carrying the
    /// element inline (version, class 0, a 2-byte inline size, then the data).
    fn compact_reference_region(address: u64) -> OhRegion {
        let mut region = message_record(
            MessageType::DATATYPE,
            &crate::type_builders::make_object_reference_type().serialize(),
        );
        let mut layout = vec![3u8, 0];
        layout.extend_from_slice(&8u16.to_le_bytes());
        layout.extend_from_slice(&address.to_le_bytes());
        region.extend_from_slice(&message_record(MessageType::DATA_LAYOUT, &layout));
        plain_region(region)
    }

    /// A *compact* dataset keeps its elements inside the data-layout message
    /// rather than in a data block, so the copy screen reads them straight out
    /// of the header region (issue #317).
    ///
    /// Driven from a hand-built region because no writer in this crate emits a
    /// compact layout — the files that carry one come from the reference C
    /// library, which is also why the userblock suite notes that its
    /// compact-layout fixtures have to come from there.
    #[test]
    fn a_copied_compact_reference_dataset_is_screened() {
        let empty = BytesSource::new(Vec::new());
        let invalidated = InvalidatedAddresses {
            removed: vec![(248, 71)],
            moved: Vec::new(),
            base: BaseAddress::ZERO,
        };
        for (address, refused) in [(248u64, true), (318, true), (319, false)] {
            let tree = CopyTree::DatasetVerbatim {
                region: compact_reference_region(address),
                dense_attrs: DenseAttrSet::default(),
            };
            let got = screen_copied_references(&tree, &invalidated, &empty);
            assert_eq!(got.is_err(), refused, "compact element {address}: {got:?}");
        }

        // A region whose layout message yields no inline data leaves a reference
        // datatype unscreened, so it is refused rather than waved through.
        // `read_object` builds this variant only from a compact layout, so this
        // is a header that did not parse as one — malformed input, not a shape
        // the copy path produces.
        let no_layout = CopyTree::DatasetVerbatim {
            region: plain_region(message_record(
                MessageType::DATATYPE,
                &crate::type_builders::make_object_reference_type().serialize(),
            )),
            dense_attrs: DenseAttrSet::default(),
        };
        let err = screen_copied_references(&no_layout, &invalidated, &empty).unwrap_err();
        assert!(
            err.to_string().contains("could not be read to screen"),
            "got: {err}"
        );
    }

    /// The committed reference points at address 248, and the source is empty,
    /// so the four-byte signature read there runs to 252 over 0 bytes.
    #[test]
    fn a_copy_whose_committed_datatype_cannot_be_read_reports_the_read_failure() {
        let empty = BytesSource::new(Vec::new());
        let invalidated = InvalidatedAddresses {
            removed: vec![(248, 71)],
            moved: Vec::new(),
            base: BaseAddress::ZERO,
        };
        let mut bytes = message_record(
            MessageType::DATATYPE,
            &crate::shared_message::encode_committed_ref(StoredAddress::new(248), OFFSET_SIZE),
        );
        bytes[3] = MessageFlags::SHARED.get();
        let tree = CopyTree::DatasetVerbatim {
            region: plain_region(bytes),
            dense_attrs: DenseAttrSet::default(),
        };

        let err = screen_copied_references(&tree, &invalidated, &empty).unwrap_err();
        let Error::CopyScreenUnreadable { message, source } = &err else {
            panic!("expected CopyScreenUnreadable, got {err:?}");
        };
        assert_eq!(*message, MessageType::DATATYPE);
        assert_eq!(
            *source,
            FormatError::UnexpectedEof {
                expected: 252,
                available: 0
            }
        );
    }

    /// A datatype that declares an object reference the element bytes have no
    /// room for cannot be walked, so its addresses cannot be screened — and a
    /// commit that reclaims space refuses it rather than writing references past
    /// a screen that could not read them (issue #317).
    ///
    /// Reachable only from a malformed source header: every datatype this crate
    /// builds sizes its element to its members. Driven directly for that reason.
    #[test]
    fn a_datatype_whose_reference_slots_do_not_fit_is_refused() {
        use crate::datatype::{CompoundMember, ReferenceType};
        // An 8-byte compound declaring an 8-byte reference at offset 4: the slot
        // runs four bytes past the element.
        let dt = Datatype::Compound {
            size: 8,
            members: vec![CompoundMember {
                name: "r".to_string(),
                byte_offset: 4,
                datatype: Datatype::Reference {
                    size: 8,
                    ref_type: ReferenceType::Object,
                },
            }],
        };
        assert!(
            embedded_reference_slots(&dt).is_none(),
            "the fixture must be one the walker cannot map"
        );

        let raw = [0u8; 8];
        let invalidated = InvalidatedAddresses {
            removed: vec![(248, 71)],
            moved: Vec::new(),
            base: BaseAddress::ZERO,
        };
        let err = screen_resolved_references(&dt, &raw, &invalidated).unwrap_err();
        assert!(
            err.to_string().contains("this screen cannot read"),
            "got: {err}"
        );

        // Unconditional: a screen holding nothing still refuses, because the
        // refusal is about addresses that cannot be read rather than about what
        // the commit is vacating. `moved` is never empty in a real commit, so
        // there is no case this could have been made conditional on.
        let nothing = InvalidatedAddresses {
            removed: Vec::new(),
            moved: Vec::new(),
            base: BaseAddress::ZERO,
        };
        assert!(screen_resolved_references(&dt, &raw, &nothing).is_err());
    }

    /// A variable-length *of object references* is refused rather than skipped:
    /// its addresses live in the global heap the elements point at, not in the
    /// element bytes this screen reads (issue #317).
    ///
    /// The reference C library writes such a datatype (`H5T_VLEN` of
    /// `H5T_STD_REF_OBJ`) and an in-file copy carries it, so it reaches the
    /// screen through `copy` even though nothing here builds one. A
    /// variable-length *string* is unaffected — the heap it points at is never
    /// reclaimed by a delete, so its elements stay valid.
    #[test]
    fn a_variable_length_of_object_references_is_refused() {
        use crate::datatype::{CharacterSet, ReferenceType};
        let of_references = Datatype::VariableLength {
            is_string: false,
            padding: None,
            charset: None,
            base_type: Box::new(Datatype::Reference {
                size: 8,
                ref_type: ReferenceType::Object,
            }),
        };
        let of_strings = crate::type_builders::make_vlen_string_type(CharacterSet::Utf8);
        let raw = vec![0u8; 32];
        let invalidated = InvalidatedAddresses {
            removed: vec![(248, 71)],
            moved: Vec::new(),
            base: BaseAddress::ZERO,
        };

        let err = screen_resolved_references(&of_references, &raw, &invalidated).unwrap_err();
        assert!(
            err.to_string().contains("this screen cannot read"),
            "got: {err}"
        );
        assert!(
            screen_resolved_references(&of_strings, &raw, &invalidated).is_ok(),
            "a variable-length string points at a heap no delete reclaims"
        );
    }

    /// An object reference wider than the 8 bytes the slot walker maps is
    /// refused, not skipped (issue #317).
    ///
    /// [`datatype_holds_object_address`] answers for an object reference of
    /// *any* width, while [`embedded_reference_slots`] locates only the 8-byte
    /// one — so the walker reports a width it cannot address as unwalkable
    /// rather than as "no slots here", which a screen would read as "nothing to
    /// check". `Dataset::dereference` reads such an element, taking the address
    /// from its first eight bytes, so the reference is live.
    #[test]
    fn an_object_reference_wider_than_eight_bytes_is_refused() {
        use crate::datatype::ReferenceType;
        let dt = Datatype::Reference {
            size: 16,
            ref_type: ReferenceType::Object,
        };
        assert!(
            embedded_reference_slots(&dt).is_none(),
            "the walker must report a width it cannot map, not an empty slot list"
        );

        let mut raw = vec![0u8; 16];
        raw[..8].copy_from_slice(&300u64.to_le_bytes());
        let invalidated = InvalidatedAddresses {
            removed: vec![(248, 71)],
            moved: Vec::new(),
            base: BaseAddress::ZERO,
        };
        let err = screen_resolved_references(&dt, &raw, &invalidated).unwrap_err();
        assert!(
            err.to_string().contains("this screen cannot read"),
            "got: {err}"
        );

        // Unconditional, for the reason
        // `a_datatype_whose_reference_slots_do_not_fit_is_refused` states.
        let nothing = InvalidatedAddresses {
            removed: Vec::new(),
            moved: Vec::new(),
            base: BaseAddress::ZERO,
        };
        assert!(screen_resolved_references(&dt, &raw, &nothing).is_err());
    }

    /// A reference target supplied as a *raw address* is screened exactly as one
    /// supplied as a path (issue #317), and the address it would have resolved
    /// to is returned unchanged when it names space this commit keeps.
    ///
    /// Driven directly because nothing stages a `Raw` target carrying a real
    /// address: a builder reachable from a session produces only
    /// [`ObjectRefTarget::Path`], and [`crate::repack`]'s faithful re-emit
    /// resolves every real target to a `Path`, leaving `Raw` for the null and
    /// undefined references alone. The arm exists so that the two ways a target
    /// can name an object answer to the same rule rather than to whichever one
    /// a caller happened to use, and this is what holds it to that.
    /// A file with nothing a reference could live in is walked once and never
    /// again, and one that holds a reference is walked every time.
    ///
    /// The first half is the whole of `proved_free_of_references`, and it is
    /// invisible from outside the session: a build that never caches the proof
    /// writes byte-identical files and passes every other test here, having
    /// silently gone back to walking the file on every commit. The second half
    /// is what stops a too-eager proof from being the fix for the first.
    #[test]
    fn a_file_proved_free_of_references_is_walked_once_and_no_more() {
        use crate::reference_patch::{reset_walks, walks};
        use tempfile::tempdir;
        let dir = tempdir().unwrap();

        let commit_three = |path: &std::path::Path| {
            let session = crate::File::open_rw(path).unwrap();
            for i in 0..3 {
                session
                    .root()
                    .create_dataset(&format!("added{i}"), |b| {
                        b.with_i32_data(&[i]);
                    })
                    .unwrap();
                session.commit().unwrap();
            }
        };

        let plain = dir.path().join("plain.h5");
        let mut b = crate::writer::FileBuilder::new();
        b.create_dataset("d").with_i32_data(&[1, 2, 3]);
        b.write(&plain).unwrap();
        reset_walks();
        commit_three(&plain);
        assert_eq!(
            walks(),
            1,
            "the first commit's walk proves the file reference-free; the rest \
             must take its word for it"
        );

        let referencing = dir.path().join("referencing.h5");
        let mut b = crate::writer::FileBuilder::new();
        b.create_dataset("d").with_i32_data(&[1, 2, 3]);
        b.create_dataset("refs").with_path_references(&["d"]);
        b.write(&referencing).unwrap();
        reset_walks();
        commit_three(&referencing);
        assert_eq!(
            walks(),
            3,
            "a file that holds a reference is never proved free of one, so every \
             commit walks it"
        );
    }

    #[test]
    fn a_raw_reference_target_is_screened_like_a_path_one() {
        use tempfile::tempdir;
        let dir = tempdir().unwrap();
        let path = dir.path().join("raw_target.h5");
        let mut b = crate::writer::FileBuilder::new();
        b.create_dataset("d").with_i32_data(&[1, 2, 3]);
        b.write(&path).unwrap();
        let engine = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();

        let nodes: BTreeMap<ObjectPathBuf, Node> = BTreeMap::new();
        let path_addr: BTreeMap<ObjectPathBuf, u64> = BTreeMap::new();
        let resolve = |address: u64, removed: Vec<(u64, u64)>, moved: Vec<u64>| {
            WriteEngine::resolve_reference_target(
                &ObjectRefTarget::Raw(StoredAddress::new(address)),
                &path_addr,
                &nodes,
                &[],
                &[],
                &[],
                &InvalidatedAddresses {
                    removed,
                    moved,
                    base: BaseAddress::ZERO,
                },
                &engine.image(),
                engine.superblock(),
            )
        };

        assert!(
            resolve(300, vec![(248, 71)], Vec::new()).is_err(),
            "an address inside a reclaimed span is refused"
        );
        assert!(
            resolve(300, Vec::new(), vec![300]).is_err(),
            "an address this commit rewrites elsewhere is refused"
        );
        assert_eq!(
            resolve(300, vec![(400, 71)], vec![299, 301]).unwrap(),
            StoredAddress::new(300),
            "an address outside every reclaimed span and every moved header is carried through"
        );
        assert_eq!(
            resolve(300, Vec::new(), Vec::new()).unwrap(),
            StoredAddress::new(300),
            "a commit that reclaims nothing screens nothing"
        );
        // The two sentinels name no object, so they are carried through even
        // when they fall inside a reclaimed span.
        assert_eq!(
            resolve(0, vec![(0, 4096)], vec![0]).unwrap(),
            StoredAddress::new(0)
        );
        assert_eq!(
            resolve(UNDEF, vec![(0, u64::MAX)], vec![UNDEF]).unwrap(),
            StoredAddress::new(UNDEF)
        );
    }

    #[rstest]
    #[case("a/b")]
    #[case("/a/b")]
    #[case("a/./b")]
    #[case("a//b/")]
    fn a_reference_target_is_keyed_by_the_object_its_spelling_names(#[case] spelling: &str) {
        use tempfile::tempdir;
        let dir = tempdir().unwrap();
        let path = dir.path().join("path_target.h5");
        let mut b = crate::writer::FileBuilder::new();
        b.create_dataset("d").with_i32_data(&[1, 2, 3]);
        b.write(&path).unwrap();
        let engine = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();

        let nodes: BTreeMap<ObjectPathBuf, Node> = BTreeMap::new();
        let resolve = |path_addr: &BTreeMap<ObjectPathBuf, u64>, add_targets: &[ObjectPathBuf]| {
            WriteEngine::resolve_reference_target(
                &ObjectRefTarget::Path(ObjectPathBuf::parse(spelling)),
                path_addr,
                &nodes,
                add_targets,
                &[],
                &[],
                &InvalidatedAddresses {
                    removed: Vec::new(),
                    moved: Vec::new(),
                    base: BaseAddress::ZERO,
                },
                &engine.image(),
                engine.superblock(),
            )
        };

        let placed: BTreeMap<ObjectPathBuf, u64> =
            std::iter::once((ObjectPathBuf::parse("a/b"), 4096)).collect();
        assert_eq!(resolve(&placed, &[]).unwrap(), StoredAddress::new(4096));

        let err = resolve(&BTreeMap::new(), &[ObjectPathBuf::parse("a/b")]).unwrap_err();
        let Error::EditUnsupported(reason) = &err else {
            panic!("expected EditUnsupported, got {err:?}");
        };
        assert_eq!(
            *reason,
            "an object-reference dataset targets a path this commit is still writing; \
             use separate commits"
        );
    }

    /// A copied object's *attributes* are screened against the space the commit
    /// reclaims, in either storage an object can hold them in — inline in the
    /// header region, or in the fractal heap a dense object uses — and an
    /// address outside those spans passes in both (issue #317).
    #[test]
    fn a_copied_reference_attribute_is_screened_in_both_storages() {
        let empty = BytesSource::new(Vec::new());
        let invalidated = InvalidatedAddresses {
            removed: vec![(248, 71)],
            moved: Vec::new(),
            base: BaseAddress::ZERO,
        };
        // 248 is the first byte of the reclaimed span, 318 its last, 319 the
        // byte after it.
        for (address, refused) in [(248u64, true), (318, true), (319, false), (247, false)] {
            let attr = reference_attr("target", address);
            let dense = CopyTree::DatasetVerbatim {
                region: OhRegion::default(),
                dense_attrs: DenseAttrSet {
                    attrs: vec![attr.clone()],
                    creation: DenseAttrCreationOrder::Untracked,
                },
            };
            let inline = CopyTree::DatasetVerbatim {
                region: inline_attr_region(&attr),
                dense_attrs: DenseAttrSet::default(),
            };
            for (storage, tree) in [("dense", &dense), ("inline", &inline)] {
                let got = screen_copied_references(tree, &invalidated, &empty);
                assert_eq!(
                    got.is_err(),
                    refused,
                    "{storage} attribute at {address}: {got:?}"
                );
            }
        }
    }

    /// Collect the message types present in a chunk-0 region, in order.
    fn region_types(region: &OhRegion) -> Vec<MessageType> {
        let mut out = Vec::new();
        let mut p = 0;
        while let Some((mt, _, end)) = region.next_message(p).unwrap() {
            out.push(mt);
            p = end;
        }
        out
    }

    /// Stopping an in-place append (`append_inplace`) at any phase boundary must
    /// leave the file readable as a consistent prefix — the old length until the
    /// phase-4 dimension commit, the new length after it — even though a
    /// partial-tail append repoints the visible trailing element in place. Mirrors
    /// `Dataset::append`'s crash-consistency harness, but driven through
    /// the in-place edit engine's own mirror (disk-before-mirror ordering) to prove the shared
    /// engine is crash-safe under both owners. Three starting layouts: the trailing
    /// element inline in the index block (chunk 4, n 6), in a data block
    /// (chunk 2, n 9, slot 0), and a *filtered* inline one (issue #393), whose
    /// repointed element is the three-field record rather than a bare address and
    /// whose relocated chunk has to decode to the old prefix at every phase
    /// before the dimension moves.
    #[test]
    fn append_inplace_crash_consistency_partial_tail_prefix() {
        use crate::reader::File as PureFile;
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let build = |path: &std::path::Path, n: i32, chunk: u64, deflate: bool| {
            let data: Vec<i32> = (0..n).collect();
            let mut b = FileBuilder::new();
            let d = b
                .create_dataset("d")
                .with_i32_data(&data)
                .with_shape(&[n as u64])
                .with_maxshape(&[MaxExtent::Unlimited])
                .with_chunks(&[chunk]);
            if deflate {
                d.with_deflate(6);
            }
            b.write(path).unwrap();
        };

        for (n, chunk, add, deflate) in [
            (6i32, 4u64, 5i32, false),
            (9, 2, 6, false),
            (6, 4, 5, true),
            (9, 4, 3, true),
        ] {
            let dir = tempdir().unwrap();
            let base = dir.path().join("base.h5");
            build(&base, n, chunk, deflate);

            for max_phase in 1u8..=4 {
                let p = dir
                    .path()
                    .join(format!("crash_{n}_{chunk}_{deflate}_{max_phase}.h5"));
                std::fs::copy(&base, &p).unwrap();
                {
                    let mut s = WriteEngine::open_with_locking(&p, FileLocking::Enabled).unwrap();
                    s.append_inplace_i32_phased("d", &(n..n + add).collect::<Vec<_>>(), max_phase)
                        .unwrap();
                    // session dropped here, simulating a crash after `max_phase`
                }
                let expected_len = if max_phase == 4 { n + add } else { n };
                let f = PureFile::from_bytes(std::fs::read(&p).unwrap()).unwrap();
                assert_eq!(
                    f.dataset("d").unwrap().read_i32().unwrap(),
                    (0..expected_len).collect::<Vec<_>>(),
                    "inconsistent view after crash at phase {max_phase} (n={n}, chunk={chunk}, \
                     deflate={deflate})"
                );
            }
        }
    }

    /// Growing a partial trailing chunk decodes and re-encodes elements that are
    /// already committed, so a **lossy** pipeline is refused on the immediate
    /// in-place path, with the file left exactly as it was.
    ///
    /// The values are the point rather than the error: measured on this fixture
    /// before the refusal existed, the committed pair `[4.5, 5.5]` read back as
    /// `[16.0, 0.0]` once two more elements joined their ZFP block. That is
    /// quantization following the block's new contents, not a decode that lost
    /// its place — a *fresh* chunk of `[99.0, 98.0, 97.0]` at this rate reads
    /// back as `[98.0, 102.0, 90.0]` with nothing decoded anywhere, which is what
    /// eight bits per value buys and is not this test's business.
    #[cfg(feature = "zfp")]
    #[test]
    fn append_onto_a_lossy_partial_tail_is_refused() {
        use crate::reader::File as PureFile;
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let committed: Vec<f64> = vec![0.5, 1.5, 2.5, 3.5, 4.5, 5.5];

        // `d` is 6 of a chunk of 4: a partial trailing chunk holding 4.5 and 5.5.
        let build = |path: &std::path::Path| {
            let mut b = FileBuilder::new();
            b.create_dataset("d")
                .with_f64_data(&committed)
                .with_shape(&[6])
                .with_maxshape(&[MaxExtent::Unlimited])
                .with_chunks(&[4])
                .with_zfp(8.0);
            b.write(path).unwrap();
        };

        // The immediate in-place path.
        let p = dir.path().join("inplace.h5");
        build(&p);
        let before = std::fs::read(&p).unwrap();
        {
            let f = crate::reader::File::open_rw(&p).unwrap();
            let err = f
                .dataset("d")
                .unwrap()
                .append(&[99.0f64, 98.0, 97.0])
                .expect_err("a lossy pipeline must not have its trailing chunk re-encoded");
            assert!(
                matches!(&err, Error::AppendInPlaceUnsupported(m) if m.contains("lossy")),
                "got: {err:?}"
            );
        }
        assert_eq!(
            std::fs::read(&p).unwrap(),
            before,
            "the refusal wrote bytes"
        );
        assert_eq!(
            PureFile::open(&p)
                .unwrap()
                .dataset("d")
                .unwrap()
                .read_f64()
                .unwrap(),
            committed
        );

        // A chunk-aligned start needs no rewrite, so it is still accepted and the
        // committed values are still exactly what they were.
        let p = dir.path().join("aligned.h5");
        {
            let mut b = FileBuilder::new();
            b.create_dataset("d")
                .with_f64_data(&committed[..4])
                .with_shape(&[4])
                .with_maxshape(&[MaxExtent::Unlimited])
                .with_chunks(&[4])
                .with_zfp(8.0);
            b.write(&p).unwrap();
        }
        {
            let f = crate::reader::File::open_rw(&p).unwrap();
            f.dataset("d").unwrap().append(&[9.5f64, 8.5, 7.5]).unwrap();
        }
        let back = PureFile::open(&p)
            .unwrap()
            .dataset("d")
            .unwrap()
            .read_f64()
            .unwrap();
        assert_eq!(back[..4], committed[..4], "an untouched chunk changed");

        // The staged path rebuilds the index at commit and rewrote the same chunk
        // on the way; it refuses at the call that stages the append, so nothing is
        // staged and the file is left alone.
        let p = dir.path().join("staged.h5");
        build(&p);
        let before = std::fs::read(&p).unwrap();
        {
            let f = crate::reader::File::open_rw(&p).unwrap();
            let err = f
                .dataset("d")
                .unwrap()
                .append_staged(|b| {
                    b.append_f64(&[99.0, 98.0, 97.0]);
                })
                .expect_err("a staged append must not re-encode the committed tail either");
            assert!(
                matches!(&err, Error::AppendUnsupported(m) if m.contains("lossy")),
                "got: {err:?}"
            );
            f.close().unwrap();
        }
        assert_eq!(
            std::fs::read(&p).unwrap(),
            before,
            "the refusal wrote bytes"
        );

        // A buffered appender over the unaligned shape is refused when it is
        // made, since its very first write would be the rewrite above.
        let p = dir.path().join("buffered.h5");
        build(&p);
        {
            let f = crate::reader::File::open_rw(&p).unwrap();
            let mut ds = f.dataset("d").unwrap();
            let err = ds
                .buffered_appender()
                .expect_err("its first write would re-encode the committed tail");
            assert!(
                matches!(&err, Error::AppendInPlaceUnsupported(m) if m.contains("lossy")),
                "got: {err:?}"
            );
        }
    }

    /// Float D-scale scale-offset is the other lossy mode, and is refused for the
    /// same reason — while the **integer** mode, which is lossless, grows its
    /// partial trailing chunk with every value intact. Both halves matter: a
    /// predicate that refused all of scale-offset would pass the first assertion
    /// and take a working case away.
    #[test]
    fn a_lossy_scale_offset_mode_is_refused_and_a_lossless_one_is_not() {
        use crate::reader::File as PureFile;
        use crate::scaleoffset::ScaleOffset;
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();

        let p = dir.path().join("dscale.h5");
        {
            let mut b = FileBuilder::new();
            b.create_dataset("d")
                .with_f64_data(&[0.5, 1.5, 2.5, 3.5, 4.5, 5.5])
                .with_shape(&[6])
                .with_maxshape(&[MaxExtent::Unlimited])
                .with_chunks(&[4])
                .with_scale_offset(ScaleOffset::FloatDScale(1));
            b.write(&p).unwrap();
        }
        {
            let f = crate::reader::File::open_rw(&p).unwrap();
            let err = f
                .dataset("d")
                .unwrap()
                .append(&[99.0f64, 98.0, 97.0])
                .expect_err("float D-scale is lossy");
            assert!(
                matches!(&err, Error::AppendInPlaceUnsupported(m) if m.contains("lossy")),
                "got: {err:?}"
            );
        }

        let p = dir.path().join("int.h5");
        {
            let mut b = FileBuilder::new();
            b.create_dataset("d")
                .with_i32_data(&(0..6).collect::<Vec<_>>())
                .with_shape(&[6])
                .with_maxshape(&[MaxExtent::Unlimited])
                .with_chunks(&[4])
                .with_scale_offset(ScaleOffset::Integer(0));
            b.write(&p).unwrap();
        }
        {
            let f = crate::reader::File::open_rw(&p).unwrap();
            let mut ds = f.dataset("d").unwrap();
            ds.append(&[6i32, 7, 8]).unwrap();
            ds.append(&[9i32]).unwrap();
        }
        assert_eq!(
            PureFile::open(&p)
                .unwrap()
                .dataset("d")
                .unwrap()
                .read_i32()
                .unwrap(),
            (0..10).collect::<Vec<_>>()
        );
    }

    /// A *filtered* append whose length is not a whole number of chunks writes a
    /// partial last chunk. That chunk's index element is a fresh insert past the
    /// old dimension — the same position every whole chunk beside it occupies —
    /// so stopping anywhere in the durability sequence must still read back as
    /// the old prefix, exactly as an aligned filtered append does. This is the
    /// case the in-place refusal used to cover and no longer does; without the
    /// phase sweep, "it reads back fine" would only be testing phase 4.
    #[test]
    fn append_inplace_crash_consistency_filtered_partial_last_chunk() {
        use crate::reader::File as PureFile;
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let build = |path: &std::path::Path, n: i32, chunk: u64| {
            let data: Vec<i32> = (0..n).collect();
            let mut b = FileBuilder::new();
            b.create_dataset("d")
                .with_i32_data(&data)
                .with_shape(&[n as u64])
                .with_maxshape(&[MaxExtent::Unlimited])
                .with_chunks(&[chunk])
                .with_shuffle()
                .with_deflate(4);
            b.write(path).unwrap();
        };

        // Aligned starts, unaligned lengths: one that stays inside a single new
        // chunk, and one that spans several and ends partway through the last.
        // (An unaligned *start* is the sibling sweep,
        // `append_inplace_crash_consistency_partial_tail_prefix`.)
        for (n, chunk, add) in [(8i32, 4u64, 2i32), (8, 4, 9), (0, 4, 3)] {
            let dir = tempdir().unwrap();
            let base = dir.path().join("base.h5");
            build(&base, n, chunk);

            for max_phase in 1u8..=4 {
                let p = dir
                    .path()
                    .join(format!("crash_f_{n}_{chunk}_{max_phase}.h5"));
                std::fs::copy(&base, &p).unwrap();
                {
                    let mut s = WriteEngine::open_with_locking(&p, FileLocking::Enabled).unwrap();
                    s.append_inplace_i32_phased("d", &(n..n + add).collect::<Vec<_>>(), max_phase)
                        .unwrap();
                    // session dropped here, simulating a crash after `max_phase`
                }
                let expected_len = if max_phase == 4 { n + add } else { n };
                let f = PureFile::from_bytes(std::fs::read(&p).unwrap()).unwrap();
                assert_eq!(
                    f.dataset("d").unwrap().read_i32().unwrap(),
                    (0..expected_len).collect::<Vec<_>>(),
                    "inconsistent view after crash at phase {max_phase} (n={n}, chunk={chunk}, \
                     add={add})"
                );
            }
        }
    }

    /// Build a one-element-per-chunk unlimited `d` holding `0..n`, the shape the
    /// crash-consistency harnesses below grow.
    fn build_unit_chunked(path: &std::path::Path, n: i32) {
        use crate::writer::FileBuilder;
        let data: Vec<i32> = (0..n).collect();
        let mut b = FileBuilder::new();
        b.create_dataset("d")
            .with_i32_data(&data)
            .with_shape(&[n as u64])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[1]);
        b.write(path).unwrap();
    }

    /// Stop an append after `max_phase` durability phases and hand back the
    /// resulting file. Dropping the engine inside is the simulated crash: no
    /// further phases run and no close barrier is written.
    fn append_stopped_at(
        base: &std::path::Path,
        out: &std::path::Path,
        values: std::ops::Range<i32>,
        max_phase: u8,
    ) {
        std::fs::copy(base, out).unwrap();
        let mut s = WriteEngine::open_with_locking(out, FileLocking::Enabled).unwrap();
        s.append_inplace_i32_phased("d", &values.collect::<Vec<_>>(), max_phase)
            .unwrap();
    }

    /// Growing an Extensible-Array index across its inline -> direct-block ->
    /// super-block boundaries touches far more index structure than the
    /// partial-tail case above. Stopping at any phase boundary must still read
    /// back as a consistent prefix.
    ///
    /// Restores coverage lost with the deprecated `SwmrWriter` (issue #202); the
    /// owned path drives the same `apply_ea_append` engine.
    #[test]
    fn append_inplace_crash_consistency_across_ea_boundaries() {
        use crate::reader::File as PureFile;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let base = dir.path().join("base.h5");
        let (n, target) = (50i32, 250i32);
        build_unit_chunked(&base, n);

        for max_phase in 1u8..=4 {
            let p = dir.path().join(format!("crash_ea_{max_phase}.h5"));
            append_stopped_at(&base, &p, n..target, max_phase);
            let expected_len = if max_phase == 4 { target } else { n };
            let f = PureFile::from_bytes(std::fs::read(&p).unwrap()).unwrap();
            assert_eq!(
                f.dataset("d").unwrap().read_i32().unwrap(),
                (0..expected_len).collect::<Vec<_>>(),
                "inconsistent view after crash at phase {max_phase}"
            );
        }
    }

    /// The same guarantee for an append that crosses the paged-data-block
    /// boundary (~131,060 chunks), where phase 1 allocates a paged super block,
    /// paged data blocks, the per-page checksums, and the page-init bitmap. This
    /// is the most intricate in-place growth the engine performs, and truncating
    /// it partway is exactly what a power loss does.
    ///
    /// Restores coverage lost with the deprecated `SwmrWriter` (issue #202).
    /// Opening a paged persisting file must seed each free section into the list
    /// for the page type its *slot* names, not one derived from its size.
    ///
    /// The managers a paged file uses mean different things — SUPER (slot 0) is
    /// metadata, DRAW (slot 2) is raw — and a freed section's size says nothing
    /// about which it came from. Getting this wrong is invisible from the outside:
    /// the total free space is unchanged, the reference library still opens the
    /// file, and only a later allocation drawn from the wrong list would mix a
    /// page. So assert the routing directly.
    #[test]
    fn paged_open_seeds_each_manager_by_slot() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("paged_seed.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("d")
            .with_i32_data(&(0..1000).collect::<Vec<i32>>())
            .with_shape(&[1000]);
        b.with_file_space_strategy(FileSpaceStrategy::Page, true, 0)
            .with_file_space_page_size(4096);
        b.write(&path).unwrap();

        // Read the file's recorded free space *before* opening the session: the
        // session holds an exclusive OS lock, and on Windows those locks are
        // mandatory, so a concurrent `File::open` would fail outright.
        let on_disk: u64 = crate::reader::File::open(&path)
            .unwrap()
            .persisted_free_space()
            .iter()
            .map(|&(_, l)| l)
            .sum();

        let s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        let pg = s.paged.as_ref().expect("a paged file installs paged state");
        assert_eq!(pg.page_size, 4096);

        // The from-scratch writer leaves a page tail free in both the metadata and
        // the raw pages, so both per-type managers are populated. If every slot
        // were funnelled into one list, one of these would be empty.
        assert!(
            !pg.meta.sections().is_empty(),
            "SUPER (slot 0) sections seed the metadata list"
        );
        assert!(
            !pg.raw.sections().is_empty(),
            "DRAW (slot 2) sections seed the raw list, not the metadata list"
        );

        // Nothing is double-counted or dropped: the lists partition exactly the
        // free space the file records, and no two sections overlap. A file this
        // crate wrote has nothing unclassified — it only ever files a whole
        // aligned page under the generic-large manager — so summing it in is a
        // statement about that too.
        let mut all = pg.reusable_sections();
        all.extend(pg.unclassified.sections());
        assert!(
            pg.unclassified.sections().is_empty(),
            "our own paged writer files nothing whose page type is unknown"
        );
        let flat: u64 = all.iter().map(|&(_, l)| l).sum();
        assert_eq!(
            flat, on_disk,
            "the split lists hold exactly the file's free space"
        );
        all.sort_by_key(|&(a, _)| a);
        let mut prev_end = 0u64;
        for (addr, len) in all {
            assert!(addr >= prev_end, "the per-type lists do not overlap");
            prev_end = addr + len;
        }
    }

    /// Deleting a chunked dataset from a paged file the *reference library* wrote
    /// must not offer its chunk index for raw reuse.
    ///
    /// A chunk index is metadata by the format's taxonomy, and the C library
    /// allocates one accordingly — out of metadata pages, which on a small file
    /// means page 0, alongside the superblock, the root group, and every other
    /// object's header. This crate places its own indexes in raw pages beside the
    /// chunk data instead, and the reclaim path used to assume that of every file.
    /// Freeing a C-written index under that assumption puts its bytes on the raw
    /// list, and the next commit writes a dataset's values into the middle of a
    /// live metadata page.
    ///
    /// Both libraries still read such a file — every address in it is still
    /// correct — so nothing downstream reports the damage. The assertion has to be
    /// about placement itself: no byte this commit writes may land in a page that
    /// held live metadata beforehand.
    #[test]
    // The file is a committed one the reference C library wrote, see
    // `crates/crosscheck/tests/main/c_test_data.rs`.
    fn a_c_written_chunk_index_is_not_reclaimed_as_raw() {
        use tempfile::tempdir;

        const PAGE: u64 = 4096;
        let dir = tempdir().unwrap();
        let path = crate::test_data::copy("c/paged_index.h5", dir.path());

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        let page_size = s.paged.as_ref().expect("a paged file").page_size;
        assert_eq!(page_size, PAGE);
        // Where the C library put the index: every page it touches is a metadata
        // page, since the C library allocates an index as metadata.
        let victim_addr = crate::group_v2::resolve_path_any(
            s.image.as_slice().unwrap(),
            AccessMode::ReadWrite,
            &s.superblock,
            &ObjectPath::parse("victim"),
        )
        .unwrap();
        let index_spans = s
            .chunked_index_spans(victim_addr)
            .expect("the C library's extensible-array index is enumerable");
        assert!(!index_spans.is_empty());
        // Page 0 always holds the superblock and the root group header; the index
        // pages are what this test is really about.
        let mut meta_pages: Vec<u64> = vec![0];
        for (addr, len) in &index_spans {
            for p in (addr / PAGE)..=((addr + len - 1) / PAGE) {
                meta_pages.push(p);
            }
        }
        meta_pages.sort_unstable();
        meta_pages.dedup();

        // Delete the dataset, then write a small dataset that would fit in the
        // index's freed bytes.
        s.delete(&ObjectPathBuf::parse("/victim")).unwrap();
        s.commit().unwrap();
        let mut db = crate::type_builders::DatasetBuilder::new("added");
        db.with_f64_data(&[2.5f64; 8]).with_shape(&[8]);
        s.stage_created_dataset(&ObjectPathBuf::parse("/added"), db)
            .unwrap();
        s.commit().unwrap();

        // Release the session's exclusive OS lock before reading the file back;
        // those locks are mandatory on Windows.
        drop(s);

        // Every raw byte of the new dataset must be outside those pages.
        let f = crate::reader::File::open(&path).unwrap();
        let added = f.dataset("added").unwrap();
        let crate::Layout::Contiguous {
            address: Some(addr),
            size,
        } = added.layout().unwrap()
        else {
            panic!("a small f64 dataset is stored contiguously");
        };
        for p in (addr / PAGE)..=((addr + size - 1) / PAGE) {
            assert!(
                !meta_pages.contains(&p),
                "the new dataset's raw data landed at ({addr}, {size}), in page {p}, \
                 which held live metadata before the edit: {meta_pages:?}"
            );
        }

        assert_eq!(added.read_f64().unwrap(), vec![2.5f64; 8]);
        assert_eq!(
            f.dataset("keep").unwrap().read_i32().unwrap(),
            vec![1, 2, 3, 4]
        );
    }

    /// Padding a tail page whose type is unknown must leave the padding
    /// untracked, not guess a type for it.
    ///
    /// A commit pads the file to a page boundary before laying down its manager
    /// blocks. When this session has made no typed allocation, the tail page is
    /// one a previous session left non-aligned — only a crash does that, since a
    /// clean close pads — and nothing says what it holds. Recording the padding
    /// under a guess would advertise those bytes for reuse of that type, and half
    /// the time they sit in a page of the other one. Under-reporting is the safe
    /// direction, and it is the call [`PagedEdit::begin`] already makes for the
    /// same situation.
    ///
    /// Reuse is what made this reachable: before it, every commit appended at
    /// least the root group's header, so the tail type was always known by the
    /// time the padding ran.
    #[test]
    fn padding_a_tail_page_of_unknown_type_records_nothing() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        const PAGE: u64 = 4096;
        let dir = tempdir().unwrap();

        let build = |path: &std::path::Path| {
            let mut b = FileBuilder::new();
            b.create_dataset("d")
                .with_i32_data(&(0..1000).collect::<Vec<i32>>())
                .with_shape(&[1000]);
            b.with_file_space_strategy(FileSpaceStrategy::Page, true, 0)
                .with_file_space_page_size(PAGE);
            b.write(path).unwrap();
        };

        // Unknown tail type: pad, but record nothing.
        let unknown = dir.path().join("unknown_tail.h5");
        build(&unknown);
        let mut s = WriteEngine::open_with_locking(&unknown, FileLocking::Enabled).unwrap();
        s.append(&[0u8; 100]).unwrap(); // leaves the image non-page-aligned
        assert!(s.paged.as_ref().unwrap().last.is_none());
        s.pad_to_page().unwrap();
        let pg = s.paged.as_ref().unwrap();
        assert_eq!(s.image.len() % PAGE, 0, "the file is padded to a page");
        assert!(
            pg.meta_pad.is_empty() && pg.raw_pad.is_empty(),
            "padding a tail page of unknown type must claim no page type"
        );
        drop(s);

        // The control: a known tail type is recorded, so the test above is about
        // the unknown case and not about padding never being tracked at all.
        let known = dir.path().join("known_tail.h5");
        build(&known);
        let mut s = WriteEngine::open_with_locking(&known, FileLocking::Enabled).unwrap();
        s.begin_page(PageType::Meta).unwrap();
        s.append(&[0u8; 100]).unwrap();
        s.pad_to_page().unwrap();
        let pg = s.paged.as_ref().unwrap();
        assert_eq!(
            pg.meta_pad.len(),
            1,
            "a known metadata tail records its padding as metadata free space"
        );
        assert!(pg.raw_pad.is_empty());
    }

    /// The reference C library's generic-large manager holds free space of *both*
    /// page types, so a section from it is only reusable when it covers whole
    /// aligned pages.
    ///
    /// `H5F_MEM_PAGE_GENERIC` is aliased to `H5F_MEM_PAGE_LARGE_SUPER` and
    /// commented in the C library's own header as *"large-sized generic: meta and
    /// raw"*. Under paged aggregation on a contiguous-address driver — the default
    /// — `H5MF__alloc_to_fs_type` sends **every** allocation of a page or more
    /// there whatever its type, and `H5MF__alloc_pagefs` then records that
    /// allocation's page-alignment tail as a free section in the same manager. The
    /// tail is smaller than a page and sits in a page whose earlier bytes are the
    /// live object, so filing it as raw and handing it to chunk data would put raw
    /// bytes inside a metadata page.
    ///
    /// This measures that the C library really does write such a section — the
    /// assertion is worthless if it does not — and then pins that this engine
    /// keeps it out of reach: it is recorded, so a rewrite gives it back to the
    /// manager it came from, but never offered to an allocation.
    #[test]
    // The file is a committed one the reference C library wrote, see
    // `crates/crosscheck/tests/main/c_test_data.rs`.
    fn a_generic_large_section_is_only_reusable_as_whole_pages() {
        use tempfile::tempdir;

        const PAGE: u64 = 512;
        let dir = tempdir().unwrap();
        let path = crate::test_data::copy("c/generic_large.h5", dir.path());

        // The premise: at least one section in the generic-large manager (slot 6)
        // is a sub-page fragment. Read the managers slot by slot, since the
        // flattened public view cannot say which manager a section came from.
        let opened = crate::reader::File::open(&path).unwrap();
        let info = opened.file_space_info().expect("a persisting file").clone();
        drop(opened);
        assert_eq!(info.page_size, PAGE);
        let bytes = std::fs::read(&path).unwrap();
        let src = crate::source::BytesSource::new(bytes.as_slice());
        let slot6 = info.manager_addrs[6];
        assert_ne!(slot6, UNDEF, "the C library populated the large manager");
        let (sections, _) = free_space_manager::read_persisted_sections_source(
            &src,
            &[slot6],
            BaseAddress::ZERO,
            8,
        )
        .unwrap();
        let fragments: Vec<(u64, u64)> = sections
            .iter()
            .map(|s| (s.addr.get(), s.size))
            .filter(|&(addr, size)| addr % PAGE != 0 || size % PAGE != 0)
            .collect();
        assert!(
            !fragments.is_empty(),
            "the premise of this test: the C library files sub-page fragments in \
             its generic-large manager, but this file has none ({sections:?})"
        );

        // The behavior: every such fragment is recorded but unreachable, and only
        // whole aligned pages from that manager are placeable.
        let s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        let pg = s.paged.as_ref().expect("a paged file installs paged state");
        let unclassified = pg.unclassified.sections();
        let reusable = pg.reusable_sections();
        for &(addr, size) in &fragments {
            assert!(
                unclassified.contains(&(addr, size)),
                "fragment ({addr}, {size}) must be recorded as unclassified, not lost"
            );
            assert!(
                !reusable
                    .iter()
                    .any(|&(a, l)| a < addr + size && addr < a + l),
                "fragment ({addr}, {size}) must not be offered to any allocation: {reusable:?}"
            );
        }
    }

    /// A paged allocation may be served from the other page type's free list only
    /// where that space covers whole free pages — never from a hole in a page the
    /// other type still occupies.
    ///
    /// This is the allocation-side half of the rule
    /// [`paged_open_seeds_each_manager_by_slot`] pins on the seeding side, and it
    /// is just as invisible from the outside: handing a raw allocation a hole in a
    /// live metadata page puts chunk bytes inside that page, which every reader —
    /// this crate's and the reference library's — resolves correctly, since the
    /// file's addresses all still point where they should. Only the paging
    /// degrades. So assert the choice directly, in both directions, with a
    /// same-type control proving the free region really was big enough to be
    /// taken, and the whole-page case proving the exception is reached rather than
    /// merely permitted.
    #[test]
    fn a_paged_allocation_only_crosses_page_types_over_whole_free_pages() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        const PAGE: u64 = 4096;

        /// A paged session whose free lists hold exactly one region each, of the
        /// page types named. Both regions are interior, so an allocation that
        /// takes one is distinguishable from one that appends.
        fn session(
            path: &std::path::Path,
            meta: Option<(u64, u64)>,
            raw: Option<(u64, u64)>,
        ) -> WriteEngine {
            let mut b = FileBuilder::new();
            b.create_dataset("d")
                .with_i32_data(&(0..4000).collect::<Vec<i32>>())
                .with_shape(&[4000]);
            b.with_file_space_strategy(FileSpaceStrategy::Page, true, 0)
                .with_file_space_page_size(PAGE);
            b.write(path).unwrap();
            let mut s = WriteEngine::open_with_locking(path, FileLocking::Enabled).unwrap();
            let pg = s.paged.as_mut().expect("a paged file installs paged state");
            pg.meta = FreeList::new();
            pg.raw = FreeList::new();
            if let Some((addr, len)) = meta {
                pg.meta.free(addr, len);
            }
            if let Some((addr, len)) = raw {
                pg.raw.free(addr, len);
            }
            s
        }

        let dir = tempdir().unwrap();
        // Two interior regions well inside the file the builder wrote (16 KiB of
        // raw data alone), each a *fragment* of a page whose other bytes are live,
        // so either could physically hold the request and only the page-type rule
        // decides. Deliberately not page-aligned and not a whole page: that is the
        // case the rule forbids outright.
        let hole_a = (PAGE + 512, 2048);
        let hole_b = (2 * PAGE + 512, 2048);

        // Raw request, only a metadata fragment free: appends rather than mixing
        // the page.
        let mut s = session(&dir.path().join("a.h5"), Some(hole_a), None);
        assert!(
            matches!(
                s.reserve(1024, PageType::Raw).unwrap(),
                Placement::Appended { .. }
            ),
            "a raw allocation must not be served out of a live metadata page"
        );
        drop(s);

        // Metadata request, only a raw fragment free: likewise.
        let mut s = session(&dir.path().join("b.h5"), None, Some(hole_b));
        assert!(
            matches!(
                s.reserve(1024, PageType::Meta).unwrap(),
                Placement::Appended { .. }
            ),
            "a metadata allocation must not be served out of a live raw page"
        );
        drop(s);

        // The control: with the matching type free, the same request is reused —
        // so the two refusals above are the page-type rule, not a size failure.
        let mut s = session(&dir.path().join("c.h5"), Some(hole_a), Some(hole_b));
        assert!(
            matches!(
                s.reserve(1024, PageType::Raw).unwrap(),
                Placement::Reused { addr, .. } if addr == hole_b.0
            ),
            "a raw allocation takes the raw hole"
        );
        assert!(
            matches!(
                s.reserve(1024, PageType::Meta).unwrap(),
                Placement::Reused { addr, .. } if addr == hole_a.0
            ),
            "a metadata allocation takes the metadata hole"
        );
        drop(s);

        // The exception, and the whole reason the file stops growing (issue #286):
        // a page with *nothing* in it holds no type to contradict, so either kind
        // may open it. The page is claimed whole and what the request does not use
        // becomes free space of the claiming type.
        let mut s = session(&dir.path().join("d.h5"), Some((PAGE, PAGE)), None);
        assert!(
            matches!(
                s.reserve(1024, PageType::Raw).unwrap(),
                Placement::Reused { addr, .. } if addr == PAGE
            ),
            "a raw allocation may open an empty metadata page"
        );
        let pg = s.paged.as_ref().expect("still paged");
        assert_eq!(
            pg.raw.sections(),
            [(PAGE + 1024, PAGE - 1024)],
            "the rest of the claimed page is free space of the claiming type"
        );
        assert!(
            pg.meta.sections().is_empty(),
            "the page left the list it was claimed from"
        );
        drop(s);

        // And the claim starts at the empty page, not at the free run's own start.
        // A run that spans from mid-page into whole pages beyond it is the common
        // shape — the tail of a live page, then pages nothing is left in — and
        // taking it from the front would put the request in the live page.
        let mut s = session(
            &dir.path().join("e.h5"),
            None,
            Some((PAGE + 512, 3 * PAGE - 512)),
        );
        assert!(
            matches!(
                s.reserve(1024, PageType::Meta).unwrap(),
                Placement::Reused { addr, .. } if addr == 2 * PAGE
            ),
            "the claim must begin at the empty page, not at the run's start"
        );
        // Both edges the claim leaves behind survive it: they are ordinary free
        // space of the type that already held them, and dropping either is the same
        // silent leak this change exists to remove.
        let pg = s.paged.as_ref().expect("still paged");
        assert_eq!(
            pg.raw.sections(),
            [(PAGE + 512, PAGE - 512), (3 * PAGE, PAGE)],
            "the fragment below the claimed page and the page above it both stay free"
        );
        assert_eq!(
            pg.meta.sections(),
            [(2 * PAGE + 1024, PAGE - 1024)],
            "the rest of the claimed page is free space of the claiming type"
        );
    }

    /// Every byte a paged session could still hand out, across both page types.
    fn free_total(s: &WriteEngine) -> u64 {
        let pg = s.paged.as_ref().expect("a paged session");
        pg.meta
            .sections()
            .into_iter()
            .chain(pg.raw.sections())
            .map(|(_, len)| len)
            .sum()
    }

    /// A paged commit's tail removes from the free lists exactly the bytes it goes
    /// on to write — no more, and nothing at all when it declines to place itself.
    ///
    /// This is the invariant the whole change rests on, and the one an integration
    /// test cannot reach: the tail's length and the hole it lands in determine each
    /// other, so which settlement the arithmetic reaches depends on the shape of
    /// the free list, and a file the writer builds only ever produces some of them.
    /// In particular, best fit prefers the *smallest* region that fits, so a hole
    /// whose length is exactly the length the tail proposed is the one it will
    /// choose — and consuming a region outright drops a section from the managers,
    /// making the plan come out *shorter* than the space just reserved for it. A
    /// tail that accepted that would retire the difference from the free list with
    /// nothing recording it, which is issue #286 again, a few bytes at a time.
    ///
    /// Hence a contiguous sweep of hole sizes rather than a chosen one: the length
    /// the tail proposes is a property of the file and its free list, not something
    /// this test should have to know, and the sweep is certain to cross it.
    #[test]
    fn a_paged_tail_takes_exactly_the_space_it_fills() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        const PAGE: u64 = 4096;
        /// Any plausible extension length exercises the same arithmetic, and the
        /// invariant holds for every one of them; the tail's real length is settled
        /// by the commit, which is not what is under test here.
        const EXT_LEN: u64 = 100;

        let dir = tempdir().unwrap();
        let path = dir.path().join("tail_exact.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("d")
            .with_i32_data(&(0..4000).collect::<Vec<i32>>())
            .with_shape(&[4000]);
        b.with_file_space_strategy(FileSpaceStrategy::Page, true, 0)
            .with_file_space_page_size(PAGE);
        b.write(&path).unwrap();
        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        let os = s.superblock.offset_size;

        let mut placed = 0usize;
        for hole in 120..420u64 {
            {
                let pg = s.paged.as_mut().expect("a paged file installs paged state");
                pg.meta = FreeList::new();
                pg.raw = FreeList::new();
                pg.unclassified = FreeList::new();
                pg.meta.free(PAGE, hole);
            }
            let free_before = free_total(&s);
            let layout = s.tail_layout(&[], &[], EXT_LEN, PAGE, os);
            let free_after = free_total(&s);
            match layout {
                Some((_, _, at, blocks_len, _)) => {
                    placed += 1;
                    assert_eq!(
                        free_before - free_after,
                        blocks_len,
                        "hole {hole}: the tail took {} bytes at {at} to write {blocks_len}",
                        free_before - free_after
                    );
                }
                None => assert_eq!(
                    free_after, free_before,
                    "hole {hole}: a tail that declines to place itself must hand back \
                     everything it tried"
                ),
            }
        }
        assert!(
            placed > 0,
            "the sweep must reach holes the tail can actually use, or it asserts \
             nothing about placement"
        );
    }

    /// The shrink pass never grows the file: asked to rewrite the tail with
    /// nothing in the free list to put it in, it writes nothing at all
    /// (issue #418).
    ///
    /// That is the whole safety argument for running it. It exists to make the
    /// file smaller, and a tail it cannot place would otherwise go past
    /// end-of-file — a rewrite taken for the sake of a shorter file leaving a
    /// longer one. The second half of the test is what keeps the first half
    /// honest: under `TailPlacement::Anywhere` the same call on the same session
    /// does write a tail, so the fixture really did have one to write.
    #[test]
    fn a_reuse_only_tail_rewrite_that_cannot_place_itself_writes_nothing() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("reuse_only.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("d")
            .with_i32_data(&(0..400).collect::<Vec<i32>>())
            .with_shape(&[400]);
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 0);
        b.write(&path).unwrap();
        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        // No hole anywhere: the only place a tail could go is past end-of-file.
        s.free = FreeList::new();
        let root = s.superblock.root_group_address;
        let len_before = s.image.len();
        let ext_before = s.superblock.superblock_extension_address;

        s.commit_persisting(root, Vec::new(), TailPlacement::ReuseOnly)
            .unwrap();
        assert_eq!(
            s.image.len(),
            len_before,
            "a reuse-only rewrite with nothing to reuse must leave the file alone"
        );
        assert_eq!(
            s.superblock.superblock_extension_address, ext_before,
            "and must not have published a new extension either"
        );

        s.commit_persisting(root, Vec::new(), TailPlacement::Anywhere)
            .unwrap();
        assert!(
            s.image.len() > len_before,
            "the same rewrite that may append does grow the file, so the case \
             above was a tail declined rather than a tail of no size"
        );
    }

    /// A flat commit's tail removes from the free list exactly the bytes its
    /// recorded extent covers — no more, and nothing at all when it declines to
    /// place itself.
    ///
    /// The flat tail accepts a plan *shorter* than its reservation where the paged
    /// one refuses ([`WriteEngine::flat_tail_layout`] says why), so this is the
    /// invariant that keeps those spare bytes from leaking: what the layout hands
    /// back as the tail's length is the whole reservation, which the commit records
    /// as the extent to free when the next one supersedes it. Returning the plan's
    /// length instead would retire the difference from the free list with nothing
    /// recording it — issue #286 again, a few bytes at a time.
    ///
    /// Swept over hole sizes rather than a chosen one, for the reason the paged
    /// sweep gives: the length the tail proposes is a property of the file and its
    /// free list, and best fit takes a hole of exactly that length outright, which
    /// drops a section from the managers and is what makes the plan come out
    /// shorter than the reservation in the first place.
    #[test]
    fn a_flat_tail_takes_exactly_the_extent_it_records() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        /// Any plausible extension length exercises the same arithmetic; the real
        /// one is settled by the commit, which is not what is under test here.
        const EXT_LEN: u64 = 100;
        /// Any address does: the layout only ever asks the free list for a length.
        const HOLE_AT: u64 = 512;

        let dir = tempdir().unwrap();
        let path = dir.path().join("flat_tail_exact.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("d")
            .with_i32_data(&(0..4000).collect::<Vec<i32>>())
            .with_shape(&[4000]);
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 0);
        b.write(&path).unwrap();
        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        let os = s.superblock.offset_size;

        let free_total =
            |s: &WriteEngine| -> u64 { s.free.sections().into_iter().map(|(_, len)| len).sum() };

        let (mut placed, mut declined, mut with_slack) = (0usize, 0usize, 0usize);
        for hole in 120..420u64 {
            s.free = FreeList::new();
            s.free.free(HOLE_AT, hole);
            let free_before = free_total(&s);
            let (post, at, tail_len, _) = s.flat_tail_layout(&[], &[], EXT_LEN, os);
            let free_after = free_total(&s);
            // The blocks the commit will write into the extent it was handed. A hole
            // consumed outright drops a section from the managers, so this comes out
            // shorter than the extent for some hole sizes and the difference is what
            // the extent has to cover.
            let written = EXT_LEN + file_fsm_blocks_len(&s.persisted_sections(&post), os);
            match at {
                Some(at) => {
                    placed += 1;
                    assert_eq!(
                        free_before - free_after,
                        tail_len,
                        "hole {hole}: the tail took {} bytes at {at} for an extent of \
                         {tail_len}",
                        free_before - free_after
                    );
                    if written < tail_len {
                        with_slack += 1;
                    }
                }
                None => {
                    declined += 1;
                    assert_eq!(
                        free_after, free_before,
                        "hole {hole}: a tail that declines to place itself must hand \
                         back everything it tried"
                    );
                    assert_eq!(
                        tail_len, written,
                        "hole {hole}: a tail with nowhere to go is appended, and the \
                         length it reports is what it will write there"
                    );
                }
            }
        }
        assert!(
            placed > 0,
            "the sweep must reach holes the tail can actually use, or it asserts \
             nothing about placement"
        );
        assert!(
            declined > 0,
            "the sweep must reach holes too small for the tail, or the rule about \
             handing back a failed attempt is never exercised"
        );
        assert!(
            with_slack > 0,
            "the sweep must reach a hole the tail does not fill exactly, or the \
             rule about spare bytes is never exercised"
        );
    }

    /// A paged commit whose tail finds no free space to sit in opens a page for it
    /// — and hands the rest of that page back as free metadata.
    ///
    /// Every paged file this crate writes has holes in it from the start, so the
    /// suite's ordinary fixtures always reuse and this branch is never taken. It is
    /// reachable in the wild, by a file whose free space is all spoken for, and the
    /// page it opens is the same page a page-per-commit leak used to strand. Emptying
    /// the free lists by hand is what puts the file in that state deliberately.
    #[test]
    fn a_paged_tail_with_nowhere_to_go_opens_a_page_and_frees_the_rest() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        const PAGE: u64 = 4096;
        let dir = tempdir().unwrap();
        let path = dir.path().join("tail_appends.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("d")
            .with_i32_data(&(0..4000).collect::<Vec<i32>>())
            .with_shape(&[4000]);
        b.with_file_space_strategy(FileSpaceStrategy::Page, true, 0)
            .with_file_space_page_size(PAGE);
        b.write(&path).unwrap();
        let before = std::fs::metadata(&path).unwrap().len();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        {
            let pg = s.paged.as_mut().expect("a paged file installs paged state");
            pg.meta = FreeList::new();
            pg.raw = FreeList::new();
            pg.unclassified = FreeList::new();
        }
        // A commit with nothing staged returns without writing, so give it one
        // small metadata object to place. It appends for want of anywhere else,
        // and the tail follows it into the same page.
        s.create_group(&ObjectPathBuf::parse("g")).unwrap();
        s.commit().unwrap();

        let after = std::fs::metadata(&path).unwrap().len();
        assert!(
            after > before && (after - before) % PAGE == 0,
            "the commit opened whole pages ({before} -> {after})"
        );
        // What the tail did not fill of the page it opened is metadata this session
        // can still spend. Losing it is how the file used to give up most of a page
        // on every commit; the managers just written cannot record it, so the
        // session carries it to the next commit.
        let pg = s.paged.as_ref().expect("still paged");
        let (addr, len) = pg
            .meta
            .sections()
            .into_iter()
            .find(|&(addr, len)| addr + len == after)
            .expect("the page the tail opened leaves a free remainder at end-of-file");
        assert!(
            len > 0 && len < PAGE && addr >= before,
            "the tail's blocks take the front of the page it opened and the rest is \
             free ({len} of {PAGE} at {addr}, file {before} -> {after})"
        );
    }

    /// Deleting a chunked dataset from a paged file must not record its freed
    /// chunk index in the *metadata* manager.
    ///
    /// Every writer in this crate emits a chunk index in the same run as the chunk
    /// data it indexes, so the index sits in a raw page. Recording it as metadata
    /// would advertise a metadata-sized hole inside a page that still holds another
    /// dataset's live chunk data, and the reference library placing metadata there
    /// would mix the page — the one thing a paged file forbids. Page homogeneity is
    /// preserved on disk either way, so the mis-filing is invisible to a signature
    /// scan of the file and to the C library; the manager a section lands in has to
    /// be checked directly.
    #[test]
    fn deleted_chunk_index_is_freed_into_a_raw_manager() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("paged_chunk_index_free.h5");
        let page = 4096u64;
        let mut b = FileBuilder::new();
        for name in ["drop", "keep"] {
            b.create_dataset(name)
                .with_i32_data(&(0..200).collect::<Vec<i32>>())
                .with_shape(&[200])
                .with_chunks(&[50]);
        }
        b.with_file_space_strategy(FileSpaceStrategy::Page, true, 0)
            .with_file_space_page_size(page);
        b.write(&path).unwrap();

        {
            let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
            s.delete(&ObjectPathBuf::parse("/drop")).unwrap();
            s.commit().unwrap();
        }

        // Pages still occupied by the surviving dataset's chunk data. Read with the
        // session closed: its lock is mandatory on Windows.
        let live_raw_pages: Vec<u64> = {
            let f = crate::reader::File::open(&path).unwrap();
            let ds = f.dataset("keep").unwrap();
            let mut pages: Vec<u64> = ds
                .chunks()
                .unwrap()
                .iter()
                .filter(|c| c.storage_size > 0)
                .flat_map(|c| (c.address / page)..=((c.address + c.storage_size - 1) / page))
                .collect();
            pages.sort_unstable();
            pages.dedup();
            pages
        };
        assert!(!live_raw_pages.is_empty(), "expected live raw pages");

        let s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        let pg = s.paged.as_ref().expect("a paged file installs paged state");
        for (addr, len) in pg.meta.sections() {
            for p in (addr / page)..=((addr + len - 1) / page) {
                assert!(
                    !live_raw_pages.contains(&p),
                    "metadata free section ({addr}, {len}) sits in page {p}, which still \
                     holds live raw chunk data"
                );
            }
        }
        // The index really was reclaimed somewhere, so this is not vacuous.
        let reclaimed: u64 = pg.reusable_sections().iter().map(|&(_, l)| l).sum();
        assert!(reclaimed > 0, "the delete reclaimed nothing");
    }

    /// A paged commit that fails partway must leave the session's free lists
    /// exactly as it found them.
    ///
    /// Everything the commit gathers to free is still *live* until the superblock
    /// repoint: the objects occupying those regions are reachable from the old
    /// root, which a failed commit never replaces. A session that recorded them as
    /// free anyway would hand them out on the next commit, and the file would lose
    /// data with no error anywhere — in a release build, where the free list's
    /// double-free `debug_assert` is compiled out, silently.
    ///
    /// The failure is induced by pointing the superblock extension at a byte range
    /// that is not an object header, which fails the extension rewrite immediately
    /// after the regions are gathered.
    #[test]
    fn failed_paged_commit_leaves_the_free_lists_untouched() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("paged_failed_commit.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("keep")
            .with_i32_data(&(0..200).collect::<Vec<i32>>())
            .with_shape(&[200]);
        b.create_dataset("drop")
            .with_i32_data(&(0..200).collect::<Vec<i32>>())
            .with_shape(&[200]);
        b.with_file_space_strategy(FileSpaceStrategy::Page, true, 0)
            .with_file_space_page_size(4096);
        b.write(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        let before = s.space_accounting().reusable_free_space;

        // Break the extension so the commit fails *after* it has gathered the
        // regions `drop` vacates and *before* the superblock repoint.
        let good_ext = s.superblock.superblock_extension_address;
        s.superblock.superblock_extension_address = Some(0);
        s.delete(&ObjectPathBuf::parse("/drop")).unwrap();
        assert!(
            s.commit().is_err(),
            "a commit with an unreadable extension must fail"
        );

        assert_eq!(
            s.space_accounting().reusable_free_space,
            before,
            "a failed commit must not record still-live regions as free"
        );

        // The session stays usable: repair the extension and commit for real. If
        // the failed commit had folded its regions in, this second commit would
        // double-free them (a debug assertion) and publish `keep`'s live extent.
        s.superblock.superblock_extension_address = good_ext;
        s.delete(&ObjectPathBuf::parse("/drop")).unwrap();
        s.commit()
            .expect("the session is usable after a failed commit");

        // Release the session's exclusive OS lock before reading the file back.
        // Those locks are mandatory on Windows, so a `File::open` overlapping the
        // session fails outright there (advisory locks elsewhere would allow it).
        drop(s);

        let f = crate::reader::File::open(&path).unwrap();
        let kept = f.dataset("keep").unwrap().read_i32().unwrap();
        assert_eq!(kept, (0..200).collect::<Vec<i32>>(), "keep survives intact");
        let freed: u64 = f.persisted_free_space().iter().map(|&(_, l)| l).sum();
        let live_end = f.file_size();
        assert!(
            freed < live_end,
            "the recorded free space cannot cover the whole file"
        );
    }

    /// A commit that fails partway must roll back the heap-collection
    /// provenance along with the free lists (issue #321).
    ///
    /// `resolve_overwrite_bytes` places a collection and records it in the
    /// *apply* phase. A commit that then fails before its repoint gives that
    /// space back to the free list — and used to keep the record, which is the
    /// one piece of engine state naming file addresses that the rollback missed.
    /// The next overwrite of that path would free the region a *later* commit
    /// had since been handed, so the file's live variable-length data would sit
    /// in space the allocator considers free, and the next unrelated write would
    /// land on it.
    ///
    /// Induced the same way as
    /// [`failed_paged_commit_leaves_the_free_lists_untouched`]: an unreadable
    /// superblock extension fails the commit after the apply phase has run.
    #[test]
    fn a_failed_commit_rolls_back_the_heap_collection_provenance() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("failed_commit_vl_provenance.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("labels")
            .with_vlen_strings(&["seed-one", "seed-two"]);
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 0);
        b.write(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();

        // One good round, so there is a record to roll back.
        s.stage_dataset_write(&ObjectPathBuf::parse("/labels"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_vlen_strings(&["round-one-aaaa", "round-one-bbbb"]);
            db
        })
        .unwrap();
        s.commit().unwrap();
        let recorded = s.vl_overwrite_heaps.clone();
        assert!(!recorded.is_empty(), "the good round recorded nothing");

        // Break the extension so the next commit fails after the apply phase.
        let good_ext = s.superblock.superblock_extension_address;
        s.superblock.superblock_extension_address = Some(0);
        s.stage_dataset_write(&ObjectPathBuf::parse("/labels"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_vlen_strings(&["round-two-aaaa", "round-two-bbbb"]);
            db
        })
        .unwrap();
        assert!(
            s.commit().is_err(),
            "a commit with an unreadable extension must fail"
        );

        assert_eq!(
            s.vl_overwrite_heaps, recorded,
            "a failed commit must not leave a record naming space it gave back"
        );

        // The session stays usable, and the collections the rolled-back record
        // names are still the live ones.
        s.superblock.superblock_extension_address = good_ext;
        s.stage_dataset_write(&ObjectPathBuf::parse("/labels"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_vlen_strings(&["round-three-a", "round-three-b"]);
            db
        })
        .unwrap();
        s.commit()
            .expect("the session is usable after a failed commit");
        drop(s);

        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(
            f.dataset("labels").unwrap().read_string().unwrap(),
            vec!["round-three-a".to_string(), "round-three-b".to_string()]
        );
    }

    /// A failed commit must not leave the free list offering a span the image
    /// still points into (issue #321).
    ///
    /// Resolving a staged variable-length overwrite *allocates* — a global heap
    /// collection, drawn from a freed region where one fits. If the overwrite
    /// were applied in place, that allocation's address would be written into a
    /// data block the current root already reaches; a commit failing after the
    /// write and before its repoint then hands the span back to the free list,
    /// while the image still names it. The next commit places something else
    /// there and the dataset reads that object's heap bytes, with every checksum
    /// in the file intact.
    ///
    /// A relocating overwrite has no such window — its new block is reachable
    /// from nothing until the repoint — which is why `prepare_write` refuses to
    /// plan a staged variable-length overwrite in place at all. This is that
    /// rule's test.
    ///
    /// The freed hole has to exist *first*, or the failed commit's collection is
    /// appended past end-of-file where nothing reuses it and the bug hides. The
    /// filler datasets each need a metadata span of the size the rolled-back one
    /// occupied, which is what draws on it.
    #[test]
    fn a_failed_commit_leaves_no_reusable_span_the_image_names() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("failed_commit_live_span.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("labels")
            .with_vlen_strings(&["seed-one", "seed-two"]);
        b.create_dataset("big").with_u8_data(&vec![0x5A; 40960]);
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 0);
        b.write(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        // The hole the collections below are placed into by reuse.
        s.delete(&ObjectPathBuf::parse("/big")).unwrap();
        s.commit().unwrap();

        let round_one = ["round-one-aaaa", "round-one-bbbb"];
        s.stage_dataset_write(&ObjectPathBuf::parse("/labels"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_vlen_strings(&round_one);
            db
        })
        .unwrap();
        s.commit().unwrap();

        // Fail a second overwrite after its apply phase has placed a collection.
        let good_ext = s.superblock.superblock_extension_address;
        s.superblock.superblock_extension_address = Some(0);
        s.stage_dataset_write(&ObjectPathBuf::parse("/labels"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_vlen_strings(&["round-two-aaaa", "round-two-bbbb"]);
            db
        })
        .unwrap();
        assert!(
            s.commit().is_err(),
            "a commit with an unreadable extension must fail"
        );
        s.superblock.superblock_extension_address = good_ext;

        // Unrelated commits that each want a metadata span of the same size.
        // Whatever the failed attempt gave back, these are what would draw on it.
        for i in 0..4 {
            s.stage_created_dataset(&ObjectPathBuf::parse(&format!("/filler{i}")), {
                let mut db = crate::type_builders::DatasetBuilder::new("");
                db.with_vlen_strings(&["XXXXXXXXXXXXXXXXXXXX", "YYYYYYYYYYYYYYYYYYYY"]);
                db
            })
            .unwrap();
            s.commit().unwrap();
        }

        // Read only once the session has released its lock on the file: those
        // locks are mandatory on Windows, so a `File::open` overlapping the
        // session fails outright there where advisory locks elsewhere allow it.
        drop(s);

        let f = crate::reader::File::open(&path).unwrap();
        let got = f.dataset("labels").unwrap().read_string();
        assert!(
            got.as_ref()
                .is_ok_and(|v| v.iter().map(String::as_str).eq(round_one)),
            "/labels reads another dataset's heap objects: {got:?}"
        );
    }

    /// A commit that writes into reused free space and then dies must leave the
    /// file exactly as it found it.
    ///
    /// This is the safety argument for reuse, stated as a test: a commit may
    /// overwrite a freed region *before* the superblock repoint precisely because
    /// nothing reachable from the on-disk root lives there any more, so an attempt
    /// that never reaches the repoint is invisible. A chunked dataset is the case
    /// worth pinning — it is the largest thing a commit places, so it overwrites
    /// the most, and it is the one that used only to append.
    ///
    /// The failure is induced the same way as
    /// [`failed_paged_commit_leaves_the_free_lists_untouched`]: an unreadable
    /// superblock extension, which the persisting tail hits *after* the apply loop
    /// has written every object.
    #[test]
    fn a_failed_commit_that_reused_free_space_leaves_the_file_intact() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("reuse_failed_commit.h5");
        let victim: Vec<f64> = (0..4096).map(|i| (i % 13) as f64).collect();
        let ceiling: Vec<i32> = (0..500).collect();
        let mut b = FileBuilder::new();
        b.create_dataset("keep").with_i32_data(&[1, 2, 3]);
        b.create_dataset("victim")
            .with_f64_data(&victim)
            .with_shape(&[4096])
            .with_chunks(&[512]);
        // Above the victim, so the delete leaves an interior hole and the reuse
        // has live bytes on both sides of what it overwrites.
        b.create_dataset("ceiling")
            .with_i32_data(&ceiling)
            .with_shape(&[500]);
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 1);
        b.write(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        s.delete(&ObjectPathBuf::parse("/victim")).unwrap();
        s.commit().unwrap();
        let free_before = s.space_accounting().reusable_free_space;
        let len_before = std::fs::metadata(&path).unwrap().len();
        assert!(
            free_before.iter().any(|&(_, l)| l > 4096 * 8 / 2),
            "the deleted chunked dataset left a hole worth reusing: {free_before:?}"
        );

        // Break the extension so the commit fails in its persisting tail, after
        // the apply loop has written the new dataset into that hole.
        s.superblock.superblock_extension_address = Some(0);
        let mut db = crate::type_builders::DatasetBuilder::new("fresh");
        db.with_f64_data(&vec![7.5f64; 4096])
            .with_shape(&[4096])
            .with_chunks(&[512]);
        s.stage_created_dataset(&ObjectPathBuf::parse("/fresh"), db)
            .unwrap();
        assert!(
            s.commit().is_err(),
            "a commit with an unreadable extension must fail"
        );
        assert_eq!(
            s.space_accounting().reusable_free_space,
            free_before,
            "the failed commit gives back the region it drew from"
        );
        // Release the session's exclusive OS lock before reading the file back;
        // those locks are mandatory on Windows.
        drop(s);

        // The file still describes the tree the last *successful* commit left: the
        // survivors read exactly, the half-written dataset is not linked, and the
        // superblock's end-of-file still matches the file.
        assert_eq!(std::fs::metadata(&path).unwrap().len(), len_before);
        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(f.file_size(), len_before);
        assert_eq!(
            f.dataset("keep").unwrap().read_i32().unwrap(),
            vec![1, 2, 3]
        );
        assert_eq!(f.dataset("ceiling").unwrap().read_i32().unwrap(), ceiling);
        assert!(f.dataset("victim").is_err());
        assert!(f.dataset("fresh").is_err());
    }

    /// A refused commit leaves every dataset reading what it read before —
    /// including the one whose values it had already written over (issue #344).
    ///
    /// Everything else a commit places lands where nothing reaches it until the
    /// superblock is repointed, so an attempt that stops short of the repoint is
    /// invisible. A same-length value overwrite is the exception: it writes
    /// straight over the data block the *current* root already reaches, so there
    /// is no repoint to withhold and the write is live the moment it lands. Both
    /// halves of one refused batch are asserted here, because the defect was
    /// that they disagreed: the overwrite survived and the object creation
    /// beside it did not.
    ///
    /// Closing the session is enough to expose it — no later commit is needed,
    /// since the bytes are in the file already and write gathering only delays
    /// when they reach the disk.
    ///
    /// A second staged edit is what keeps this on the full commit path; a batch
    /// of nothing but same-length overwrites takes the fast path, which
    /// `a_failed_fast_path_commit_puts_back_the_value_it_overwrote` covers. The
    /// failure is induced the same way as
    /// [`failed_paged_commit_leaves_the_free_lists_untouched`]: an unreadable
    /// superblock extension, which fails the commit in its tail, after the apply
    /// phase has done the in-place write.
    #[test]
    fn a_failed_commit_puts_back_the_value_it_overwrote() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("inplace_partial.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("nums").with_i32_data(&[1, 2, 3]);
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 0);
        b.write(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();

        let good_ext = s.superblock.superblock_extension_address;
        s.superblock.superblock_extension_address = Some(0);
        s.stage_dataset_write(&ObjectPathBuf::parse("/nums"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&[9, 9, 9]);
            db
        })
        .unwrap();
        s.stage_created_dataset(&ObjectPathBuf::parse("/extra"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&[42]);
            db
        })
        .unwrap();
        let refused = s.commit();
        assert!(
            matches!(refused, Err(Error::EditUnsupported(_))),
            "the refusal, not a rollback failure: {refused:?}"
        );

        // The session stays usable, and its next commit is not carrying any of
        // the refused batch.
        s.superblock.superblock_extension_address = good_ext;
        s.stage_created_dataset(&ObjectPathBuf::parse("/later"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&[7]);
            db
        })
        .unwrap();
        s.commit()
            .expect("the session is usable after a failed commit");

        // Release the session's exclusive OS lock before reading the file back;
        // those locks are mandatory on Windows.
        drop(s);

        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(
            f.dataset("nums").unwrap().read_i32().unwrap(),
            vec![1, 2, 3],
            "the refused batch's value overwrite is in the file"
        );
        assert!(
            f.dataset("extra").is_err(),
            "/extra was correctly discarded"
        );
        assert_eq!(f.dataset("later").unwrap().read_i32().unwrap(), vec![7]);
    }

    /// The same guarantee on the commit fast path, where a batch of nothing but
    /// same-length overwrites skips the header rebuild and the superblock flip
    /// entirely (issue #344).
    ///
    /// That path has no tail to fail in, so the only failure it can suffer is
    /// one of its own writes — which is also the one failure the rollback cannot
    /// fully repair, since putting the prior bytes back means writing to the
    /// address that just refused them. Both outcomes are pinned here: the
    /// overwrite that *did* land is put back, the one that could not be is
    /// reported as [`Error::CommitPartiallyApplied`] rather than as a plain I/O
    /// error, and the values in the file are exactly what those two statements
    /// predict.
    ///
    /// `TornWriteImage` supplies the failure at a byte range rather than at a
    /// call ordinal, so the test names the block it means (`victim`'s) instead
    /// of depending on how many writes a commit happens to issue.
    ///
    /// `kept` is staged first on purpose. Staged writes are applied in the order
    /// they were staged (`StagedEdits::writes` is a `Vec`), so it is the entry
    /// already in the journal when `victim` fails — and restoring it is what
    /// shows the rollback carrying on past a failure rather than giving up at
    /// the first one. Stage them the other way round and that half goes quiet.
    #[test]
    fn a_failed_fast_path_commit_puts_back_the_value_it_overwrote() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("inplace_fast_path_torn.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("kept").with_i32_data(&[1, 2, 3]);
        b.create_dataset("victim").with_i32_data(&[4, 5, 6]);
        b.write(&path).unwrap();

        // The data block the fake device refuses, named rather than guessed.
        let victim_block = {
            let f = crate::reader::File::open(&path).unwrap();
            match f.dataset("victim").unwrap().layout().unwrap() {
                crate::Layout::Contiguous {
                    address: Some(a),
                    size,
                } => a..a + size,
                other => panic!("expected a contiguous victim: {other:?}"),
            }
        };

        let mut s = WriteEngine::open_torn_writes(&path, victim_block).unwrap();
        for (name, data) in [("/kept", [9, 9, 9]), ("/victim", [8, 8, 8])] {
            s.stage_dataset_write(&ObjectPathBuf::parse(name), {
                let mut db = crate::type_builders::DatasetBuilder::new("");
                db.with_i32_data(&data);
                db
            })
            .unwrap();
        }
        let refused = s.commit();
        assert!(
            matches!(refused, Err(Error::CommitPartiallyApplied { .. })),
            "a rollback that could not run is not an ordinary refusal: {refused:?}"
        );
        drop(s);

        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(
            f.dataset("kept").unwrap().read_i32().unwrap(),
            vec![1, 2, 3],
            "the overwrite the rollback could reach is put back"
        );
        assert_eq!(
            f.dataset("victim").unwrap().read_i32().unwrap(),
            vec![8, 8, 8],
            "the overwrite it could not reach is the value the error is about"
        );
    }

    /// Past the superblock repoint the overwrite is part of what the commit
    /// *published*, so a failure after that point must leave it standing —
    /// the same boundary [`FreeSnapshot`] draws for the free lists (issue #344).
    ///
    /// The window is real rather than theoretical: `repoint_stored_references`
    /// runs after the repoint and writes into live objects. A reference to the
    /// root group is what makes it run on any commit at all, since every commit
    /// rebuilds the root; the fake device then fails that write, so the commit
    /// returns an error having already committed.
    ///
    /// What the file holds afterwards is the new tree — the value overwrite
    /// included — with one stale reference. That is not a state a rollback can
    /// improve on: the root the reference would have to be re-pointed at is
    /// already the live one.
    #[test]
    fn a_commit_that_fails_past_its_repoint_keeps_the_value_it_published() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("inplace_after_repoint.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("nums").with_i32_data(&[1, 2, 3]);
        b.create_dataset("refs").with_path_references(&[""]);
        b.write(&path).unwrap();

        let refs_block = {
            let f = crate::reader::File::open(&path).unwrap();
            match f.dataset("refs").unwrap().layout().unwrap() {
                crate::Layout::Contiguous {
                    address: Some(a),
                    size,
                } => a..a + size,
                other => panic!("expected a contiguous reference dataset: {other:?}"),
            }
        };

        let mut s = WriteEngine::open_torn_writes(&path, refs_block).unwrap();
        s.stage_dataset_write(&ObjectPathBuf::parse("/nums"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&[9, 9, 9]);
            db
        })
        .unwrap();
        // A second edit, so the batch takes the full commit path and reaches a
        // repoint at all.
        s.stage_created_dataset(&ObjectPathBuf::parse("/extra"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&[42]);
            db
        })
        .unwrap();
        let failed = s.commit();
        assert!(
            matches!(failed, Err(Error::Io(_))),
            "the reference write is what failed, after the commit published: {failed:?}"
        );
        assert!(
            s.publish_attempted,
            "the failure has to be past the publish for this test to mean anything"
        );
        drop(s);

        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(
            f.dataset("nums").unwrap().read_i32().unwrap(),
            vec![9, 9, 9],
            "the published overwrite must not be rolled back under it"
        );
        assert_eq!(f.dataset("extra").unwrap().read_i32().unwrap(), vec![42]);
    }

    #[test]
    fn append_inplace_crash_consistency_paged_prefix() {
        use crate::reader::File as PureFile;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let base = dir.path().join("base.h5");
        let (start, target) = (131_000i32, 132_000i32);
        build_unit_chunked(&base, start);

        for max_phase in 1u8..=4 {
            let p = dir.path().join(format!("crash_paged_{max_phase}.h5"));
            append_stopped_at(&base, &p, start..target, max_phase);
            let expected_len = if max_phase == 4 { target } else { start };
            let f = PureFile::from_bytes(std::fs::read(&p).unwrap()).unwrap();
            assert_eq!(
                f.dataset("d").unwrap().read_i32().unwrap(),
                (0..expected_len).collect::<Vec<_>>(),
                "inconsistent paged view after crash at phase {max_phase}"
            );
        }
    }

    /// After an append stopped at any phase, a reader sees the committed prefix
    /// and nothing of the rest, for both starting layouts: a partial trailing
    /// chunk and the EA-boundary growth above, which exercise different index
    /// writes.
    ///
    /// This reader is the lenient one. It bounds chunk reads by `min(EA count,
    /// dimension)`, so it tolerates a phase-3 state where the element count has
    /// advanced past the dimension, where the reference C library walks strictly
    /// by the dataspace dimension and re-validates block checksums. The C
    /// library read these states until it became a dependency of the crosscheck
    /// package alone; the phased append that produces them is private, so that
    /// coverage is gone, and a stale end-of-file, a half-grown index or a
    /// mis-checksummed block that this reader forgives is not caught here.
    ///
    /// Restores coverage lost with the deprecated `SwmrWriter` and `AppendWriter`
    /// (issue #202).
    #[test]
    fn append_inplace_crash_consistency_leaves_the_committed_prefix() {
        use tempfile::tempdir;

        // (initial length, chunk length, appended length): a partial trailing
        // chunk, a chunk-aligned start, and the EA-boundary crossing.
        for (n, chunk, add) in [(6i32, 4u64, 5i32), (8, 2, 6), (50, 1, 200)] {
            let dir = tempdir().unwrap();
            let base = dir.path().join("base.h5");
            {
                use crate::writer::FileBuilder;
                let mut b = FileBuilder::new();
                b.create_dataset("d")
                    .with_i32_data(&(0..n).collect::<Vec<i32>>())
                    .with_shape(&[n as u64])
                    .with_maxshape(&[MaxExtent::Unlimited])
                    .with_chunks(&[chunk]);
                b.write(&base).unwrap();
            }

            for max_phase in 1u8..=4 {
                let p = dir
                    .path()
                    .join(format!("crash_c_{n}_{chunk}_{max_phase}.h5"));
                append_stopped_at(&base, &p, n..n + add, max_phase);
                let expected_len = if max_phase == 4 { n + add } else { n };
                let pf = crate::reader::File::from_bytes(std::fs::read(&p).unwrap()).unwrap();
                assert_eq!(
                    pf.dataset("d").unwrap().read_i32().unwrap(),
                    (0..expected_len).collect::<Vec<_>>(),
                    "inconsistent view after crash at phase {max_phase} (n={n}, chunk={chunk})"
                );
            }
        }
    }

    /// Crash recovery across the phase-3/phase-4 gap.
    ///
    /// A writer that crashes after publishing the Extensible-Array element count
    /// (phase 3) but before publishing the dataspace dimension (phase 4) leaves
    /// the on-disk count ahead of the committed dimension. A fresh writer must
    /// roll forward from the *committed dimension*, overwriting the uncommitted
    /// slots, rather than appending past them and leaving a gap.
    ///
    /// The crashed and recovering appends deliberately write different values at
    /// the overlapping positions, so a regression that seeds the chunk count from
    /// the stale EA header surfaces the crashed writer's values rather than
    /// merely producing plausible-looking data.
    ///
    /// Restores coverage lost with the deprecated `SwmrWriter` (issue #202); the
    /// surviving `recover_and_reappend_after_clean_phase4` covers only the clean
    /// case.
    #[test]
    fn append_inplace_recover_and_reappend_after_phase3_crash() {
        use crate::reader::File as PureFile;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("phase3_recover.h5");
        let n = 50i32;
        build_unit_chunked(&path, n);

        // Writer 1 crashes after phase 3: the element count advances but the
        // dimension stays at `n`. Its values are far from the correct
        // continuation, so a leak is unmistakable.
        {
            let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
            s.append_inplace_i32_phased("d", &(1000..1200).collect::<Vec<_>>(), 3)
                .unwrap();
        }
        let committed: Vec<i32> = (0..n).collect();
        let pf = PureFile::from_bytes(std::fs::read(&path).unwrap()).unwrap();
        assert_eq!(
            pf.dataset("d").unwrap().read_i32().unwrap(),
            committed,
            "phase-3 crash exposed uncommitted data to the pure reader"
        );

        // Writer 2 recovers: roll forward from the committed dimension,
        // overwriting the uncommitted slots with the real continuation.
        {
            let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
            s.append_inplace_i32_phased("d", &(n..150).collect::<Vec<_>>(), 4)
                .unwrap();
        }

        let expected: Vec<i32> = (0..150).collect();
        let pf = PureFile::from_bytes(std::fs::read(&path).unwrap()).unwrap();
        assert_eq!(
            pf.dataset("d").unwrap().read_i32().unwrap(),
            expected,
            "recovery did not roll forward correctly (pure reader)"
        );
    }

    #[test]
    fn raw_appendable_recurses_into_aggregates() {
        let f64_with = |byte_order| Datatype::FloatingPoint {
            size: 8,
            byte_order,
            layout: FloatingPointLayout {
                bit_offset: 0,
                bit_precision: 64,
                exponent_location: 52,
                exponent_size: 11,
                mantissa_location: 0,
                mantissa_size: 52,
                exponent_bias: 1023,
            },
        };
        let le_f64 = f64_with(DatatypeByteOrder::LittleEndian);
        let be_f64 = f64_with(DatatypeByteOrder::BigEndian);

        // Little-endian scalar: appendable. Big-endian scalar: not.
        assert!(datatype_is_raw_appendable(&le_f64));
        assert!(!datatype_is_raw_appendable(&be_f64));

        // The confirmed bug: a compound / array whose leaf is big-endian must be
        // refused (it was wrongly accepted before recursion was added).
        let be_member = Datatype::Compound {
            size: 8,
            members: vec![CompoundMember {
                name: "x".into(),
                byte_offset: 0,
                datatype: be_f64.clone(),
            }],
        };
        assert!(!datatype_is_raw_appendable(&be_member));
        let le_member = Datatype::Compound {
            size: 8,
            members: vec![CompoundMember {
                name: "x".into(),
                byte_offset: 0,
                datatype: le_f64.clone(),
            }],
        };
        assert!(datatype_is_raw_appendable(&le_member));
        assert!(!datatype_is_raw_appendable(&Datatype::Array {
            base_type: Box::new(be_f64.clone()),
            dimensions: vec![4],
        }));

        // Variable-length / reference leaves are never raw-appendable, even LE.
        assert!(!datatype_is_raw_appendable(&Datatype::VariableLength {
            is_string: false,
            padding: None,
            charset: None,
            base_type: Box::new(le_f64.clone()),
        }));
        assert!(!datatype_is_raw_appendable(&Datatype::Reference {
            size: 8,
            ref_type: crate::datatype::ReferenceType::Object,
        }));
    }

    #[test]
    fn fresh_group_region_pairs_link_info_with_group_info() {
        // A new-style group must carry both a Link Info and a Group Info message
        // (the C library requires the pair before it will insert a link).
        let types = region_types(&fresh_group_region());
        assert_eq!(types, vec![MessageType::LINK_INFO, MessageType::GROUP_INFO]);
    }

    #[test]
    fn ensure_group_info_appends_when_missing() {
        // A region with a Link Info message but no Group Info message (how older
        // hdf5-pure releases wrote groups) gains exactly one Group Info message.
        let li_body = {
            let mut b = vec![0u8, 0];
            b.extend_from_slice(&u64::MAX.to_le_bytes());
            b.extend_from_slice(&u64::MAX.to_le_bytes());
            b
        };
        let mut region = plain_region(message_record(MessageType::LINK_INFO, &li_body));
        ensure_group_info(&mut region).unwrap();
        assert_eq!(
            region_types(&region),
            vec![MessageType::LINK_INFO, MessageType::GROUP_INFO]
        );

        // The appended message decodes as a minimal Group Info body.
        let mut p = 0;
        while let Some((mt, body, end)) = region.next_message(p).unwrap() {
            if mt == MessageType::GROUP_INFO {
                assert_eq!(&region[body..end], &GROUP_INFO_BODY);
            }
            p = end;
        }
    }

    /// A group header region that tracks link creation order: a Link Info
    /// message recording `max` as the next index it would hand out (with the
    /// creation-order *index* declared when `indexed`, as h5py's
    /// `track_order=True` does), a Group Info message, and one compact Link
    /// message per name in `links`, numbered from zero.
    fn link_tracked_region(max: u64, indexed: bool, links: &[&str]) -> OhRegion {
        let mut li = vec![0u8, if indexed { 0x03 } else { 0x01 }];
        li.extend_from_slice(&max.to_le_bytes());
        li.extend_from_slice(&u64::MAX.to_le_bytes()); // fractal heap: compact
        li.extend_from_slice(&u64::MAX.to_le_bytes()); // b-tree name index
        if indexed {
            li.extend_from_slice(&u64::MAX.to_le_bytes()); // b-tree creation order
        }
        let mut region = OhRegion::empty(OhHeaderProps::PLAIN);
        region.push(MessageType::LINK_INFO, &li);
        region.push(MessageType::GROUP_INFO, &GROUP_INFO_BODY);
        for (i, name) in links.iter().enumerate() {
            region.push_link(name, StoredAddress::new(0x100 + i as u64), Some(i as u64));
        }
        region
    }

    /// Every Link message in `region`, as `(name, creation index)`.
    fn region_links(region: &OhRegion) -> Vec<(String, Option<u64>)> {
        let mut out = Vec::new();
        let mut p = 0;
        while let Some((mt, body, end)) = region.next_message(p).unwrap() {
            if mt == MessageType::LINK {
                let link = LinkMessage::parse(&region[body..end], OFFSET_SIZE).unwrap();
                out.push((link.name, link.creation_order));
            }
            p = end;
        }
        out
    }

    /// The maximum creation index `region`'s Link Info message records.
    fn recorded_link_max(region: &OhRegion) -> Option<u64> {
        find_link_info(region)
            .unwrap()
            .and_then(|(_, _, info)| info.max_creation_order)
    }

    #[test]
    fn links_added_to_a_tracked_group_take_consecutive_creation_indexes() {
        // A group that has handed out three indexes numbers the next two links
        // 3 and 4, in the order they are placed, and its Link Info message ends
        // up recording 5 — the next index, not the highest in use.
        let mut region = link_tracked_region(3, false, &["a", "b", "c"]);
        let mut order = LinkCreationOrder::for_region(&region).unwrap();
        region.push_link("d", StoredAddress::new(0x200), order.take().unwrap());
        region.push_link("e", StoredAddress::new(0x300), order.take().unwrap());
        order.record(&mut region).unwrap();

        assert_eq!(
            region_links(&region),
            vec![
                ("a".to_string(), Some(0)),
                ("b".to_string(), Some(1)),
                ("c".to_string(), Some(2)),
                ("d".to_string(), Some(3)),
                ("e".to_string(), Some(4)),
            ],
        );
        assert_eq!(recorded_link_max(&region), Some(5));
    }

    #[test]
    fn a_tracked_group_resumes_past_a_gap_a_deletion_left() {
        // Two links of three deleted, so the surviving link is index 1 and the
        // recorded maximum still says 3: the addition takes 3, not one of the
        // indexes the deletions freed.
        let region = link_tracked_region(3, false, &["a", "b", "c"]);
        let mut region = remove_link_from_region(&region, "a").unwrap();
        region = remove_link_from_region(&region, "c").unwrap();
        let mut order = LinkCreationOrder::for_region(&region).unwrap();
        region.push_link("d", StoredAddress::new(0x200), order.take().unwrap());
        order.record(&mut region).unwrap();

        assert_eq!(
            region_links(&region),
            vec![("b".to_string(), Some(1)), ("d".to_string(), Some(3))],
        );
        assert_eq!(recorded_link_max(&region), Some(4));
    }

    #[test]
    fn a_creation_order_index_shifts_no_field_the_maximum_is_written_into() {
        // A group that also declares the creation-order *index* (h5py's
        // `track_order=True`) has a longer Link Info body with a third address
        // in it. The maximum still precedes every address, so the patch lands on
        // it and leaves the three addresses undefined.
        let mut region = link_tracked_region(1, true, &["a"]);
        let mut order = LinkCreationOrder::for_region(&region).unwrap();
        region.push_link("b", StoredAddress::new(0x200), order.take().unwrap());
        order.record(&mut region).unwrap();

        let (_, _, info) = find_link_info(&region).unwrap().unwrap();
        assert_eq!(info.max_creation_order, Some(2));
        assert_eq!(info.fractal_heap_address, None);
        assert_eq!(info.btree_name_index_address, None);
        assert_eq!(info.btree_creation_order_address, None);
        assert_eq!(region_links(&region).last().unwrap().1, Some(1));
    }

    #[test]
    fn an_untracked_group_numbers_none_of_its_links() {
        // The groups this crate writes itself: no maximum recorded, so no link
        // carries a creation index and the Link Info message is left as it was.
        let mut region = fresh_group_region();
        let before = region.clone();
        let mut order = LinkCreationOrder::for_region(&region).unwrap();
        assert_eq!(order.take().unwrap(), None);
        region.push_link("d", StoredAddress::new(0x200), None);
        order.record(&mut region).unwrap();

        assert_eq!(region_links(&region), vec![("d".to_string(), None)]);
        assert_eq!(recorded_link_max(&region), None);
        // Only the appended link is new; every message that was there is
        // untouched.
        assert_eq!(&region[..before.len()], &*before);
    }

    #[test]
    fn a_tracked_group_that_gains_no_link_keeps_its_maximum_byte_for_byte() {
        // A commit that only deletes, or only edits attributes, must not bump
        // the counter — the gap it leaves is never handed out again.
        let region = link_tracked_region(3, false, &["a", "b", "c"]);
        let mut after = remove_link_from_region(&region, "b").unwrap();
        let order = LinkCreationOrder::for_region(&after).unwrap();
        let before = after.clone();
        order.record(&mut after).unwrap();
        assert_eq!(*after, *before);
        assert_eq!(recorded_link_max(&after), Some(3));
    }

    #[test]
    fn max_compact_links_reads_the_declared_threshold_or_the_library_default() {
        // A minimal Group Info message stores no link-phase-change values, so
        // the C library's default of 8 applies.
        assert_eq!(max_compact_links(&fresh_group_region()).unwrap(), 8);

        // One that stores them (flags bit 0) declares its own maximum compact
        // value in the two bytes after the flags.
        let mut gi = vec![0u8, 0x01];
        gi.extend_from_slice(&4u16.to_le_bytes()); // max compact
        gi.extend_from_slice(&2u16.to_le_bytes()); // min dense
        let mut region = OhRegion::empty(OhHeaderProps::PLAIN);
        region.push(MessageType::GROUP_INFO, &gi);
        assert_eq!(max_compact_links(&region).unwrap(), 4);
    }

    #[test]
    fn a_tracked_group_is_refused_only_where_the_addition_would_go_dense() {
        // Eight links is the default threshold, so a group holding seven takes
        // one more and is refused the second.
        let region = link_tracked_region(7, false, &["a", "b", "c", "d", "e", "f", "g"]);
        reject_dense_link_creation_order(&region, 8).expect("eight links still store compactly");
        let err = reject_dense_link_creation_order(&region, 9).unwrap_err();
        assert!(
            matches!(&err, Error::EditUnsupported(m) if m.contains("link creation order")
                && m.contains("dense")),
            "got: {err}",
        );

        // A group that does not track the order is not screened at all: this
        // crate writes more than eight compact links routinely.
        let mut untracked = fresh_group_region();
        untracked.push_link("a", StoredAddress::new(0x100), None);
        reject_dense_link_creation_order(&untracked, 99)
            .expect("an untracked group has no creation order to lose");
    }

    #[test]
    fn ensure_group_info_is_idempotent() {
        // A region that already has a Group Info message is left untouched, so
        // re-editing a healed (or C-written) group does not duplicate it.
        let mut region = fresh_group_region();
        let before = region.clone();
        ensure_group_info(&mut region).unwrap();
        assert_eq!(*region, *before);
    }

    // ---- shared (SOHM) messages (issue #417) ----

    /// A shared-message index record naming an object header is a stored address
    /// like any other, and the one no repoint walk reaches: it lives outside
    /// every object a commit rebuilds. A commit that removes or moves that header
    /// is refused rather than leaving the index naming bytes that have gone.
    #[test]
    fn a_commit_that_strands_a_shared_message_record_is_refused() {
        let named = crate::sohm::SohmRecord {
            hash: 1,
            location: crate::sohm::SohmLocation::ObjectHeader {
                message_type: MessageType::ATTRIBUTE.to_u16() as u8,
                creation_index: 0,
                address: StoredAddress::new(0x400),
            },
        };
        let moved = InvalidatedAddresses {
            removed: Vec::new(),
            moved: vec![0x400],
            base: BaseAddress::ZERO,
        };
        let err = screen_shared_message_records(std::slice::from_ref(&named), &moved).unwrap_err();
        assert_eq!(
            err.to_string(),
            Error::EditUnsupported(SHARED_MESSAGE_INDEX_NAMES_A_MOVED_OBJECT).to_string()
        );

        let elsewhere = InvalidatedAddresses {
            removed: Vec::new(),
            moved: vec![0x800],
            base: BaseAddress::ZERO,
        };
        screen_shared_message_records(std::slice::from_ref(&named), &elsewhere).unwrap();
    }

    /// A heap-stored record names no object header, so no address of it can go
    /// stale and the screen passes it. What a commit can spoil about one is its
    /// reference count, which the attribute paths refuse to change instead.
    #[test]
    fn a_heap_stored_shared_message_record_is_not_screened_for_addresses() {
        let heap = crate::sohm::SohmRecord {
            hash: 1,
            location: crate::sohm::SohmLocation::Heap {
                reference_count: 2,
                // The same eight bytes the address screen would refuse if it read
                // them as one.
                heap_id: 0x400u64.to_le_bytes(),
            },
        };
        let moved = InvalidatedAddresses {
            removed: Vec::new(),
            moved: vec![0x400],
            base: BaseAddress::ZERO,
        };
        screen_shared_message_records(&[heap], &moved).unwrap();
    }

    /// A version 1 reference to a committed object becomes the version 2 form
    /// this crate writes everywhere else, naming the same address: the whole
    /// point of rewrapping is that the message still resolves to one copy.
    #[test]
    fn a_version_1_reference_is_rewrapped_as_the_modern_committed_form() {
        let mut v1 = vec![1u8, 0];
        v1.extend_from_slice(&[0u8; 6]); // reserved
        v1.extend_from_slice(&0x1111u64.to_le_bytes()); // local heap address
        v1.extend_from_slice(&0x5678u64.to_le_bytes()); // object header address

        let modern = modernize_shared_reference(&v1, OFFSET_SIZE, LENGTH_SIZE).unwrap();
        assert_eq!(modern[0], 2, "the version this crate writes");
        assert_eq!(
            crate::shared_message::parse_shared_ref(&modern, OFFSET_SIZE, LENGTH_SIZE)
                .unwrap()
                .location,
            crate::shared_message::SharedLocation::ObjectHeader(StoredAddress::new(0x5678))
        );
    }

    /// A reference into the shared-message heap keeps its heap ID and the only
    /// version that encodes one. Re-encoding it as a committed reference would
    /// turn eight bytes of heap ID into a file address.
    #[test]
    fn a_heap_reference_is_rewrapped_as_a_heap_reference() {
        let mut v3 = vec![3u8, 1];
        v3.extend_from_slice(&[9, 8, 7, 6, 5, 4, 3, 2]);

        let modern = modernize_shared_reference(&v3, OFFSET_SIZE, LENGTH_SIZE).unwrap();
        assert_eq!(modern, v3);
    }

    /// The rewrapped record keeps the shared flag. Without it every reader
    /// decodes the reference as the attribute it stands for.
    #[test]
    fn a_rewrapped_shared_message_keeps_its_flag() {
        let mut region = OhRegion::empty(OhHeaderProps::PLAIN);
        region.push_shared(MessageType::ATTRIBUTE, &[3, 1, 0, 0, 0, 0, 0, 0, 0, 0]);
        assert_eq!(region[3], MessageFlags::SHARED.get());
        assert!(region_has_shared_attr(&region).unwrap());
    }

    #[test]
    fn reject_foreign_addresses_refuses_any_shared_message() {
        // A shared (SOHM) message of *any* type — here a Dataspace — stores a
        // source-file reference in place of its body, so a verbatim cross-file
        // copy must refuse it, not only shared datatypes/attributes. (A plain,
        // non-shared dataspace embeds no foreign address and is accepted.)
        let mut shared = message_record(MessageType::DATASPACE, &[0u8; 8]);
        shared[3] = MessageFlags::SHARED.get(); // set the message's shared flag
        let err = reject_foreign_addresses(&plain_region(shared)).unwrap_err();
        assert!(err.to_string().contains("shared"), "got: {err}");

        let plain = message_record(MessageType::DATASPACE, &[0u8; 8]);
        reject_foreign_addresses(&plain_region(plain)).unwrap();
    }

    // ---- version 2 object-header record layout (issue #416) ----

    /// The layout of a header that tracks *and* indexes attribute creation
    /// order: object-header flag bits 2 and 3, and 6-byte message records.
    const TRACKED: OhRecordLayout = OhRecordLayout::from_header_flags(0x0C);

    /// One compact Attribute message body for `name`.
    fn attr_body(name: &str, value: i64) -> Vec<u8> {
        crate::type_builders::build_attr_message(name, &AttrValue::I64(value))
            .serialize(LENGTH_SIZE)
    }

    /// [`attr_region_in`] for a header carrying neither optional prefix block.
    fn attr_region(layout: OhRecordLayout, attrs: &[(&str, u16)], max: u16) -> OhRegion {
        attr_region_in(OhHeaderProps::with_layout(layout), attrs, max)
    }

    /// A region belonging to a header with `props`, holding one Attribute message
    /// per `(name, creation index)`, preceded by the Attribute Info message a
    /// tracked object carries.
    fn attr_region_in(props: OhHeaderProps, attrs: &[(&str, u16)], max: u16) -> OhRegion {
        let layout = props.layout;
        let mut region = OhRegion::empty(props);
        let info = AttributeInfoMessage {
            max_creation_index: layout.tracks_creation_order().then_some(max),
            indexes_creation_order: layout.indexes_creation_order(),
            fractal_heap_address: None,
            btree_name_index_address: None,
            btree_creation_order_address: None,
        };
        let mut record = layout.record(MessageType::ATTRIBUTE_INFO, &info.serialize(OFFSET_SIZE));
        // The reference C library sets this flag on the Attribute Info message it writes
        // (`H5Oattribute.c`, HDF5 2.2.0), so a rewrite of that message has to keep it.
        record[3] = MessageFlags::FORBID_SHARING.get();
        region.push_bytes(&record);
        for (name, index) in attrs {
            let record = layout.record_with_creation_index(
                MessageType::ATTRIBUTE,
                &attr_body(name, 1),
                *index,
            );
            region.push_bytes(&record);
        }
        region
    }

    /// The attribute names a region carries, in the order its records hold them,
    /// each with the creation index its record declares.
    fn walk_attrs(region: &OhRegion) -> Vec<(String, Option<u16>)> {
        let mut out = Vec::new();
        let mut p = 0;
        while let Some((msg_type, body, body_end)) = region.next_message(p).unwrap() {
            if msg_type == MessageType::ATTRIBUTE {
                let name = parse_compact_attr_name(region, p, body, body_end).unwrap();
                out.push((name, region.creation_index(p)));
            }
            p = body_end;
        }
        out
    }

    /// The message walk steps by the width the header's flags declare: four
    /// bytes of record prefix without creation-order tracking, six with it.
    #[test]
    fn a_record_walk_steps_by_the_width_the_header_declares() {
        for layout in [OhRecordLayout::PLAIN, TRACKED] {
            let region = attr_region(layout, &[("alpha", 0), ("beta", 7)], 8);
            assert_eq!(
                walk_attrs(&region),
                vec![
                    (
                        "alpha".to_string(),
                        layout.tracks_creation_order().then_some(0)
                    ),
                    (
                        "beta".to_string(),
                        layout.tracks_creation_order().then_some(7)
                    ),
                ],
                "layout {layout:?}",
            );
        }

        // Non-vacuity: the two widths are not interchangeable. Reading a tracked
        // region four bytes at a time misparses it from the first record on,
        // which is the defect this width exists to prevent.
        let tracked = attr_region(TRACKED, &[("alpha", 0), ("beta", 7)], 8);
        let misread = OhRegion::new(tracked.to_vec(), OhHeaderProps::PLAIN);
        let mut p = 0;
        let mut agreed = true;
        while let Ok(Some((_, _, body_end))) = misread.next_message(p) {
            p = body_end;
        }
        agreed &= p == tracked.len();
        assert!(
            !agreed,
            "a four-byte walk of a six-byte-record region must not come out even"
        );
    }

    /// An emitted record carries the creation index only where the layout has
    /// the field, and the flags byte stays where every reader looks for it.
    #[test]
    fn an_emitted_record_carries_a_creation_index_only_where_the_header_tracks_one() {
        let body = [0xABu8; 5];
        let plain =
            OhRecordLayout::PLAIN.record_with_creation_index(MessageType::ATTRIBUTE, &body, 9);
        assert_eq!(plain.len(), 4 + body.len());
        assert_eq!(&plain[4..], &body);

        let tracked = TRACKED.record_with_creation_index(MessageType::ATTRIBUTE, &body, 0x1234);
        assert_eq!(tracked.len(), 6 + body.len());
        assert_eq!(
            &tracked[..4],
            &plain[..4],
            "type, size and flags are shared"
        );
        assert_eq!(&tracked[4..6], &0x1234u16.to_le_bytes());
        assert_eq!(&tracked[6..], &body);

        // Every non-attribute message the reference C library writes carries a
        // zero creation index, which is what the plain emitter passes.
        let group_info = TRACKED.record(MessageType::GROUP_INFO, &body);
        assert_eq!(&group_info[4..6], &0u16.to_le_bytes());
    }

    /// A rebuilt header declares the record layout its bytes are written in, so
    /// re-reading it walks them the same way.
    #[test]
    fn a_rebuilt_header_declares_the_record_layout_it_used() {
        for layout in [OhRecordLayout::PLAIN, TRACKED] {
            let region = attr_region(layout, &[("alpha", 3)], 4);
            let header = build_v2_object_header(&region).unwrap();
            let (start, end, read_back) =
                oh_region_at(&header, 0, header.len() as u64).expect("the header parses");
            assert_eq!(
                read_back,
                OhHeaderProps::with_layout(layout),
                "the flags lost the layout"
            );
            let round_tripped =
                OhRegion::new(header[start as usize..end as usize].to_vec(), read_back);
            assert_eq!(
                walk_attrs(&round_tripped),
                walk_attrs(&region),
                "layout {layout:?}",
            );
        }
    }

    // ---- the optional prefix blocks of a version 2 object header (PR #422) ----

    /// Timestamps distinct enough that a field read at the wrong offset — or one
    /// copied from a neighbour — shows up as a different number.
    const FIXTURE_TIMES: ObjectTimes = ObjectTimes {
        access: 0x1111_1111,
        modification: 0x2222_2222,
        change: 0x3333_3333,
        birth: 0x4444_4444,
    };

    /// A phase-change pair the reference C library would never write by default
    /// (its defaults are 8 and 6), so a rebuild that dropped the block and a
    /// rebuild that substituted the defaults both fail.
    const FIXTURE_PHASE: AttrPhaseChange = AttrPhaseChange {
        max_compact: 32,
        min_dense: 24,
    };

    /// Hand-assemble the version 2 object header `props` describes around
    /// `region`'s bytes, with a 1-byte chunk-0 size field.
    ///
    /// Written out here rather than taken from [`build_v2_object_header`] so the
    /// parse is checked against bytes the emitter did not produce: an emitter and
    /// a parser that agreed on a wrong prefix layout would round-trip perfectly.
    fn v2_header_bytes(props: OhHeaderProps, region: &OhRegion) -> Vec<u8> {
        let mut buf = Vec::new();
        buf.extend_from_slice(b"OHDR");
        buf.push(2); // version
        buf.push(props.header_flags()); // size flags 0: a 1-byte length field
        if let Some(times) = props.times {
            buf.extend_from_slice(&times.to_bytes());
        }
        if let Some(phase) = props.attr_phase_change {
            buf.extend_from_slice(&phase.to_bytes());
        }
        buf.push(u8::try_from(region.len()).expect("the fixture region is under 256 bytes"));
        buf.extend_from_slice(region);
        let checksum = jenkins_lookup3(&buf);
        buf.extend_from_slice(&checksum.to_le_bytes());
        buf
    }

    /// Every combination of the two optional prefix blocks and the record layout:
    /// the parse finds the message region, and a rebuild puts both blocks back
    /// with the flag bits that announce them.
    ///
    /// A dropped block is not a lost field alone — the chunk-0 size field sits
    /// *after* both, so the message walk is checked too.
    #[test]
    fn a_headers_optional_prefix_blocks_survive_a_rebuild() {
        for layout in [OhRecordLayout::PLAIN, TRACKED] {
            for times in [None, Some(FIXTURE_TIMES)] {
                for attr_phase_change in [None, Some(FIXTURE_PHASE)] {
                    let props = OhHeaderProps {
                        layout,
                        times,
                        attr_phase_change,
                    };
                    let region = attr_region_in(props, &[("alpha", 3), ("beta", 5)], 6);
                    let header = v2_header_bytes(props, &region);

                    let (start, end, read_back) = oh_region_at(&header, 0, header.len() as u64)
                        .expect("the hand-built header parses");
                    assert_eq!(read_back, props, "the prefix lost a block");
                    let parsed =
                        OhRegion::new(header[start as usize..end as usize].to_vec(), read_back);
                    assert_eq!(
                        walk_attrs(&parsed),
                        walk_attrs(&region),
                        "the message region was located wrongly for {props:?}",
                    );

                    let rebuilt = build_v2_object_header(&parsed).unwrap();
                    let (_, _, again) = oh_region_at(&rebuilt, 0, rebuilt.len() as u64)
                        .expect("the rebuilt header parses");
                    assert_eq!(
                        again.attr_phase_change, attr_phase_change,
                        "the rebuild lost the attribute phase-change thresholds",
                    );
                    assert_eq!(
                        again.times.is_some(),
                        times.is_some(),
                        "the rebuild changed whether the header stores times",
                    );
                    assert_eq!(again.layout, layout, "the rebuild lost the record layout");
                }
            }
        }
    }

    /// The flag bits the emitter sets are the ones the format assigns: bit 4 for
    /// the phase-change block, bit 5 for the timestamps.
    #[test]
    fn the_optional_blocks_are_announced_by_their_own_flag_bits() {
        let with_times = OhHeaderProps {
            times: Some(FIXTURE_TIMES),
            ..OhHeaderProps::PLAIN
        };
        let with_phase = OhHeaderProps {
            attr_phase_change: Some(FIXTURE_PHASE),
            ..OhHeaderProps::PLAIN
        };
        assert_eq!(OhHeaderProps::PLAIN.header_flags(), 0);
        assert_eq!(with_times.header_flags(), 0x20);
        assert_eq!(with_phase.header_flags(), 0x10);
        assert_eq!(OhHeaderProps::PLAIN.optional_len(), 0);
        assert_eq!(with_times.optional_len(), 16);
        assert_eq!(with_phase.optional_len(), 4);
    }

    /// A rebuild of a header that stores times moves the modification and change
    /// times to now and leaves the access and birth times alone.
    #[cfg(feature = "std")]
    #[test]
    fn a_rebuild_stamps_the_modification_and_change_times() {
        let props = OhHeaderProps {
            times: Some(FIXTURE_TIMES),
            ..OhHeaderProps::PLAIN
        };
        let region = attr_region_in(props, &[("alpha", 0)], 1);

        let before = unix_time_now().expect("a std test build reads the clock");
        let header = build_v2_object_header(&region).unwrap();
        let after = unix_time_now().expect("a std test build reads the clock");

        let (_, _, read_back) = oh_region_at(&header, 0, header.len() as u64).unwrap();
        let times = read_back.times.expect("the header still stores times");
        assert_eq!(
            (times.access, times.birth),
            (FIXTURE_TIMES.access, FIXTURE_TIMES.birth),
            "a rewrite is not an access and not a birth",
        );
        assert!(
            (before..=after).contains(&times.modification),
            "the modification time {} is outside [{before}, {after}]",
            times.modification,
        );
        assert!(
            (before..=after).contains(&times.change),
            "the change time {} is outside [{before}, {after}]",
            times.change,
        );
    }

    /// A prefix that declares a block it does not carry is a malformed file, not
    /// a panic: the parse reads only what the buffer holds.
    #[test]
    fn a_prefix_truncated_inside_an_optional_block_is_refused() {
        let props = OhHeaderProps {
            times: Some(FIXTURE_TIMES),
            attr_phase_change: Some(FIXTURE_PHASE),
            ..OhHeaderProps::PLAIN
        };
        let region = attr_region_in(props, &[("alpha", 0)], 1);
        let header = v2_header_bytes(props, &region);
        // Every prefix length short of the chunk-0 size field, which is the last
        // thing the parse reads: 6 bytes of signature and flags, then 20 bytes of
        // optional blocks.
        for cut in 6..6 + 16 + 4 + 1 {
            assert!(
                oh_region_at(&header[..cut], 0, header.len() as u64).is_err(),
                "a {cut}-byte prefix must not parse as a whole header",
            );
        }
    }

    /// A fresh attribute takes the object's next unused creation index, and the
    /// Attribute Info message records the one after it — the counter the
    /// reference C library hands out from.
    #[test]
    fn a_new_compact_attribute_takes_the_next_creation_index() {
        let region = attr_region(TRACKED, &[("alpha", 0), ("beta", 1)], 2);
        let out = set_attr_in_region(&region, "gamma", &AttrValue::I64(5)).unwrap();
        assert_eq!(
            walk_attrs(&out),
            vec![
                ("alpha".to_string(), Some(0)),
                ("beta".to_string(), Some(1)),
                ("gamma".to_string(), Some(2)),
            ],
        );
        let (record, _, info) = find_attribute_info(&out).unwrap().expect("an info message");
        assert_eq!(info.max_creation_index, Some(3));
        assert!(
            info.indexes_creation_order,
            "the index flag survives a rewrite"
        );
        assert_eq!(
            out[record + 3],
            MessageFlags::FORBID_SHARING.get(),
            "the rewritten record lost the message flags it carried"
        );
    }

    /// Overwriting an attribute keeps the creation index it had, so an
    /// iteration by creation order does not reorder it.
    #[test]
    fn an_overwritten_compact_attribute_keeps_its_creation_index() {
        let region = attr_region(TRACKED, &[("alpha", 0), ("beta", 1)], 2);
        assert_eq!(attr_creation_index(&region, "alpha").unwrap(), Some(0));
        let out = set_attr_in_region(&region, "alpha", &AttrValue::I64(9)).unwrap();
        // The rewritten message moves to the end of the region — the editor
        // rebuilds rather than patches — but its creation index does not move.
        assert_eq!(
            walk_attrs(&out),
            vec![
                ("beta".to_string(), Some(1)),
                ("alpha".to_string(), Some(0))
            ],
        );
        let (_, _, info) = find_attribute_info(&out).unwrap().expect("an info message");
        assert_eq!(
            info.max_creation_index,
            Some(2),
            "an overwrite must not advance the counter"
        );
    }

    /// A deletion leaves a gap and does not lower the recorded maximum, so the
    /// next attribute created cannot land in the middle of the order.
    #[test]
    fn a_deletion_leaves_the_creation_index_counter_where_it_was() {
        let region = attr_region(TRACKED, &[("alpha", 0), ("beta", 1), ("gamma", 2)], 3);
        let after = remove_attr_from_region(&region, "gamma", true).unwrap();
        assert_eq!(
            walk_attrs(&after),
            vec![
                ("alpha".to_string(), Some(0)),
                ("beta".to_string(), Some(1))
            ],
        );
        assert_eq!(
            next_attr_creation_index(&after).unwrap(),
            3,
            "the highest index still in use is 1, but the counter is 3"
        );

        let out = put_attr_message(&after, "delta", &attr_body("delta", 1), None).unwrap();
        assert_eq!(attr_creation_index(&out, "delta").unwrap(), Some(3));
    }

    /// A header that tracks the order but carries no Attribute Info message —
    /// which nothing this crate writes, and nothing the C library writes, but
    /// which the format permits — gets one recording a counter derived from the
    /// records themselves.
    #[test]
    fn a_tracked_header_without_an_info_message_gains_one() {
        let mut region = OhRegion::empty(OhHeaderProps::with_layout(TRACKED));
        for (name, index) in [("alpha", 0u16), ("beta", 4)] {
            let record = TRACKED.record_with_creation_index(
                MessageType::ATTRIBUTE,
                &attr_body(name, 1),
                index,
            );
            region.push_bytes(&record);
        }
        assert!(find_attribute_info(&region).unwrap().is_none());
        ensure_attribute_info(&mut region).unwrap();
        let (_, _, info) = find_attribute_info(&region)
            .unwrap()
            .expect("an info message");
        assert_eq!(info.max_creation_index, Some(5));
        assert!(info.indexes_creation_order);
    }

    /// Build a compact data-layout message body: version, class=0, 2-byte inline
    /// size, then the data.
    fn compact_layout_body(version: u8, data: &[u8]) -> Vec<u8> {
        let mut b = vec![version, 0];
        b.extend_from_slice(&(data.len() as u16).to_le_bytes());
        b.extend_from_slice(data);
        b
    }

    #[test]
    fn rebuild_compact_layout_replaces_inline_data_only() {
        // A region with a Dataspace message, a compact Data Layout, and a trailing
        // Attribute message: rewriting the inline data must replace exactly the
        // layout's bytes and leave every other message verbatim.
        let mut region = message_record(MessageType::DATASPACE, &[0xAB; 8]);
        region.extend_from_slice(&message_record(
            MessageType::DATA_LAYOUT,
            &compact_layout_body(3, &[1, 2, 3, 4]),
        ));
        region.extend_from_slice(&message_record(MessageType::ATTRIBUTE, &[0xCD; 5]));

        let out = rebuild_compact_layout_region(&plain_region(region), &[9, 8, 7, 6]).unwrap();

        // Same messages in the same order; only the layout's inline data changed.
        assert_eq!(
            region_types(&out),
            vec![
                MessageType::DATASPACE,
                MessageType::DATA_LAYOUT,
                MessageType::ATTRIBUTE,
            ]
        );
        let mut p = 0;
        while let Some((mt, body, end)) = out.next_message(p).unwrap() {
            match mt {
                MessageType::DATASPACE => assert_eq!(&out[body..end], &[0xAB; 8]),
                MessageType::DATA_LAYOUT => {
                    assert_eq!(out[body], 3, "version preserved");
                    assert_eq!(out[body + 1], 0, "still compact");
                    let size = u16::from_le_bytes([out[body + 2], out[body + 3]]) as usize;
                    assert_eq!(size, 4);
                    assert_eq!(&out[body + 4..body + 4 + size], &[9, 8, 7, 6]);
                }
                MessageType::ATTRIBUTE => assert_eq!(&out[body..end], &[0xCD; 5]),
                other => panic!("unexpected message {other:?}"),
            }
            p = end;
        }
    }

    #[test]
    fn rebuild_compact_layout_refuses_non_compact() {
        // A contiguous (class 1) data layout is not compact, so the rebuild refuses
        // rather than corrupt it.
        let mut region = message_record(MessageType::DATA_LAYOUT, &{
            let mut b = vec![3u8, 1]; // version 3, class 1 (contiguous)
            b.extend_from_slice(&0u64.to_le_bytes());
            b.extend_from_slice(&0u64.to_le_bytes());
            b
        });
        region.extend_from_slice(&message_record(MessageType::DATASPACE, &[0; 8]));
        let err = rebuild_compact_layout_region(&plain_region(region), &[1, 2]).unwrap_err();
        assert!(err.to_string().contains("non-compact"), "got: {err}");
    }

    #[test]
    fn a_refused_open_never_builds_the_image() {
        // The image is what may cost `O(file size)` — the mirror reads the whole
        // file — so every refusal has to come first. Asserting on the build
        // closure states that directly; measuring memory or elapsed time would
        // only correlate with it.
        use crate::writer::FileBuilder;

        let dir = tempfile::tempdir().unwrap();

        // One refusal from each family: the superblock's status flags (issue
        // #245), and an unsupported superblock version, which has always been
        // refused after the read this reorders.
        let flagged = dir.path().join("flagged.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("d").with_i32_data(&[1, 2, 3]);
        b.write(&flagged).unwrap();
        let ancient = dir.path().join("ancient.h5");
        std::fs::copy(&flagged, &ancient).unwrap();
        for (path, version, flags) in [(&flagged, 3, SWMR_WRITE_FLAGS), (&ancient, 9, 0)] {
            let mut data = std::fs::read(path).unwrap();
            let off = signature::find_signature(&data).unwrap();
            let mut sb = Superblock::parse(&data, off).unwrap();
            sb.version = version;
            sb.consistency_flags = flags;
            let bytes = sb.serialize();
            data[off..off + bytes.len()].copy_from_slice(&bytes);
            std::fs::write(path, &data).unwrap();
        }

        for path in [&flagged, &ancient] {
            let built = std::cell::Cell::new(false);
            let err = match WriteEngine::open_imaged(path, Some(FileLocking::Enabled), |h, len| {
                built.set(true);
                Ok(Box::new(HandleImage::new(
                    h,
                    len,
                    MetadataCacheConfig::disabled(),
                )))
            }) {
                Err(e) => e,
                Ok(_) => panic!("{} must be refused", path.display()),
            };
            assert!(
                !built.get(),
                "{} was refused with {err:?}, but the image was built first",
                path.display()
            );
        }
    }

    #[test]
    fn a_stale_consistency_flag_is_refused_then_cleared_by_a_commit() {
        // A v3 file a crashed SWMR writer left flagged is refused by the editor
        // (issue #245) rather than edited under a writer the file still records.
        // On a v2 file, where the check is gated off to match the C library, the
        // editor opens — and the commit clears the stale flag rather than
        // re-emitting it, so the file stays properly closed for the C library
        // (issue #73).
        use crate::writer::FileBuilder;

        let dir = tempfile::tempdir().unwrap();
        let path = dir.path().join("stale_flag.h5");

        let mut b = FileBuilder::new();
        b.create_dataset("d").with_i32_data(&[1, 2, 3]);
        b.write(&path).unwrap();

        // Simulate a crashed SWMR writer by stamping the on-disk write+SWMR flag
        // (0x05) into the superblock, recomputing its checksum.
        {
            let mut data = std::fs::read(&path).unwrap();
            let off = signature::find_signature(&data).unwrap();
            let mut sb = Superblock::parse(&data, off).unwrap();
            assert!(
                sb.version >= 2,
                "FileBuilder should emit a v2/v3 superblock"
            );
            sb.consistency_flags = 0x05;
            let bytes = sb.serialize();
            data[off..off + bytes.len()].copy_from_slice(&bytes);
            std::fs::write(&path, &data).unwrap();
            // Sanity: the stale flag is really set on disk now.
            assert_eq!(
                Superblock::parse(&data, off).unwrap().consistency_flags,
                0x05
            );
        }

        // The editor refuses it while the flag stands.
        match WriteEngine::open_with_locking(&path, FileLocking::Enabled) {
            Err(Error::FileMarkedInUse(_)) => {}
            Err(e) => panic!("expected the flag refusal, got {e:?}"),
            Ok(_) => panic!("a flagged file must not be edited in place"),
        }

        // The flag survives a *version-2* superblock, where the check is gated
        // off to match the C library — which is the one state that still carries
        // a stale flag into a commit, and so the one that keeps the healing below
        // load-bearing. (v2 and v3 superblocks share a byte layout, so restamping
        // the version is the whole difference.) A crashed C writer leaves plain
        // write access, without the SWMR bit.
        {
            let mut data = std::fs::read(&path).unwrap();
            let off = signature::find_signature(&data).unwrap();
            let mut sb = Superblock::parse(&data, off).unwrap();
            sb.version = 2;
            sb.consistency_flags = crate::file_lock::WRITE_ACCESS;
            let bytes = sb.serialize();
            data[off..off + bytes.len()].copy_from_slice(&bytes);
            std::fs::write(&path, &data).unwrap();
        }

        // A clean edit-and-commit cycle heals it.
        {
            let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled)
                .expect("the gate skips a v2 superblock, so this opens");
            let mut b = DatasetBuilder::new("e");
            b.with_i32_data(&[4, 5]);
            s.stage_created_dataset(&ObjectPathBuf::parse("e"), b)
                .unwrap();
            s.commit().unwrap();
        }

        let data = std::fs::read(&path).unwrap();
        let off = signature::find_signature(&data).unwrap();
        assert_eq!(
            Superblock::parse(&data, off).unwrap().consistency_flags,
            0,
            "commit must clear the stale consistency flag"
        );
    }

    #[test]
    fn add_vlen_string_dataset_with_null_elements_via_edit_session() {
        // Regression test for a silent-corruption bug (issue #105): a
        // VL-string dataset added via the in-place edit engine used to commit `Ok(())`
        // without ever writing its global heap collection or patching its
        // placeholder references, so the dataset failed to read back. A null
        // element (no heap object at all, distinct from an empty string) must
        // stay untouched by the patch — only heap-backed elements'
        // placeholder addresses are resolved; exercising both keeps the mask
        // itself, not just the common all-`Bytes` case, under test.
        use crate::type_builders::VlStringElement;
        use crate::writer::FileBuilder;

        let dir = tempfile::tempdir().unwrap();
        let path = dir.path().join("vlen_null.h5");

        let mut b = FileBuilder::new();
        b.create_dataset("seed").with_i32_data(&[0]);
        b.write(&path).unwrap();

        let datatype =
            crate::type_builders::make_vlen_string_type(crate::datatype::CharacterSet::Utf8);
        let elements = vec![
            VlStringElement::Bytes(b"alpha".to_vec()),
            VlStringElement::Null,
            VlStringElement::Bytes(b"gamma".to_vec()),
        ];

        {
            let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
            let mut b = DatasetBuilder::new("labels");
            b.with_vlen_string_elements(datatype, &elements).unwrap();
            s.stage_created_dataset(&ObjectPathBuf::parse("labels"), b)
                .unwrap();
            s.commit().unwrap();
        }

        let file = crate::reader::File::open(&path).unwrap();
        let ds = file.dataset("labels").unwrap();
        assert_eq!(
            ds.read_string().unwrap(),
            vec!["alpha".to_string(), String::new(), "gamma".to_string()]
        );
    }

    #[test]
    fn edit_session_root_group_base_address_overflow_is_rejected() {
        // The edit-path sibling of issue #137. A userblock file has a nonzero base
        // address that `WriteEngine::open` adds to the stored root-group address.
        // A crafted address of HADDR_UNDEF must be rejected rather than overflow
        // (panicking in debug, wrapping in release).
        use crate::writer::FileBuilder;

        let dir = tempfile::tempdir().unwrap();
        let path = dir.path().join("edit_root_overflow.h5");

        const UB: u64 = 512;
        let mut b = FileBuilder::new();
        b.with_userblock(UB);
        b.create_dataset("d").with_i32_data(&[1, 2, 3]);
        b.write(&path).unwrap();

        // Rewrite the stored (base-relative) root-group address to HADDR_UNDEF,
        // recomputing the superblock checksum via `serialize`. The base address
        // still equals the superblock offset, so the file stays editable and the
        // editor reaches the `root_group_address + base` normalization.
        let mut data = std::fs::read(&path).unwrap();
        let off = signature::find_signature(&data).unwrap();
        let mut sb = Superblock::parse(&data, off).unwrap();
        assert_eq!(
            sb.base_address,
            BaseAddress::new(UB),
            "userblock file must have base == UB"
        );
        sb.root_group_address = u64::MAX;
        let bytes = sb.serialize();
        data[off..off + bytes.len()].copy_from_slice(&bytes);
        std::fs::write(&path, &data).unwrap();

        let err = WriteEngine::open_with_locking(&path, FileLocking::Enabled)
            .err()
            .expect("open must fail");
        match err {
            Error::Format(FormatError::OffsetOverflow { offset, length }) => {
                assert_eq!(offset, u64::MAX);
                assert_eq!(length, UB);
            }
            other => panic!("expected root-group address overflow, got {other:?}"),
        }
    }

    use tempfile::tempdir;

    // -----------------------------------------------------------------------
    // Bounded sessions: the same engine over a `HandleImage`, which holds no
    // whole-file mirror. These came across from the standalone bounded engine
    // deleted in issue #198; what they cover is unchanged, but they now exercise
    // the shared code the mirror sessions use.
    // -----------------------------------------------------------------------

    /// Build a rank-1 unlimited chunked i32 dataset `d` seeded with `0..n`.
    fn build_appendable(path: &Path, n: i32, chunk: u64) {
        let data: Vec<i32> = (0..n).collect();
        let mut b = crate::writer::FileBuilder::new();
        b.create_dataset("d")
            .with_i32_data(&data)
            .with_shape(&[n as u64])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[chunk]);
        b.write(path).unwrap();
    }

    fn open_bounded_session(path: &Path) -> WriteEngine {
        WriteEngine::open_rw_with_strategy(
            path,
            crate::source::MetadataCacheConfig::disabled(),
            FileLocking::Enabled,
            MemoryStrategy::Bounded,
        )
        .unwrap()
    }

    fn dataset_addr(engine: &WriteEngine) -> u64 {
        crate::group_v2::resolve_path_any_from_source(
            &engine.image(),
            AccessMode::ReadWrite,
            engine.superblock(),
            &ObjectPath::parse("d"),
        )
        .unwrap()
    }

    /// Crash consistency on a bounded session: stop the append after only the
    /// first `max_phase` durability phases (simulating a crash at that boundary)
    /// and assert the reopened file reads either the old length (phases 1-3) or
    /// the new one (phase 4), never a torn view. Layouts cover a partial trailing
    /// chunk (relocated tail) and a chunk-aligned start.
    #[test]
    fn bounded_append_crash_consistency_partial_tail_prefix() {
        let dir = tempdir().unwrap();
        for (case, (n, chunk, add)) in [(0usize, (6i32, 4u64, 5i32)), (1, (8, 2, 6))] {
            let base = dir.path().join(std::format!("base_{case}.h5"));
            build_appendable(&base, n, chunk);
            for max_phase in 1u8..=4 {
                let p = dir.path().join(std::format!("crash_{case}_{max_phase}.h5"));
                std::fs::copy(&base, &p).unwrap();
                {
                    let mut engine = open_bounded_session(&p);
                    let addr = dataset_addr(&engine);
                    let mut b = AppendBuilder::new();
                    b.append_i32(&(n..n + add).collect::<Vec<_>>());
                    engine
                        .append_inplace_gathered(AppendTarget::Header(addr), &b, max_phase)
                        .unwrap();
                    // Dropping the engine simulates the crash: no further phases,
                    // no close barrier.
                }
                let expected_len = if max_phase == 4 { n + add } else { n };
                let got = crate::File::open(&p)
                    .unwrap()
                    .dataset("d")
                    .unwrap()
                    .read_i32()
                    .unwrap();
                assert_eq!(
                    got,
                    (0..expected_len).collect::<Vec<_>>(),
                    "case {case} phase {max_phase}"
                );
            }
        }
    }

    /// The batching loop only honors `max_phase < 4` on its first batch, and a
    /// full multi-batch append leaves every batch fully committed: after a large
    /// append the file reads the complete sequence.
    #[test]
    fn bounded_multi_batch_append_commits_every_batch() {
        let dir = tempdir().unwrap();
        let p = dir.path().join("multibatch.h5");
        build_appendable(&p, 5, 512);
        let total = 700_000i32;
        {
            let mut engine = open_bounded_session(&p);
            let addr = dataset_addr(&engine);
            let mut b = AppendBuilder::new();
            b.append_i32(&(5..total).collect::<Vec<_>>());
            engine
                .append_inplace_gathered(AppendTarget::Header(addr), &b, 4)
                .unwrap();
        }
        let got = crate::File::open(&p)
            .unwrap()
            .dataset("d")
            .unwrap()
            .read_i32()
            .unwrap();
        assert_eq!(got.len(), total as usize);
        assert!(got.iter().enumerate().all(|(i, &v)| v == i as i32));
    }

    /// A bounded session batches; a mirror session does not. The distinction is
    /// a deliberate trade — bounded peak memory against whole-call crash
    /// atomicity — so it is asserted rather than left to the batching code's
    /// arithmetic.
    #[test]
    fn only_a_bounded_session_batches_a_large_append() {
        let dir = tempdir().unwrap();
        let p = dir.path().join("batching.h5");
        build_appendable(&p, 8, 4);

        let bounded_batch = {
            let mut engine = open_bounded_session(&p);
            engine
                .append_geometry(AppendTarget::Path(&ObjectPathBuf::parse("d")))
                .unwrap()
                .full_batch_elems
        };
        let mirror_batch = {
            let mut engine = WriteEngine::open_with_locking(&p, FileLocking::Enabled).unwrap();
            engine
                .append_geometry(AppendTarget::Path(&ObjectPathBuf::parse("d")))
                .unwrap()
                .full_batch_elems
        };

        assert_eq!(
            mirror_batch,
            u64::MAX,
            "a mirror session must take the whole append as one crash-atomic batch"
        );
        assert!(
            bounded_batch < u64::MAX,
            "a bounded session must cap a batch, got {bounded_batch}"
        );
        assert_eq!(
            bounded_batch % 4,
            0,
            "a batch must be a whole number of chunks"
        );
    }

    /// A persisting file appended through a bounded session and dropped WITHOUT
    /// `finalize_persist` (the true-crash case) still reads back every durable
    /// append. Dropping the engine releases the exclusive lock, so the reopen is
    /// portable (no leaked lock). The finalize-at-close path is covered by the
    /// `tests/main/bounded_append.rs` integration tests.
    #[test]
    fn bounded_persist_append_without_finalize_is_readable() {
        let dir = tempdir().unwrap();
        let p = dir.path().join("persist_crash.h5");
        let mut b = crate::writer::FileBuilder::new();
        b.with_file_space_strategy(crate::FileSpaceStrategy::FsmAggr, true, 1);
        b.create_dataset("d")
            .with_i32_data(&(0..6).collect::<Vec<i32>>())
            .with_shape(&[6])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[4]);
        b.write(&p).unwrap();
        {
            let mut engine = open_bounded_session(&p);
            assert!(engine.persist.is_some(), "persist state is armed at open");
            let addr = dataset_addr(&engine);
            let mut ab = AppendBuilder::new();
            ab.append_i32(&(6..20).collect::<Vec<_>>());
            engine
                .append_inplace_gathered(AppendTarget::Header(addr), &ab, 4)
                .unwrap();
            // Drop without finalizing: models a true crash and releases the lock.
        }
        let got = crate::File::open(&p)
            .unwrap()
            .dataset("d")
            .unwrap()
            .read_i32()
            .unwrap();
        assert_eq!(got, (0..20).collect::<Vec<_>>());
    }

    /// A bounded session grows a PAGED persisting file and is killed before
    /// finalize (models a crash), leaving the file non-page-aligned. Reopening it
    /// must not panic, and the next append must re-align the crashed tail page
    /// before writing raw data (so no page mixes metadata and raw); a clean close
    /// then re-page-aligns the file and every row reads back.
    #[test]
    fn bounded_paged_reopen_after_crash_realigns_and_stays_readable() {
        let dir = tempdir().unwrap();
        let p = dir.path().join("paged_crash.h5");
        let mut b = crate::writer::FileBuilder::new();
        b.with_file_space_strategy(crate::FileSpaceStrategy::Page, true, 0)
            .with_file_space_page_size(4096);
        b.create_dataset("d")
            .with_i32_data(&(0..64).collect::<Vec<i32>>())
            .with_shape(&[64])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[64]);
        b.write(&p).unwrap();

        // Grow enough to force extensible-array index growth, so the last write of
        // the session is metadata and the tail page is a partial metadata page.
        {
            let mut engine = open_bounded_session(&p);
            let addr = dataset_addr(&engine);
            let mut ab = AppendBuilder::new();
            ab.append_i32(&(64..2000).collect::<Vec<_>>());
            engine
                .append_inplace_gathered(AppendTarget::Header(addr), &ab, 4)
                .unwrap();
            // Drop without finalize: models a crash and releases the OS lock.
        }
        assert_ne!(
            std::fs::metadata(&p).unwrap().len() % 4096,
            0,
            "a crashed (un-finalized) paged session leaves the file non-page-aligned"
        );

        // Reopen must not panic on the non-aligned file; the next append re-aligns
        // the crashed tail page, and finalize re-page-aligns the whole file.
        {
            let mut engine = open_bounded_session(&p);
            let addr = dataset_addr(&engine);
            let mut ab = AppendBuilder::new();
            ab.append_i32(&(2000..2500).collect::<Vec<_>>());
            engine
                .append_inplace_gathered(AppendTarget::Header(addr), &ab, 4)
                .unwrap();
            engine.finalize_persist().unwrap();
            engine.barrier().unwrap();
        }
        assert_eq!(
            std::fs::metadata(&p).unwrap().len() % 4096,
            0,
            "reopen + append + finalize re-aligns the paged file"
        );
        let got = crate::File::open(&p)
            .unwrap()
            .dataset("d")
            .unwrap()
            .read_i32()
            .unwrap();
        assert_eq!(got, (0..2500).collect::<Vec<_>>());
    }

    /// A staged commit on a bounded session must stay bounded: it may read the
    /// metadata it edits, but never the file's bulk. Measured rather than
    /// asserted from the design — the engine is shared with the mirror sessions
    /// now, and a single slice-taking read added anywhere on the commit path
    /// would silently make a bounded open cost as much as a mirrored one.
    #[test]
    fn a_bounded_commit_reads_far_less_than_the_file() {
        use std::sync::Arc;
        use std::sync::atomic::{AtomicU64, Ordering};

        let dir = tempdir().unwrap();
        let p = dir.path().join("bulk.h5");
        // ~8 MiB of chunked data, so "reads the whole file" and "reads only the
        // metadata" differ by orders of magnitude rather than by a margin.
        let rows = 2_000_000i32;
        build_appendable(&p, rows, 8192);
        let file_len = std::fs::metadata(&p).unwrap().len();
        assert!(file_len > 4 << 20, "file is only {file_len} bytes");

        let read_bytes = Arc::new(AtomicU64::new(0));
        {
            let mut engine =
                WriteEngine::open_bounded_counting(&p, Arc::clone(&read_bytes)).unwrap();
            engine.create_group(&ObjectPathBuf::parse("g")).unwrap();
            engine.commit().unwrap();
        }
        let read = read_bytes.load(Ordering::Relaxed);

        assert!(
            read > 0,
            "the commit read nothing, so the test proves nothing"
        );
        // Measured at 310 bytes here. The bound is loose enough to survive a
        // changed header layout and still orders of magnitude below the file.
        assert!(
            read < 64 << 10,
            "a bounded commit read {read} bytes of a {file_len}-byte file"
        );
    }

    /// An in-place append leaves a partially-filled **raw** page, so the next
    /// commit's metadata must pad it rather than pack into it.
    ///
    /// This is what keeps [`PagedEdit::begin`] reachable from [`EditStore`] now
    /// that an append allocates raw pages only: the append's job is to record that
    /// the tail page turned raw, and the commit's job is to act on it. It is also
    /// the interleaving that a single session-level page tracker makes possible —
    /// with a tracker per engine, the commit path could not see what the append
    /// path had done, which is why the whole-file editor refused an in-place append
    /// to a paged file at all (issue #198).
    #[test]
    fn a_commit_after_an_append_pads_the_raw_page_the_append_left() {
        const PAGE: u64 = 4096;
        let dir = tempdir().unwrap();
        let p = dir.path().join("paged_interleave.h5");
        let mut b = crate::writer::FileBuilder::new();
        b.with_file_space_strategy(crate::FileSpaceStrategy::Page, true, 0)
            .with_file_space_page_size(PAGE);
        b.create_dataset("d")
            .with_i32_data(&(0..64).collect::<Vec<i32>>())
            .with_shape(&[64])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[64]);
        b.write(&p).unwrap();

        let mut engine = WriteEngine::open_with_locking(&p, FileLocking::Enabled).unwrap();
        let mut ab = AppendBuilder::new();
        ab.append_i32(&(64..2000).collect::<Vec<_>>());
        engine
            .append_inplace_gathered(AppendTarget::Path(&ObjectPathBuf::parse("d")), &ab, 4)
            .unwrap();

        assert_eq!(
            engine.paged.as_ref().unwrap().last,
            Some(PageType::Raw),
            "the append must record that the tail page now holds raw data"
        );
        assert_ne!(
            engine.image.len() % PAGE,
            0,
            "the append must leave a partially-filled page for the commit to pad"
        );

        engine.create_group(&ObjectPathBuf::parse("g")).unwrap();
        engine.commit().unwrap();

        // The commit padded the raw tail before laying down metadata, and folded
        // that padding into the raw list (`meta_pad`/`raw_pad` are cleared into the
        // free lists as part of the paged tail).
        let pg = engine.paged.as_ref().expect("the file is paged");
        let raw_free = pg.raw.sections();
        assert!(
            !raw_free.is_empty(),
            "the commit packed metadata into the raw page the append left open"
        );
        for (addr, len) in raw_free {
            assert_eq!(
                (addr + len) % PAGE,
                0,
                "padding {addr}+{len} does not reach a page boundary"
            );
        }

        drop(engine);
        assert_eq!(
            crate::File::open(&p)
                .unwrap()
                .dataset("d")
                .unwrap()
                .read_i32()
                .unwrap(),
            (0..2000).collect::<Vec<_>>()
        );
    }

    /// An in-place append to a paged file must allocate **only raw pages** — the
    /// chunk data and the extensible-array blocks indexing it alike.
    ///
    /// The reclaim path (`chunked_storage_spans`) reports both halves of a chunked
    /// dataset as raw free space, because that is where this crate places them. An
    /// append that put its index blocks in a metadata page instead would make the
    /// reclaim advertise metadata-page bytes for raw reuse, mixing the page a paged
    /// file exists to keep homogeneous. Measured here rather than through the
    /// reference C library, which reads a mixed-page file without complaint: an
    /// interop test proves interop and says nothing about segregation.
    #[test]
    fn an_inplace_append_to_a_paged_file_allocates_only_raw_pages() {
        const PAGE: u64 = 4096;
        let dir = tempdir().unwrap();
        let p = dir.path().join("paged_raw.h5");
        let mut b = crate::writer::FileBuilder::new();
        b.with_file_space_strategy(crate::FileSpaceStrategy::Page, true, 0)
            .with_file_space_page_size(PAGE);
        b.create_dataset("d")
            .with_i32_data(&(0..64).collect::<Vec<i32>>())
            .with_shape(&[64])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[64]);
        b.write(&p).unwrap();

        let mut engine = WriteEngine::open_with_locking(&p, FileLocking::Enabled).unwrap();
        let before = engine.image().len();
        // Two appends, each large enough to grow the extensible-array index, so the
        // run allocates index blocks as well as chunk data.
        for range in [64..2000, 2000..4000] {
            let mut ab = AppendBuilder::new();
            ab.append_i32(&range.collect::<Vec<_>>());
            engine
                .append_inplace_gathered(AppendTarget::Path(&ObjectPathBuf::parse("d")), &ab, 4)
                .unwrap();
        }

        let pg = engine.paged.as_ref().expect("the file is paged");
        assert_eq!(
            pg.last,
            Some(PageType::Raw),
            "the append left the tail page holding something other than raw data"
        );
        assert!(
            pg.meta_pad.is_empty() && pg.raw_pad.is_empty(),
            "an in-place append switched page type: meta_pad={:?} raw_pad={:?}",
            pg.meta_pad,
            pg.raw_pad
        );

        // Not vacuous: the append really did allocate index structure above the
        // pre-append end-of-file, which is what would have opened a metadata page.
        let addr = crate::group_v2::resolve_path_any_from_source(
            &engine.image(),
            AccessMode::ReadWrite,
            engine.superblock(),
            &ObjectPath::parse("d"),
        )
        .unwrap();
        let spans = engine
            .chunked_storage_spans(addr)
            .expect("a chunked dataset has reclaimable spans");
        let fresh = spans.iter().filter(|&&(a, _, _)| a >= before).count();
        assert!(
            fresh > 0,
            "the append allocated nothing above {before}, so the assertion above proves nothing"
        );
        assert!(
            spans
                .iter()
                .all(|&(_, _, class)| class == FreeClass::Page(PageType::Raw)),
            "the reclaim tags every chunked span raw; a metadata or dead tag here would \
             need the placement rule above to change with it"
        );
    }

    /// Write a small file of `tables` unlimited chunked datasets, paged when
    /// asked, for the write-gathering tests below.
    fn gather_fixture(path: &std::path::Path, tables: usize, paged: bool) {
        use crate::writer::FileBuilder;
        let mut b = FileBuilder::new();
        if paged {
            // Deliberately *not* DEFAULT_GATHER_PAGE: a paged fixture at the
            // default page size cannot tell a session that reads the file's page
            // size from one that assumes the default.
            b.with_file_space_strategy(FileSpaceStrategy::Page, true, 1)
                .with_file_space_page_size(16 * 1024);
        }
        for t in 0..tables {
            b.create_dataset(&std::format!("t{t}"))
                .with_i32_data(&(0..256).collect::<Vec<_>>())
                .with_shape(&[256])
                .with_maxshape(&[MaxExtent::Unlimited])
                .with_chunks(&[64]);
        }
        b.write(path).unwrap();
    }

    /// Run the same appends and the same commit on `session`, and report what
    /// each cost in writes: the in-place appends, then the staged commit that
    /// follows them. They are counted apart because the gathering earns its keep
    /// in only one of them — see the caller. The file the workload leaves is the
    /// other half of what the callers compare, and they read it off the path
    /// themselves.
    fn gather_workload(session: &mut WriteEngine) -> (u64, u64) {
        let before = session.image.issued_writes();
        for round in 0..4 {
            for t in 0..4 {
                session
                    .append_inplace_i32_phased(&std::format!("t{t}"), &[round; 64], 4)
                    .unwrap();
            }
        }
        for t in 0..4 {
            let mut db = crate::type_builders::DatasetBuilder::new(&std::format!("n{t}"));
            db.with_f64_data(&[2.5f64; 32]).with_shape(&[32]);
            session
                .stage_created_dataset(&ObjectPathBuf::parse(&std::format!("/n{t}")), db)
                .unwrap();
        }
        let after_appends = session.image.issued_writes();
        session.commit().unwrap();
        (
            after_appends - before,
            session.image.issued_writes() - after_appends,
        )
    }

    /// Gathering a session's writes lowers what it costs and changes nothing
    /// about what it produces (issue #288).
    ///
    /// Both halves matter and neither implies the other. A gatherer that dropped
    /// a run, wrote one twice in the wrong order, or filled the space between two
    /// runs sharing a page with zeros would lower the count exactly as well — so
    /// the two files are compared **byte for byte**, which is the only assertion
    /// a wrong merge cannot pass. And a gatherer that merged nothing would keep
    /// them identical, which is what the count is for.
    ///
    /// The comparison is against this same engine with the gathering turned off,
    /// rather than against a recorded number: how many writes a commit needs is
    /// an implementation detail that should be free to fall further.
    #[test]
    fn gathering_writes_costs_fewer_of_them_and_changes_no_byte() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        for paged in [false, true] {
            let straight = dir.path().join(std::format!("straight_{paged}.h5"));
            let gathered = dir.path().join(std::format!("gathered_{paged}.h5"));
            gather_fixture(&straight, 4, paged);
            gather_fixture(&gathered, 4, paged);

            let mut a = WriteEngine::open_with_locking(&straight, FileLocking::Enabled).unwrap();
            a.set_sync_policy(SyncPolicy::OnClose);
            a.image
                .set_write_buffering(WriteBuffering::Unbuffered)
                .unwrap();
            let (straight_appends, straight_commit) = gather_workload(&mut a);
            a.force_sync().unwrap();
            drop(a);

            let mut b = WriteEngine::open_with_locking(&gathered, FileLocking::Enabled).unwrap();
            b.set_sync_policy(SyncPolicy::OnClose);
            let (gathered_appends, gathered_commit) = gather_workload(&mut b);
            b.force_sync().unwrap();
            drop(b);

            // The commit tail is where the gathering earns its keep, and the half
            // to assert a ratio on. Measured: 2 writes against 10 unpaged, 4
            // against 16 paged. A commit rebuilds a group, repoints a root and
            // re-homes the free-space managers, all inside one phase and all into
            // a handful of pages, which is what merging within a barrier is for.
            // Merging *across* barriers goes further and is what an explicit page
            // buffer does; `a_page_buffer_holds_dirty_pages_across_operations`
            // measures that.
            assert!(
                gathered_commit * 3 < straight_commit,
                "paged={paged}: gathering must cost meaningfully fewer writes for a \
                 commit, but cost {gathered_commit} against {straight_commit}"
            );
            // The appends are the other half, and since issue #307 they are a
            // near-tie: 88 against 92 unpaged, 92 against 92 paged. Publishing a
            // checksummed structure is one write from the engine now rather than
            // two the gatherer had to rejoin, so the buffering has almost nothing
            // left to merge here — before that fix this half was 92 against 144.
            // What is still worth pinning is that it never costs *more*. Note the
            // limit of that: a publish write made wider still merges the same way,
            // so widening one to the whole structure it sits in changes no count
            // here or anywhere — `a_publish_writes_from_the_byte_it_changed` is
            // what holds that, by counting bytes rather than writes.
            assert!(
                gathered_appends <= straight_appends,
                "paged={paged}: gathering must not cost more writes for the appends, \
                 but cost {gathered_appends} against {straight_appends}"
            );
            assert_eq!(
                std::fs::read(&straight).unwrap(),
                std::fs::read(&gathered).unwrap(),
                "paged={paged}: gathering changed the file it produced"
            );
        }
    }

    /// A session merges writes within the page its *file* was laid out on, and
    /// falls back to the format default only when the file is not paged.
    ///
    /// Asserted directly because the fixtures cannot assert it indirectly: the
    /// merge quantum is only visible in which writes coalesce, and any fixture
    /// built at the default page size makes the two answers identical. Reading
    /// the file's page size is the whole point of resolving this after the open
    /// rather than at it.
    #[test]
    fn the_merge_page_follows_the_file_rather_than_the_default() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let paged = dir.path().join("paged.h5");
        let plain = dir.path().join("plain.h5");
        gather_fixture(&paged, 1, true);
        gather_fixture(&plain, 1, false);

        let s = WriteEngine::open_with_locking(&paged, FileLocking::Enabled).unwrap();
        assert_eq!(
            s.gather_page_size(),
            16 * 1024,
            "a paged file merges within its own file-space page"
        );
        assert_ne!(
            16 * 1024,
            DEFAULT_GATHER_PAGE,
            "the fixture must not be built at the default, or the assertion above \
             holds for a session that ignores the file entirely"
        );
        drop(s);

        let s = WriteEngine::open_with_locking(&plain, FileLocking::Enabled).unwrap();
        assert_eq!(
            s.gather_page_size(),
            DEFAULT_GATHER_PAGE,
            "an unpaged file has no page size of its own"
        );
    }

    /// Writing the status-flags byte on a file with a userblock changes that byte
    /// and nothing else.
    ///
    /// `set_consistency_flags` serializes the whole superblock to rewrite one
    /// byte of it, and the root address it carries is held absolute in memory but
    /// stored relative to the base address. Getting that wrong repoints the root
    /// past the end of the file, so a call that promised to touch a flag silently
    /// makes the file unreadable.
    ///
    /// Driven directly rather than through a caller, because for a long time
    /// neither caller could reach a non-zero base: SWMR refuses a userblock
    /// outright, and a page buffer required persisted free space, which is
    /// declined for a non-zero base. Both were refusals about something else that
    /// settled this as a side effect — and the day one of them moved is the day a
    /// test routed through it would have gone quiet. Issue #357 moved the second,
    /// so `a_page_buffer_marks_a_userblock_file_without_repointing_its_root`
    /// exercises the same conversion through a real session; this one stays as
    /// the direct statement of the rule.
    #[test]
    fn a_status_flag_write_leaves_a_userblock_files_root_alone() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("userblock.h5");
        let mut b = FileBuilder::new();
        b.with_userblock(4096);
        b.create_dataset("d")
            .with_i32_data(&[1, 2, 3, 4])
            .with_shape(&[4]);
        b.write(&path).unwrap();
        let before = std::fs::read(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        assert_eq!(
            s.superblock.base_address,
            BaseAddress::new(4096),
            "the fixture must have a userblock, or this test holds for a file \
             whose absolute and relative roots are the same number"
        );
        // Up and back down, which is the sequence a marked session performs.
        s.set_consistency_flags(file_lock::WRITE_ACCESS).unwrap();
        s.set_consistency_flags(0).unwrap();
        s.force_sync().unwrap();
        drop(s);

        assert_eq!(
            std::fs::read(&path).unwrap(),
            before,
            "raising and clearing the flags must leave the file as it was"
        );
        assert_eq!(
            crate::reader::File::open(&path)
                .unwrap()
                .dataset("d")
                .unwrap()
                .read_i32()
                .unwrap(),
            vec![1, 2, 3, 4],
            "and the file must still read"
        );
    }

    /// The budget a caller asks for is the budget the gatherer is given, so a
    /// session that outruns it flushes rather than growing without bound.
    ///
    /// `set_page_buffer_size` spends most of its length on refusals, and the one
    /// line that does the work hands `max_bytes` on. Passing `usize::MAX` instead
    /// fails nothing else in the suite: every other page-buffer test runs a
    /// workload smaller than the budget, so it holds everything to the end either
    /// way and the file it leaves is identical. What that would cost is a
    /// long-running session holding the whole of what it wrote in memory, which
    /// is the one thing the budget is for.
    ///
    /// Asserted as "some writes went out before the session ended" rather than as
    /// a count, since how many a 4 MiB append needs is free to change.
    #[test]
    fn a_page_buffered_session_flushes_when_it_outruns_its_budget() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("outrun.h5");
        gather_fixture(&path, 1, true);

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        s.set_sync_policy(SyncPolicy::OnClose);
        s.set_page_buffer_size(WRITE_GATHER_BYTES).unwrap();
        let before = s.image.issued_writes();
        // Four times the budget, in one append, so nothing but the budget can
        // account for a write going out before the teardown below.
        s.append_inplace_i32_phased("t0", &vec![7i32; 1 << 20], 4)
            .unwrap();
        let during = s.image.issued_writes() - before;
        s.force_sync().unwrap();
        s.release_status_flags().unwrap();
        drop(s);

        assert!(
            during > 0,
            "a session that wrote four times its budget issued nothing until it \
             closed, so the budget it was given is not the one it asked for"
        );
        assert_eq!(
            crate::reader::File::open(&path)
                .unwrap()
                .dataset("t0")
                .unwrap()
                .read_i32()
                .unwrap()
                .len(),
            256 + (1 << 20),
            "and the flushed append must still read back whole"
        );
    }

    /// A page-buffer budget below the 1 MiB a session already gathers under
    /// issues *more* writes — on the workload a page buffer is sold for, and on
    /// the one it is worst at.
    ///
    /// This is the price the public property quotes for a small budget, so it
    /// belongs in the suite rather than in a comment. Such a budget was refused
    /// outright until issue #391, on the strength of exactly this measurement;
    /// what changed is who decides, not what it costs. A claim about a cost that
    /// nothing checks is one that quietly stops being true, and a caller sizing a
    /// buffer against a memory cap is who pays for that.
    ///
    /// The C library charges nothing for a small buffer: `H5PB_write` sends any
    /// I/O of a page or more straight to the driver. Nothing bypasses this
    /// gatherer, so the budget is also the point at which a long run is flushed
    /// and restarted — which is why the second workload is one long run.
    ///
    /// Asserted as an **ordering** rather than as counts. The counts are
    /// deterministic on one target and are quoted on the public property, but how
    /// many writes a commit needs is free to fall; what must not change is which
    /// budget issues fewer (issues #357, #391).
    #[test]
    fn a_smaller_budget_issues_more_writes() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let fixture = |path: &std::path::Path, tables: usize| {
            use crate::writer::FileBuilder;
            let mut b = FileBuilder::new();
            b.with_file_space_strategy(FileSpaceStrategy::Page, true, 1)
                .with_file_space_page_size(4096);
            for t in 0..tables {
                b.create_dataset(&std::format!("t{t}"))
                    .with_i32_data(&(0..256).collect::<Vec<_>>())
                    .with_shape(&[256])
                    .with_maxshape(&[MaxExtent::Unlimited])
                    .with_chunks(&[64]);
            }
            b.write(path).unwrap();
        };

        // The public path, which now accepts every budget down to this fixture's
        // 4 KiB page — so the arms differ in the budget a caller asked for and in
        // nothing else.
        let run = |path: &std::path::Path, budget: usize, workload: &dyn Fn(&mut WriteEngine)| {
            let mut s = WriteEngine::open_with_locking(path, FileLocking::Enabled).unwrap();
            s.set_sync_policy(SyncPolicy::OnClose);
            s.set_page_buffer_size(budget).unwrap();
            let before = s.image.issued_writes();
            workload(&mut s);
            s.force_sync().unwrap();
            s.release_status_flags().unwrap();
            let issued = s.image.issued_writes() - before;
            drop(s);
            issued
        };

        // Many small scattered writes — what the property is for — and one long
        // contiguous run, which is where a small budget collapses.
        let workload = |scattered: bool| {
            move |s: &mut WriteEngine| {
                if scattered {
                    for round in 0..4 {
                        for t in 0..8 {
                            s.append_inplace_i32_phased(&std::format!("t{t}"), &[round; 64], 4)
                                .unwrap();
                        }
                    }
                } else {
                    s.append_inplace_i32_phased("t0", &vec![7i32; 1 << 20], 4)
                        .unwrap();
                }
            }
        };

        for (label, tables, scattered) in [("scattered", 8, true), ("one long run", 1, false)] {
            let workload = workload(scattered);
            let at = |budget: usize| {
                let path = dir
                    .path()
                    .join(std::format!("{}_{budget}.h5", label.replace(' ', "_")));
                fixture(&path, tables);
                run(&path, budget, &workload)
            };
            let ample = at(WRITE_GATHER_BYTES);
            let smallest = at(4096);
            for smaller in [4096, 64 * 1024] {
                let below = at(smaller);
                assert!(
                    ample <= below,
                    "{label}: a {smaller}-byte budget issued {below} writes against \
                     {ample} at {WRITE_GATHER_BYTES}, so a smaller budget is not the \
                     trade the property documents"
                );
            }
            // Not vacuous: a gatherer that ignored `max_bytes` outright would
            // report the same count at every budget and satisfy the ordering
            // above without honoring any of them. One page is a quarter of the
            // smallest step measured here, so a workload that does not separate
            // there is one this assertion cannot speak for.
            assert!(
                smallest > ample,
                "{label}: the budget changed nothing — {smallest} writes at 4 KiB \
                 against {ample} at 1 MiB — so this workload cannot say which \
                 budget is better"
            );
        }
    }

    /// A page-buffered session on a file with a userblock raises and clears its
    /// crash mark without disturbing the root address that mark's superblock
    /// rewrite carries — and the userblock's own bytes stay put.
    ///
    /// The caller-side half of
    /// `a_status_flag_write_leaves_a_userblock_files_root_alone`, reachable only
    /// since issue #357 scoped the persisted-free-space refusal to paged files.
    /// A userblock file persists no free space whatever its creation properties
    /// asked for, so before that the mark could not be raised on one at all, and
    /// `set_consistency_flags`'s base conversion had no caller that needed it.
    #[test]
    fn a_page_buffer_marks_a_userblock_file_without_repointing_its_root() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("ub_buffered.h5");
        let mut b = FileBuilder::new();
        b.with_userblock(4096);
        b.create_dataset("t0")
            .with_i32_data(&(0..256).collect::<Vec<_>>())
            .with_shape(&[256]);
        b.write(&path).unwrap();
        let userblock_before = std::fs::read(&path).unwrap()[..4096].to_vec();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        assert_eq!(
            s.superblock.base_address,
            BaseAddress::new(4096),
            "the fixture must have a userblock, or this test says nothing about \
             the conversion it exists for"
        );
        s.set_sync_policy(SyncPolicy::OnClose);
        s.set_page_buffer_size(1 << 20)
            .expect("an unpaged userblock file must accept a page buffer");
        // The mark is the point: a session that accepted the property and raised
        // nothing would pass every assertion below, since the conversion under
        // test runs only when the flags byte is written, and `raise_crash_mark`
        // is the only thing that writes it here. Asserted in memory rather than
        // off the disk because the session holds an exclusive OS lock on the
        // file, which is mandatory on Windows.
        assert_eq!(
            s.held_status_flags,
            file_lock::WRITE_ACCESS,
            "a page-buffered session must hold the crash mark"
        );
        // A staged commit rather than an in-place append: in-place appending is
        // refused on a userblock file for reasons of its own, which is a
        // different subject and would only mask this one.
        for t in 0..3 {
            let mut db = crate::type_builders::DatasetBuilder::new(&std::format!("n{t}"));
            db.with_f64_data(&[2.5f64; 32]).with_shape(&[32]);
            s.stage_created_dataset(&ObjectPathBuf::parse(&std::format!("/n{t}")), db)
                .unwrap();
        }
        s.commit().unwrap();
        s.force_sync().unwrap();
        s.release_status_flags().unwrap();
        drop(s);

        assert_eq!(
            &std::fs::read(&path).unwrap()[..4096],
            &userblock_before[..],
            "the page-buffered session rewrote the userblock"
        );
        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(
            f.dataset("t0").unwrap().read_i32().unwrap(),
            (0..256).collect::<Vec<_>>(),
            "the commit must have left the original dataset alone"
        );
        for t in 0..3 {
            assert_eq!(
                f.dataset(&std::format!("n{t}"))
                    .unwrap()
                    .read_f64()
                    .unwrap(),
                vec![2.5f64; 32],
                "n{t}: the committed dataset must read back"
            );
        }
    }

    /// A page buffer keeps dirty pages across operations, so a workload that
    /// touches the same pages again and again pays for them once rather than
    /// once per operation (issue #288) — and still produces the same file.
    ///
    /// This is what the default gathering deliberately does *not* do, so the
    /// comparison is against that default rather than against no gathering at
    /// all: what is being measured is the second reduction, not the first.
    ///
    /// Run on a paged file **and an unpaged one**. `H5Pset_page_buffer_size`
    /// requires the paged allocator because the C page buffer is a page cache
    /// whose `min_meta_perc`/`min_raw_perc` reservations count pages that
    /// allocator keeps segregated by kind; this is a write gatherer, which needs
    /// only a window to merge within, and
    /// [`gather_page_size`](WriteEngine::gather_page_size) supplies one either
    /// way. The unpaged arm is what says so (issue #357), and it is the arm that
    /// matters most in practice: unpaged is the default strategy, so before this
    /// the property was reachable only from a file deliberately created paged.
    #[test]
    fn a_page_buffer_holds_dirty_pages_across_operations() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let run = |path: &std::path::Path, page_buffer: usize| {
            let mut s = WriteEngine::open_with_locking(path, FileLocking::Enabled).unwrap();
            s.set_sync_policy(SyncPolicy::OnClose);
            if page_buffer != 0 {
                s.set_page_buffer_size(page_buffer).unwrap();
            }
            let (appends, commit) = gather_workload(&mut s);
            // The teardown `File::close` and `FileInner::drop` perform, in their
            // order. A bare engine owes it too — it is what takes the crash mark
            // down — and nothing does it for one: this crate deliberately keeps
            // teardown writes off `WriteEngine::drop`, so that dropping a probe
            // session (as the bounded/mirrored dispatch does) writes nothing.
            // Without it the buffered file ends still marked, and the byte
            // comparison below is what says so.
            s.force_sync().unwrap();
            s.release_status_flags().unwrap();
            drop(s);
            appends + commit
        };

        for paged in [true, false] {
            let label = if paged { "paged" } else { "unpaged" };
            let per_op = dir.path().join(std::format!("{label}_per_op.h5"));
            let buffered = dir.path().join(std::format!("{label}_buffered.h5"));
            gather_fixture(&per_op, 4, paged);
            gather_fixture(&buffered, 4, paged);

            let per_op_writes = run(&per_op, 0);
            let buffered_writes = run(&buffered, 1 << 20);

            assert!(
                buffered_writes * 4 < per_op_writes,
                "{label}: a page buffer must cost meaningfully fewer writes than the \
                 per-operation default, but cost {buffered_writes} against {per_op_writes}"
            );
            assert_eq!(
                std::fs::read(&per_op).unwrap(),
                std::fs::read(&buffered).unwrap(),
                "{label}: a page buffer changed the file it produced"
            );
        }
    }

    /// A budget below the 1 MiB a session already gathers under is honored end
    /// to end, and leaves the same file 1 MiB leaves (issue #391).
    ///
    /// The floor this replaces was a write-count argument with no correctness
    /// claim behind it, so the assertion that carries the weight is the byte
    /// comparison rather than the read-back: a buffer that flushes and restarts
    /// more often must produce the *same* file, not merely a readable one.
    ///
    /// Run through the public API — create, append, commit, close, reopen —
    /// because `File::create_with_options` is where the refusal used to fire, on
    /// a pair it had already decided it could not reopen with. A 16 KiB-paged
    /// file at 256 KiB is the case the issue reported.
    #[test]
    fn a_page_buffer_below_the_gather_budget_writes_the_same_file() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let run = |path: &std::path::Path, budget: usize| -> Vec<i32> {
            let file = crate::reader::File::create_with_options(
                path,
                crate::FileCreateProperties::new()
                    .with_file_space_strategy(FileSpaceStrategy::Page, true, 1)
                    .with_file_space_page_size(16 * 1024),
                crate::FileAccessProperties::new()
                    .with_sync_policy(SyncPolicy::OnClose)
                    .with_page_buffer_size(budget),
            )
            .unwrap();
            file.root()
                .create_dataset("d", |b| {
                    b.with_i32_data(&(0..64).collect::<Vec<i32>>())
                        .with_shape(&[64])
                        .with_maxshape(&[MaxExtent::Unlimited])
                        .with_chunks(&[64]);
                })
                .unwrap();
            file.commit().unwrap();
            for round in 0..8i32 {
                let mut ds = file.dataset("d").unwrap();
                ds.append(&vec![round; 64]).unwrap();
            }
            file.root()
                .create_dataset("added", |b| {
                    b.with_f64_data(&[2.5f64; 32]).with_shape(&[32]);
                })
                .unwrap();
            file.commit().unwrap();
            file.close().unwrap();

            let reopened = crate::reader::File::open(path).unwrap();
            assert_eq!(
                reopened.dataset("added").unwrap().read_f64().unwrap(),
                vec![2.5f64; 32],
                "the dataset committed through the buffer must read back"
            );
            reopened.dataset("d").unwrap().read_i32().unwrap()
        };

        let small = dir.path().join("small_budget.h5");
        let ample = dir.path().join("ample_budget.h5");
        let from_small = run(&small, 256 * 1024);
        let from_ample = run(&ample, 1 << 20);

        let mut expected: Vec<i32> = (0..64).collect();
        for round in 0..8i32 {
            expected.extend(std::iter::repeat_n(round, 64));
        }
        assert_eq!(
            from_small, expected,
            "a 256 KiB page buffer did not append what it was given"
        );
        assert_eq!(
            from_ample, expected,
            "and neither did the 1 MiB one, so the file comparison below would be \
             two arms agreeing on the wrong answer"
        );
        assert_eq!(
            std::fs::read(&small).unwrap(),
            std::fs::read(&ample).unwrap(),
            "a 256 KiB page buffer produced a different file from a 1 MiB one"
        );
    }

    /// A barrier issues what has been gathered under **every** policy, so the
    /// bytes a publish point names are on the disk before the publish point is.
    ///
    /// Gathered writes are issued in address order, and the superblock lives at
    /// address 0 — so a commit whose barriers issued nothing would put its new
    /// root pointer on the disk *first* and the content it names last. A write
    /// that then failed, or a process that died mid-flush, would leave a
    /// superblock naming bytes that are not in the file, where before this
    /// gathering existed it left the previous file intact.
    ///
    /// `SyncPolicy::Always` cannot see this: its barriers are `fsync`s, which
    /// flush on their way out. Every crash-consistency test in this crate runs on
    /// that default, which is exactly why nothing caught it (issue #288). So this
    /// runs on `OnClose`, and asserts the order rather than a count — a count
    /// would be satisfied by a session that issued everything at the wrong time.
    ///
    /// It covers two of the three ordering sites: `barrier` (the commit tail) and
    /// `EditStore::sync` (the append phases). The third is `barrier_data`, which
    /// `EditStore::sync` now shares rather than duplicates, so it is correct by
    /// identity rather than by argument. It is covered too, one test over:
    /// mutating its `OnClose` arm to `Ok(())` fails
    /// `sync_policy_governs_the_persisting_and_flag_barriers` and nothing else.
    #[test]
    fn a_barrier_orders_the_publish_point_last_under_every_policy() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        for policy in [SyncPolicy::Always, SyncPolicy::OnClose] {
            let path = dir.path().join(std::format!("order_{policy:?}.h5"));
            gather_fixture(&path, 1, false);

            let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
            s.set_sync_policy(policy);

            let mut db = crate::type_builders::DatasetBuilder::new("added");
            db.with_f64_data(&[2.5f64; 64]).with_shape(&[64]);
            s.stage_created_dataset(&ObjectPathBuf::parse("/added"), db)
                .unwrap();
            let before = s.image.issued_write_order().len();
            s.commit().unwrap();

            let order = s.image.issued_write_order()[before..].to_vec();
            let superblock = order
                .iter()
                .position(|&(at, _)| at == s.sb_sig_off)
                .unwrap_or_else(|| panic!("{policy:?}: the commit never wrote the superblock"));
            let content = order
                .iter()
                .rposition(|&(at, _)| at != s.sb_sig_off)
                .unwrap_or_else(|| panic!("{policy:?}: the commit wrote nothing but a superblock"));
            assert!(
                superblock > content,
                "{policy:?}: the superblock was issued at position {superblock} of \
                 {order:?}, ahead of content at {content} — a failure in that window \
                 leaves a root pointing at bytes that are not in the file"
            );

            // The same for the append engine's four phases, whose publish point is
            // the dataspace dimension in the object header — a *low* address, with
            // the chunk bytes it makes visible at end-of-file. Stated as "the last
            // write is not the highest one", which is precisely what a single
            // address-ordered flush of the whole append would make it.
            let before = s.image.issued_write_order().len();
            s.append_inplace_i32_phased("t0", &[7; 64], 4).unwrap();
            let order = s.image.issued_write_order()[before..].to_vec();
            let highest = order
                .iter()
                .map(|&(at, _)| at)
                .max()
                .expect("the append wrote");
            assert!(
                order.last().expect("the append wrote").0 < highest,
                "{policy:?}: the append's last write is its highest-addressed one, so \
                 the whole append went out in address order and the dimension that \
                 publishes the new rows preceded the chunk bytes: {order:?}"
            );
        }
    }

    /// The page buffer produces the same bytes through the **bounded** backing,
    /// which is the one `File::open_rw` actually picks for a latest-format file.
    ///
    /// The test above drives the whole-file mirror, whose reads come from memory
    /// and so never meet the pending writes at all. The bounded image has no
    /// mirror: every read it serves goes to the disk and is then patched with
    /// whatever is still gathered, so it is the backing where a wrong overlay
    /// silently plans the next edit against bytes that are neither on the disk nor
    /// in the buffer. Byte identity against the same session without the buffer is
    /// the assertion a wrong overlay cannot pass.
    #[test]
    fn a_page_buffer_changes_no_byte_through_the_bounded_backing() {
        use crate::reader::File;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let run = |name: &str, page_buffer: usize| {
            let path = dir.path().join(name);
            gather_fixture(&path, 4, true);
            {
                let props = crate::FileAccessProperties::new()
                    .with_sync_policy(SyncPolicy::OnClose)
                    .with_memory_strategy(MemoryStrategy::Bounded)
                    .with_page_buffer_size(page_buffer);
                let file = File::open_rw_with_options(&path, props).unwrap();
                assert_eq!(
                    file.edit_backing(),
                    Some(crate::EditBacking::Bounded),
                    "{name}: this must exercise the mirrorless backing"
                );
                let root = file.root();
                for round in 0..4u8 {
                    for t in 0..4 {
                        let mut ds = file.dataset(&std::format!("t{t}")).unwrap();
                        ds.append(&[i32::from(round); 64]).unwrap();
                    }
                }
                for t in 0..4 {
                    root.create_dataset(&std::format!("n{t}"), |b| {
                        b.with_f64_data(&[2.5f64; 32]).with_shape(&[32]);
                    })
                    .unwrap();
                }
                file.commit().unwrap();
                root.set_attr("tag", crate::AttrValue::I32(1)).unwrap();
                file.commit().unwrap();
                file.close().unwrap();
            }
            std::fs::read(&path).unwrap()
        };

        assert_eq!(
            run("bounded_plain.h5", 0),
            run("bounded_buffered.h5", 1 << 20),
            "a page buffer changed the file the bounded backing produced"
        );
    }

    /// An operation that has returned has put its bytes in the operating system,
    /// whatever the [`SyncPolicy`] says.
    ///
    /// This is what makes the default gathering free rather than a trade: the
    /// bytes are held only *inside* a commit or an append, never across one. It
    /// is asserted as "a forced sync afterwards finds nothing left to write",
    /// because the alternative — reading the file through a second handle — is
    /// what the session's own exclusive lock exists to prevent, mandatorily so on
    /// Windows.
    #[test]
    fn a_finished_operation_has_nothing_left_to_write() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("finished_op.h5");
        gather_fixture(&path, 1, false);

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        // The policy that issues no `fsync` at all, so the ordering barriers are
        // the only thing that can be draining the buffer.
        s.set_sync_policy(SyncPolicy::OnClose);

        let before_append = s.image.issued_writes();
        s.append_inplace_i32_phased("t0", &[7; 64], 4).unwrap();
        let after_append = s.image.issued_writes();
        assert!(
            after_append > before_append,
            "the append issued nothing at all, so the equality below holds for a \
             session that did no work"
        );
        s.force_sync().unwrap();
        assert_eq!(
            s.image.issued_writes(),
            after_append,
            "a finished append left writes in this process's memory"
        );

        let mut db = crate::type_builders::DatasetBuilder::new("added");
        db.with_f64_data(&[1.5f64; 8]).with_shape(&[8]);
        s.stage_created_dataset(&ObjectPathBuf::parse("/added"), db)
            .unwrap();
        s.commit().unwrap();
        let after_commit = s.image.issued_writes();
        assert!(
            after_commit > after_append,
            "the commit issued nothing at all"
        );
        s.force_sync().unwrap();
        assert_eq!(
            s.image.issued_writes(),
            after_commit,
            "a finished commit left writes in this process's memory"
        );

        // And the bytes are the ones they were meant to be — an image that issued
        // writes at the right moments but the wrong contents passes everything
        // above.
        drop(s);
        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(f.dataset("t0").unwrap().read_i32().unwrap().len(), 320);
        assert_eq!(
            f.dataset("added").unwrap().read_f64().unwrap(),
            vec![1.5f64; 8]
        );
    }

    /// Overwriting the root group is refused by name.
    ///
    /// The refusal lives in `stage_dataset_write` rather than in the commit,
    /// because the staged dataset is flattened as it is staged and the root's
    /// empty path would otherwise be reported as a dataset with no name. No
    /// public entry point can reach it — `Dataset::write_staged` runs off a
    /// resolved dataset path — so this is where it stays covered.
    #[test]
    fn overwriting_the_root_group_is_refused_at_staging() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("root_overwrite.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("d").with_i32_data(&[1]);
        b.write(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        let mut db = crate::type_builders::DatasetBuilder::new("whatever");
        db.with_i32_data(&[1]);
        let err = s
            .stage_dataset_write(&ObjectPathBuf::parse("/"), db)
            .unwrap_err();
        assert!(
            matches!(&err, Error::EditUnsupported(m) if m.contains("root group")),
            "unexpected error: {err:?}"
        );
        assert!(!s.has_staged_edits());
    }

    /// Unallocated storage is refused by this engine rather than materialized.
    ///
    /// The whole-file writer leaves such a dataset's data address undefined
    /// (issue #293); this engine appends into an existing layout and has no
    /// equivalent, so the flag has to be answered rather than dropped. Dropping
    /// it would write out the grid of fill values the caller asked not to have,
    /// and every value assertion would pass.
    ///
    /// The message is checked, not just the failure: the arm below this one
    /// refuses a dataset with no data for the opposite reason — the caller forgot
    /// it — and reporting that one here would send a caller looking for a bug in
    /// their own code.
    #[test]
    fn staging_unallocated_storage_into_an_existing_file_is_refused() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("unallocated.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("d").with_i32_data(&[1]);
        b.write(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        let mut db = crate::type_builders::DatasetBuilder::new("sparse");
        db.with_unallocated_storage(make_i32_type(), &[1000]);
        db.with_chunks(&[100]);
        let err = s
            .stage_created_dataset(&ObjectPathBuf::parse("/sparse"), db)
            .unwrap_err();
        assert!(
            matches!(&err, Error::EditUnsupported(m) if m.contains("unallocated storage")),
            "unexpected error: {err:?}"
        );
        assert!(!s.has_staged_edits());
    }

    /// The same obligation on the one operation that does not go through
    /// [`commit`](WriteEngine::commit): the free-space finalize a session owes at
    /// teardown.
    ///
    /// Its two callers force a sync straight after it, so nothing observable
    /// breaks when it forgets — which is exactly why it needs its own test. It is
    /// `pub(crate)`, and the next caller would inherit the omission silently.
    #[test]
    fn finalize_persist_has_nothing_left_to_write() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("persisting.h5");
        let mut b = FileBuilder::new();
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 1);
        b.create_dataset("t0")
            .with_i32_data(&(0..256).collect::<Vec<_>>())
            .with_shape(&[256])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[64]);
        b.write(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        s.set_sync_policy(SyncPolicy::OnClose);
        // Grow the file past the managers, which is what gives the finalize
        // something to re-home.
        s.append_inplace_i32_phased("t0", &[7; 64], 4).unwrap();

        s.finalize_persist().unwrap();
        let after = s.image.issued_writes();
        s.force_sync().unwrap();
        assert_eq!(
            s.image.issued_writes(),
            after,
            "finalize_persist returned with writes still gathered"
        );
    }

    /// [`SpaceAccounting::reusable_free_space`] stays coalesced once an in-place
    /// append holds a reserve (issue #387).
    ///
    /// The reserve is drawn *out of* the session's free list, so what is left of
    /// the hole it came from sits immediately beside it — and as the append
    /// spends the reserve down, the spent end walks toward that remainder until
    /// the two abut exactly. Reporting them as two regions would break the
    /// field's documented contract ("no two regions touch or overlap") in the one
    /// way that misleads: a caller sizing an allocation against the largest
    /// region would be told nothing that big fits when it does.
    ///
    /// Sixteen chunk-sized appends, checked after every one, because the abutment
    /// appears only at a particular point in spending a batch rather than at the
    /// first append.
    #[test]
    fn the_append_reserve_is_reported_as_one_coalesced_free_list() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        /// Elements of the dataset the delete vacates: three reserve batches, so
        /// the hole outlives several draws.
        const VICTIM: usize = (APPEND_RESERVE_BYTES as usize * 3) / 4;
        /// Elements per chunk: 64 KiB, so sixteen appends spend a whole batch.
        const CHUNK: u64 = 16384;

        let dir = tempdir().unwrap();
        let path = dir.path().join("reserve_coalesced.h5");
        let mut b = FileBuilder::new();
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 1);
        b.create_dataset("t0")
            .with_i32_data(&[0i32])
            .with_shape(&[1])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[CHUNK]);
        b.create_dataset("victim")
            .with_i32_data(&vec![7i32; VICTIM]);
        b.create_dataset("ceiling").with_i32_data(&[9i32, 9, 9]);
        b.write(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        s.set_sync_policy(SyncPolicy::OnClose);
        s.delete(&ObjectPathBuf::parse("victim")).unwrap();
        s.commit().unwrap();

        let batch: Vec<i32> = (0..CHUNK as i32).collect();
        let mut reserved_ever = false;
        for round in 0..16 {
            s.append_inplace_i32_phased("t0", &batch, 4).unwrap();
            reserved_ever |= !s.reserved.is_empty();
            let acct = s.space_accounting();
            let regions = &acct.reusable_free_space;
            for w in regions.windows(2) {
                let ((a_addr, a_len), (b_addr, _)) = (w[0], w[1]);
                assert!(
                    a_addr + a_len < b_addr,
                    "round {round}: [{a_addr}, {}) and [{b_addr}, ..) touch or overlap, \
                     though reusable_free_space is documented as fully coalesced: {regions:?}",
                    a_addr + a_len
                );
            }
            assert_eq!(
                acct.reusable_free_bytes,
                regions.iter().map(|&(_, len)| len).sum::<u64>(),
                "round {round}: the total must be the summed length of the regions"
            );
        }
        assert!(
            reserved_ever,
            "no reserve was ever held, so this measured the plain free list"
        );
    }

    /// A persisting session gives its **unspent** append reserve back to the
    /// on-disk managers before it closes (issue #387).
    ///
    /// An append on such a file takes a batch out of the managers before it may
    /// spend any of it, and takes a whole batch however little the append needs.
    /// What is left over is space nothing occupies, so leaving it out of the
    /// managers would turn every appending session into a leak of up to
    /// [`APPEND_RESERVE_BYTES`] — invisible from inside the session, since its own
    /// accounting still counts the reserve as reusable.
    ///
    /// Measured from outside the session for that reason: the persisted free
    /// space after the close, against what it was before the append.
    #[test]
    fn an_unspent_append_reserve_goes_back_to_the_managers() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        /// Elements of the dataset the delete below vacates. Its blocks are the
        /// hole the append draws from, so it has to exceed one reserve batch or
        /// nothing is drawn and this measures the old behaviour.
        const VICTIM: usize = (APPEND_RESERVE_BYTES as usize * 2) / 4;

        let dir = tempdir().unwrap();
        let path = dir.path().join("reserve_return.h5");
        let mut b = FileBuilder::new();
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 1);
        b.create_dataset("t0")
            .with_i32_data(&(0..256).collect::<Vec<_>>())
            .with_shape(&[256])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[64]);
        b.create_dataset("victim")
            .with_i32_data(&vec![7i32; VICTIM]);
        // Above the hole, so the delete leaves an interior region rather than a
        // run reaching end-of-file that the commit truncates away.
        b.create_dataset("ceiling").with_i32_data(&[9i32, 9, 9]);
        b.write(&path).unwrap();

        let persisted = |p: &std::path::Path| -> u64 {
            crate::reader::File::open(p)
                .unwrap()
                .persisted_free_space()
                .iter()
                .map(|&(_, len)| len)
                .sum()
        };

        {
            let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
            s.set_sync_policy(SyncPolicy::OnClose);
            s.delete(&ObjectPathBuf::parse("victim")).unwrap();
            s.commit().unwrap();
        }
        let before = persisted(&path);
        assert!(
            before > APPEND_RESERVE_BYTES,
            "the fixture must leave more than one reserve batch on disk, not {before} bytes"
        );

        // A few kilobytes of append against a megabyte of reserve.
        let appended: Vec<i32> = (0..1024).collect();
        {
            let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
            s.set_sync_policy(SyncPolicy::OnClose);
            s.append_inplace_i32_phased("t0", &appended, 4).unwrap();
            s.finalize_persist().unwrap();
        }

        let after = persisted(&path);
        // What the append actually placed: its chunks and the index blocks beside
        // them, plus the manager rewrite's own churn. Generous, and still an order
        // of magnitude under the batch that would go missing.
        let spent = 64 * 1024;
        assert!(
            after + spent >= before,
            "the session reserved {APPEND_RESERVE_BYTES} bytes and spent a few of them, \
             so the managers should still describe nearly all of the {before} they did \
             before — they describe {after}"
        );
        assert!(
            after < before,
            "the append placed bytes inside the hole, so the managers must describe \
             less than the {before} they did (they describe {after})"
        );

        let f = crate::reader::File::open(&path).unwrap();
        let mut want = (0..256).collect::<Vec<i32>>();
        want.extend_from_slice(&appended);
        assert_eq!(f.dataset("t0").unwrap().read_i32().unwrap(), want);
        assert_eq!(f.dataset("ceiling").unwrap().read_i32().unwrap(), [9, 9, 9]);
    }

    /// A persisting session's fixture for the draw tests: `t0` to append onto,
    /// `victims` datasets of `victim_elems` `i32`s each, every one followed by a
    /// small keeper so that deleting the victims leaves that many *separate*
    /// holes, and a `ceiling` above them all so none of the holes is trailing.
    fn fragmented_persisting_fixture(
        path: &std::path::Path,
        victims: usize,
        victim_elems: usize,
    ) -> WriteEngine {
        use crate::writer::FileBuilder;

        let mut b = FileBuilder::new();
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 1);
        b.create_dataset("t0")
            .with_i32_data(&[0i32])
            .with_shape(&[1])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[64]);
        for v in 0..victims {
            b.create_dataset(&std::format!("victim{v}"))
                .with_i32_data(&vec![7i32; victim_elems]);
            b.create_dataset(&std::format!("keeper{v}"))
                .with_i32_data(&[v as i32]);
        }
        b.create_dataset("ceiling").with_i32_data(&[9i32, 9, 9]);
        b.write(path).unwrap();

        let mut s = WriteEngine::open_with_locking(path, FileLocking::Enabled).unwrap();
        s.set_sync_policy(SyncPolicy::OnClose);
        for v in 0..victims {
            s.delete(&ObjectPathBuf::parse(&std::format!("victim{v}")))
                .unwrap();
        }
        s.commit().unwrap();
        s
    }

    /// One draw takes every hole the append fits in, not only the largest, so a
    /// file fragmented into small holes pays one manager rewrite for all of them
    /// rather than one per hole (issue #413).
    ///
    /// Three holes of a few chunks each, each far under a batch, separated by
    /// live objects so they cannot coalesce. One chunk-sized append then has to
    /// leave the reserve holding runs from more than one of them: a draw that
    /// took the largest hole alone would have served the chunk just as well and
    /// held exactly one run, so the count is what tells the two apart.
    #[test]
    fn a_draw_gathers_every_hole_the_append_fits_in() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("draw_gathers.h5");
        let mut s = fragmented_persisting_fixture(&path, 3, 4 * 64);
        assert!(
            s.free.sections().len() >= 3,
            "the fixture must leave three separate holes, not {:?}",
            s.free.sections()
        );
        let before = s.free.sections();

        s.append_inplace_i32_phased("t0", &[1i32; 64], 4).unwrap();

        let held = s.reserved.sections();
        // A run per hole, less what the chunk and the rewrite's tail took out of
        // them: the chunk out of one, the tail out of at most one more.
        assert!(
            held.len() >= 2,
            "one draw should have gathered runs from several of the holes {before:?}, \
             not only {held:?}"
        );
        let drawn: u64 = held.iter().map(|&(_, len)| len).sum();
        assert!(
            drawn > before.iter().map(|&(_, len)| len).max().unwrap(),
            "the reserve holds {drawn} bytes, no more than the largest hole alone"
        );
    }

    /// The manager rewrite that publishes a draw places its own tail inside the
    /// space the draw is taking when the draw has emptied the free list, rather
    /// than appending it (issue #413).
    ///
    /// On a flat file the draw and the tail are served from the same list, and
    /// a draw takes every hole the append fits in, so the case is the ordinary
    /// one, not a corner: without the fallback every draw appended one tail at
    /// end-of-file and freed the previous one, a persisting file's whole growth
    /// under churn once the floor on the draw was gone. Measured as the file's
    /// length across the append, which a tail landing anywhere inside leaves
    /// alone.
    ///
    /// Measured as the tail's own address rather than as an unchanged length: the
    /// rewrite supersedes the fixture's tail, which sat at end-of-file, so
    /// releasing that trailing run leaves the file *shorter* than it started
    /// (issue #418). An appended tail would land at exactly the pre-append
    /// end-of-file, which is the case both assertions rule out.
    #[test]
    fn a_draws_rewrite_places_its_tail_inside_the_draw() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("draw_tail.h5");
        let mut s = fragmented_persisting_fixture(&path, 1, 8 * 64);
        let len_before = s.image.len();

        s.append_inplace_i32_phased("t0", &[1i32; 64], 4).unwrap();

        assert!(
            !s.reserved.is_empty(),
            "no reserve was drawn, so this measured an ordinary end-of-file append"
        );
        let ext = s.superblock.superblock_extension_address.unwrap();
        assert!(
            ext < len_before,
            "the rewrite appended its tail at {ext}, the pre-append end-of-file \
             {len_before}, instead of placing it in the hole it was publishing as taken"
        );
        assert!(
            s.image.len() <= len_before,
            "a rewrite that placed its tail inside the file must not have grown it \
             (was {len_before}, now {})",
            s.image.len()
        );
    }

    /// One in-place append costs a small constant number of writes even where
    /// nothing gathers them — the case the Extensible-Array header's six
    /// statistics were written separately for.
    ///
    /// Under gathering, six adjacent writes and one merge into the same page
    /// write, so the whole suite is blind to which one the engine made. The SWMR
    /// writer is the regime where it still shows: it gathers nothing by design,
    /// so every write the engine makes is a syscall. Stated as a ceiling on the
    /// count rather than as an exact figure, since what an append needs is free to
    /// fall — and it has: for the append this measures, the first into a dataset,
    /// 12 before issue #307 published each checksummed structure in one write and
    /// 8 after; an append that reuses the data block that one allocates costs 5.
    /// Six statistics written singly puts it five over; a checksum written apart
    /// from the value it covers, four.
    #[test]
    fn an_unbuffered_append_costs_a_small_constant_number_of_writes() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("swmr_cost.h5");
        gather_fixture(&path, 1, false);

        let mut s = WriteEngine::open_swmr_writer(&path, SyncPolicy::OnClose).unwrap();
        let before = s.image.issued_writes();
        s.append_inplace_i32_phased("t0", &[7; 64], 4).unwrap();
        let cost = s.image.issued_writes() - before;

        assert!(
            cost > 0,
            "the append issued nothing, so the ceiling below proves nothing"
        );
        assert!(
            cost <= 10,
            "an unbuffered append costs {cost} writes; the array header's six \
             statistics belong in one write, not six, and each checksum belongs in \
             the write that changed what it covers (measured at 8)"
        );
    }

    /// A SWMR writer gathers nothing: its readers follow the ordered phases as
    /// they become visible, and a phase that has not reached the operating system
    /// is a phase the reader cannot see.
    ///
    /// Stopping inside the durability sequence is what makes this selective. A
    /// completed append ends with a barrier, so it would land on disk under
    /// either setting; only a stop *inside* the sequence distinguishes a writer
    /// that gathers from one that does not. The file is read through a second handle, which SWMR
    /// permits precisely because it takes no lock.
    #[test]
    fn the_swmr_writer_holds_no_write_back() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("swmr.h5");
        gather_fixture(&path, 1, false);

        let mut s = WriteEngine::open_swmr_writer(&path, SyncPolicy::OnClose).unwrap();
        let before = std::fs::read(&path).unwrap().len();
        // Phase 1 only: the chunk bytes and the superblock's end-of-file.
        s.append_inplace_i32_phased("t0", &[7; 64], 1).unwrap();

        assert!(
            std::fs::read(&path).unwrap().len() > before,
            "a SWMR reader must see the phase-1 chunk bytes as soon as they are written"
        );
    }

    /// The `fsync` cadence belongs to whoever the fapl says: every durability
    /// point in the engine — an immediate append's ordered barriers, a commit's
    /// barrier and repoint, the barrier a commit issues after truncating, the
    /// same-length-overwrite fast path, and the barrier `close` issues — answers
    /// to the session's [`SyncPolicy`], while `force_sync` (the explicit
    /// `File::sync`) answers to nobody (issue #263).
    ///
    /// The counts are compared as a table across the two policies rather than
    /// pinned to literals: how many `fsync`s a commit costs is an implementation
    /// detail that may fall, but that `OnClose` costs *none* and `Always` costs
    /// some at each of those points is the contract. The file is read back
    /// under both, since a skipped barrier must cost durability and nothing else
    /// — the bytes have already reached the operating system.
    ///
    /// Every stage here is a *distinct* barrier site. A commit tail that syncs
    /// twice does not stand in for the fast path that syncs once, nor for the
    /// post-truncate barrier that only a shrinking commit reaches: each is its
    /// own `self.barrier()` call that a later edit could regress to a bare
    /// `self.image.sync_all()` on its own — verified by mutating each site
    /// separately and watching this fail. The persisting tails and the
    /// consistency-flag write are covered by the test below.
    ///
    /// One site is left uncovered: the version 0/1 repoint branch of the
    /// non-persisting tail. A pre-v2 file cannot be produced by this crate's own
    /// writer — the crosscheck tests get one from the C library — and the
    /// counting image is a bounded one, which such a file needs the mirror
    /// instead of. Covering it needs a checked-in fixture and a second counting
    /// opener; it is named here so the gap is a known one rather than an
    /// assumed-covered one.
    #[test]
    fn sync_policy_governs_every_barrier() {
        use crate::writer::FileBuilder;
        use std::sync::Arc;
        use std::sync::atomic::{AtomicU64, Ordering};
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        // [after an immediate append, after a staged commit, after a same-length
        // overwrite, after a commit that truncates, after the close barrier,
        // after an explicit sync].
        let run = |name: &str, policy: SyncPolicy| -> [u64; 5] {
            let path = dir.path().join(name);
            let mut b = FileBuilder::new();
            b.create_dataset("d")
                .with_i32_data(&(0..8).collect::<Vec<_>>())
                .with_shape(&[8])
                .with_maxshape(&[MaxExtent::Unlimited])
                .with_chunks(&[4]);
            b.write(&path).unwrap();

            let syncs = Arc::new(AtomicU64::new(0));
            let mut s = WriteEngine::open_sync_counting(&path, policy, Arc::clone(&syncs)).unwrap();
            s.append_inplace_i32_phased("d", &[8, 9, 10, 11], 4)
                .unwrap();
            let after_append = syncs.load(Ordering::Relaxed);
            let mut db = crate::type_builders::DatasetBuilder::new("added");
            db.with_f64_data(&[2.5f64; 8]).with_shape(&[8]);
            s.stage_created_dataset(&ObjectPathBuf::parse("/added"), db)
                .unwrap();
            s.commit().unwrap();
            let after_commit = syncs.load(Ordering::Relaxed);

            // Same-length value overwrite: the commit fast path, which patches
            // the bytes where they lie and syncs without repointing anything.
            let mut ow = crate::type_builders::DatasetBuilder::new("added");
            ow.with_f64_data(&[4.5f64; 8]).with_shape(&[8]);
            s.stage_dataset_write(&ObjectPathBuf::parse("/added"), ow)
                .unwrap();
            s.commit().unwrap();
            let after_overwrite = syncs.load(Ordering::Relaxed);

            // A delete whose freed run reaches end-of-file, so the commit
            // truncates and takes the barrier that only a shrinking commit does.
            s.delete(&ObjectPathBuf::parse("/added")).unwrap();
            s.commit().unwrap();
            let after_truncate = syncs.load(Ordering::Relaxed);

            s.force_sync().unwrap();
            let after_forced = syncs.load(Ordering::Relaxed);

            // Release the exclusive OS lock before reading the file back; those
            // locks are mandatory on Windows.
            drop(s);
            let f = crate::reader::File::open(&path).unwrap();
            assert_eq!(
                f.dataset("d").unwrap().read_i32().unwrap(),
                (0..12).collect::<Vec<_>>(),
                "the append must land under {policy:?}"
            );
            assert!(
                f.dataset("added").is_err(),
                "the deleting commit must land under {policy:?}"
            );
            [
                after_append,
                after_commit,
                after_overwrite,
                after_truncate,
                after_forced,
            ]
        };

        let always = run("always.h5", SyncPolicy::Always);
        assert!(always[0] > 0, "an immediate append syncs under Always");
        assert!(always[1] > always[0], "so does a commit");
        assert!(
            always[2] > always[1],
            "so does the same-length-overwrite fast path"
        );
        assert!(
            always[3] > always[2],
            "so does a commit that truncates the file"
        );
        assert_eq!(
            always[4],
            always[3] + 1,
            "and a forced sync is exactly one more"
        );

        let deferred = run("on_close.h5", SyncPolicy::OnClose);
        assert_eq!(
            &deferred[..4],
            &[0, 0, 0, 0],
            "OnClose must leave every one of those in-session barriers unissued"
        );
        assert_eq!(
            deferred[4], 1,
            "a forced sync is issued whatever the policy says — it is what the \
             teardown path and File::sync both take"
        );
    }

    /// The two barrier sites the table above cannot reach: the commit tail of a
    /// file that *persists* its free space (a different tail, with its own
    /// barrier and repoint, plus the manager re-homing `close` owes), and the
    /// superblock consistency-flag write a SWMR session makes on open and close.
    ///
    /// Both are `self.barrier()`/`self.barrier_data()` calls on paths the
    /// ordinary non-persisting session never executes, so without this a
    /// regression at either would pass the whole suite (issue #263).
    #[test]
    fn sync_policy_governs_the_persisting_and_flag_barriers() {
        use crate::writer::FileBuilder;
        use std::sync::Arc;
        use std::sync::atomic::{AtomicU64, Ordering};
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        // [after a commit on a persisting file, after the flag write, after the
        // close barrier and its manager re-homing].
        let run = |name: &str, strategy: FileSpaceStrategy, policy: SyncPolicy| -> [u64; 3] {
            let path = dir.path().join(name);
            let mut b = FileBuilder::new();
            b.create_dataset("d")
                .with_i32_data(&(0..8).collect::<Vec<_>>())
                .with_shape(&[8])
                .with_maxshape(&[MaxExtent::Unlimited])
                .with_chunks(&[4]);
            b.create_dataset("victim")
                .with_f64_data(&[1.5f64; 64])
                .with_shape(&[64]);
            // A paged file takes the page-aware tail, a different pair of barrier
            // calls from the plain persisting one; `0` asks for the default page
            // size, which the plain strategy ignores.
            b.with_file_space_strategy(
                strategy,
                true,
                if strategy == FileSpaceStrategy::Page {
                    0
                } else {
                    1
                },
            );
            b.write(&path).unwrap();

            let syncs = Arc::new(AtomicU64::new(0));
            let mut s = WriteEngine::open_sync_counting(&path, policy, Arc::clone(&syncs)).unwrap();
            assert!(
                s.persist.is_some(),
                "the fixture must persist its free space, or this tests the wrong tail"
            );
            // A delete on a persisting file takes `commit_persisting`: the free
            // space is recorded on disk rather than truncated away.
            s.delete(&ObjectPathBuf::parse("/victim")).unwrap();
            s.commit().unwrap();
            let after_commit = syncs.load(Ordering::Relaxed);

            s.set_consistency_flags(0).unwrap();
            let after_flags = syncs.load(Ordering::Relaxed);
            // The flag write's barrier orders as well as syncs, like the other
            // two. Its production callers all force a sync straight after, which
            // is why nothing else would notice it stopping — the same reason
            // `finalize_persist_has_nothing_left_to_write` exists.
            let issued = s.image.issued_writes();
            s.force_sync().unwrap();
            assert_eq!(
                s.image.issued_writes(),
                issued,
                "{policy:?}: the consistency-flag write was left gathered"
            );

            // An immediate append leaves the on-disk managers mid-file, which is
            // the debt `finalize_persist` settles with a second commit tail at
            // close — the writes no earlier `sync` can have covered.
            s.append_inplace_i32_phased("d", &[8, 9, 10, 11], 4)
                .unwrap();
            let before_close = syncs.load(Ordering::Relaxed);
            s.finalize_persist().unwrap();
            let after_close = syncs.load(Ordering::Relaxed) - before_close;

            drop(s);
            let f = crate::reader::File::open(&path).unwrap();
            assert!(
                f.dataset("victim").is_err(),
                "the delete must land under {policy:?}"
            );
            assert_eq!(
                f.dataset("d").unwrap().read_i32().unwrap(),
                (0..12).collect::<Vec<_>>(),
                "and so must the append under {policy:?}"
            );
            [after_commit, after_flags, after_close]
        };

        // Both persisting tails: the plain one and the page-aware one, which is a
        // separate pair of barrier calls reached only by a paged file.
        for (label, strategy) in [
            ("fsmaggr", FileSpaceStrategy::FsmAggr),
            ("paged", FileSpaceStrategy::Page),
        ] {
            let always = run(&format!("{label}_always.h5"), strategy, SyncPolicy::Always);
            assert!(
                always[0] > 0,
                "the {label} persisting commit tail syncs under Always"
            );
            assert!(
                always[1] > always[0],
                "so does the consistency-flag write a SWMR session makes ({label})"
            );
            assert!(
                always[2] > 0,
                "so does the manager re-homing close owes ({label})"
            );

            assert_eq!(
                run(
                    &format!("{label}_on_close.h5"),
                    strategy,
                    SyncPolicy::OnClose
                ),
                [0, 0, 0],
                "OnClose must leave the {label} persisting tail, the flag write, and the \
                 close-time manager re-homing with no fsync at all"
            );
        }
    }

    /// A write that alters the file and *then* reports failure is a real device
    /// behaviour — it is the one `TornWriteImage` models — so a `write_at`
    /// returning an error does not mean its bytes missed the disk. When the
    /// write in question is the superblock repoint, that makes "did this commit
    /// publish?" unanswerable from inside the engine, and every rollback
    /// unsound: undoing the values would tear a commit that did land.
    ///
    /// So the flag that gates the rollback is raised *before* the publish is
    /// issued, not after it returns, and this pins that. The commit here does
    /// publish — `/extra` is readable afterwards, which is only possible
    /// through the new root — while its `commit` returns the write's error.
    /// Rolling back under it would leave `/nums` holding the pre-commit value
    /// inside a tree that is otherwise entirely the new one (issue #344).
    #[test]
    fn a_commit_whose_publish_write_fails_rolls_nothing_back() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("torn_publish.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("nums").with_i32_data(&[1, 2, 3]);
        b.write(&path).unwrap();

        // The superblock sits at offset 0 on a file with no userblock.
        let mut s = WriteEngine::open_torn_writes(&path, 0..48).unwrap();
        s.stage_dataset_write(&ObjectPathBuf::parse("/nums"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&[9, 9, 9]);
            db
        })
        .unwrap();
        s.stage_created_dataset(&ObjectPathBuf::parse("/extra"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&[42]);
            db
        })
        .unwrap();
        let failed = s.commit();
        assert!(
            matches!(failed, Err(Error::Io(_))),
            "the publish write is what failed: {failed:?}"
        );
        drop(s);

        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(
            f.dataset("extra").unwrap().read_i32().unwrap(),
            vec![42],
            "the publish landed despite reporting failure, so this commit is live"
        );
        assert_eq!(
            f.dataset("nums").unwrap().read_i32().unwrap(),
            vec![9, 9, 9],
            "a rollback under a commit that did publish would tear it"
        );
    }

    /// Two hard links to one dataset are two paths over one data block, and the
    /// commit's duplicate guard is by path — so one commit can journal the same
    /// address twice, the second entry holding what the first write put there.
    /// The undo therefore has to replay newest-first; oldest-first finishes on a
    /// value from the very batch being rolled back (issue #344).
    ///
    /// A same-length overwrite through either link is deliberately allowed —
    /// it rewrites the block every link sees, so unlike a relocating write it
    /// needs no single-hard-link rule (`write_dataset_shared_hard_link_crosscheck`
    /// is the interop half of that). Which is what makes this reachable.
    ///
    /// The second link is made with the reference C library because this crate
    /// has no API that creates one: the file is a committed one, see
    /// `crates/crosscheck/tests/main/c_test_data.rs`.
    #[test]
    fn an_undo_replays_two_hard_links_to_one_block_newest_first() {
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = crate::test_data::copy("c/hard_link_undo.h5", dir.path());

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        // Fail the commit in its tail, after both writes have landed.
        s.superblock.superblock_extension_address = Some(0);
        for (name, data) in [("/aa", [9, 9, 9]), ("/bb", [8, 8, 8])] {
            s.stage_dataset_write(&ObjectPathBuf::parse(name), {
                let mut db = crate::type_builders::DatasetBuilder::new("");
                db.with_i32_data(&data);
                db
            })
            .unwrap();
        }
        // A second kind of edit, so the batch takes the full commit path and has
        // a tail to fail in.
        s.stage_created_dataset(&ObjectPathBuf::parse("/extra"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&[42]);
            db
        })
        .unwrap();
        assert!(s.commit().is_err());
        drop(s);

        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(
            f.dataset("aa").unwrap().read_i32().unwrap(),
            vec![1, 2, 3],
            "the block both links share must hold the value it held before"
        );
        assert_eq!(f.dataset("bb").unwrap().read_i32().unwrap(), vec![1, 2, 3]);
    }

    /// The chunked in-place path (`WritePlan::InPlaceChunks`) journals one entry
    /// per chunk, and — when the chunks shrink — the rewritten chunk index
    /// beside them. A partial restore would leave the index recording the new
    /// sizes against the old bytes, so this is the case where the rollback has
    /// most to put back (issue #344).
    ///
    /// Filtered, and overwritten with data that compresses to almost nothing, so
    /// every chunk's recorded size changes and the index write is part of what
    /// is rolled back rather than an unchanged block.
    #[test]
    fn a_failed_commit_puts_back_every_chunk_it_overwrote() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("chunked_undo.h5");
        let original: Vec<i32> = (0..4096i32)
            .map(|i| i.wrapping_mul(2_654_435_761u32 as i32) ^ i)
            .collect();
        let mut b = FileBuilder::new();
        b.create_dataset("grid")
            .with_i32_data(&original)
            .with_shape(&[4096])
            .with_chunks(&[512])
            .with_deflate(6);
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 0);
        b.write(&path).unwrap();

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        s.superblock.superblock_extension_address = Some(0);
        s.stage_dataset_write(&ObjectPathBuf::parse("/grid"), {
            // Values only: an overwrite that restated the chunking or the
            // filter would be refused as asking for more than an overwrite
            // (issue #318). The layout comes from the dataset already there.
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&vec![0i32; 4096]);
            db
        })
        .unwrap();
        s.stage_created_dataset(&ObjectPathBuf::parse("/extra"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&[42]);
            db
        })
        .unwrap();
        assert!(s.commit().is_err());
        drop(s);

        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(
            f.dataset("grid").unwrap().read_i32().unwrap(),
            original,
            "every chunk, and the index sizing them, must read as it did before"
        );
    }

    /// A commit clears the journal on entry, so it can never replay entries it
    /// did not write itself (issue #344).
    ///
    /// Only an attempt that *unwound* leaves any — a panic caught by the session
    /// lock, which recovers from poisoning — and the state it leaves is what
    /// this constructs directly rather than by finding something to panic in:
    /// an entry naming an address whose contents have moved on. Replaying it
    /// would write bytes captured before the unwind over whatever lives there
    /// now, and the entry clear on the way *out* cannot help, because that is
    /// the step the unwind skipped.
    ///
    /// `superseded_heaps` two fields up clears at entry for the same reason and
    /// says so; this is that rule applied to the journal.
    #[test]
    fn a_commit_does_not_replay_a_journal_it_did_not_write() {
        use crate::writer::FileBuilder;
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let path = dir.path().join("stale_journal.h5");
        let mut b = FileBuilder::new();
        b.create_dataset("nums").with_i32_data(&[1, 2, 3]);
        b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 0);
        b.write(&path).unwrap();

        let nums_block = {
            let f = crate::reader::File::open(&path).unwrap();
            match f.dataset("nums").unwrap().layout().unwrap() {
                crate::Layout::Contiguous {
                    address: Some(a), ..
                } => a,
                other => panic!("expected a contiguous dataset: {other:?}"),
            }
        };

        let mut s = WriteEngine::open_with_locking(&path, FileLocking::Enabled).unwrap();
        // Exactly what an attempt that unwound past `commit` leaves behind.
        let stale: Vec<u8> = [7i32, 7, 7].iter().flat_map(|v| v.to_le_bytes()).collect();
        s.inplace_undo.push((nums_block, stale));

        // An unrelated commit, failed in its tail so that it rolls back at all.
        s.superblock.superblock_extension_address = Some(0);
        s.stage_created_dataset(&ObjectPathBuf::parse("/extra"), {
            let mut db = crate::type_builders::DatasetBuilder::new("");
            db.with_i32_data(&[42]);
            db
        })
        .unwrap();
        assert!(s.commit().is_err());
        drop(s);

        let f = crate::reader::File::open(&path).unwrap();
        assert_eq!(
            f.dataset("nums").unwrap().read_i32().unwrap(),
            vec![1, 2, 3],
            "the commit replayed a journal entry that was not its own"
        );
    }

    /// Putting the values back is a write like any other, so it has to be
    /// ordered like one: a restore left sitting in the gathering buffer can be
    /// issued after a later write to the same block, which is the reordering
    /// hazard write gathering introduced (issue #288).
    ///
    /// Measured as a difference rather than a total, so it states the rule and
    /// not one commit's arithmetic: a refused commit with nothing to put back
    /// issues no barrier at all — it fails before the one it would have made —
    /// and the same refusal carrying a value overwrite issues exactly the one
    /// that orders the restore.
    #[test]
    fn a_rollback_orders_the_values_it_puts_back() {
        use crate::writer::FileBuilder;
        use std::sync::Arc;
        use std::sync::atomic::{AtomicU64, Ordering};
        use tempfile::tempdir;

        let mut counted = Vec::new();
        for with_overwrite in [false, true] {
            let dir = tempdir().unwrap();
            let path = dir.path().join("rollback_syncs.h5");
            let mut b = FileBuilder::new();
            b.create_dataset("nums").with_i32_data(&[1, 2, 3]);
            b.with_file_space_strategy(FileSpaceStrategy::FsmAggr, true, 0);
            b.write(&path).unwrap();

            let syncs = Arc::new(AtomicU64::new(0));
            let mut s =
                WriteEngine::open_sync_counting(&path, SyncPolicy::Always, Arc::clone(&syncs))
                    .unwrap();
            // Fail the commit in its tail, after the apply phase has written.
            s.superblock.superblock_extension_address = Some(0);
            if with_overwrite {
                s.stage_dataset_write(&ObjectPathBuf::parse("/nums"), {
                    let mut db = crate::type_builders::DatasetBuilder::new("");
                    db.with_i32_data(&[9, 9, 9]);
                    db
                })
                .unwrap();
            }
            s.stage_created_dataset(&ObjectPathBuf::parse("/extra"), {
                let mut db = crate::type_builders::DatasetBuilder::new("");
                db.with_i32_data(&[42]);
                db
            })
            .unwrap();
            let before = syncs.load(Ordering::Relaxed);
            assert!(s.commit().is_err());
            counted.push(syncs.load(Ordering::Relaxed) - before);
        }

        assert_eq!(
            counted[0], 0,
            "a refusal with nothing to put back reaches no barrier"
        );
        assert_eq!(
            counted[1],
            counted[0] + 1,
            "a refusal that put values back must order them: {counted:?}"
        );
    }

    /// The journal reads back what it is about to overwrite, so a bounded
    /// commit's reads scale with the value being replaced rather than staying
    /// near the metadata-only figure
    /// [`a_bounded_commit_reads_far_less_than_the_file`] pins for a commit that
    /// overwrites nothing.
    ///
    /// This is the cost of the guarantee in issue #344 and it is charged even
    /// when the commit succeeds, since whether the bytes will be needed is not
    /// known until the attempt ends. Stated as a rule — reads land between the
    /// payload and a small multiple of it — rather than as one target's exact
    /// count, so it holds wherever the suite runs.
    #[test]
    fn a_bounded_commit_reads_the_value_it_is_replacing() {
        use crate::writer::FileBuilder;
        use std::sync::Arc;
        use std::sync::atomic::{AtomicU64, Ordering};
        use tempfile::tempdir;

        let dir = tempdir().unwrap();
        let p = dir.path().join("journal_reads.h5");
        let data: Vec<i32> = (0..250_000).collect();
        let payload = (data.len() * 4) as u64;
        let mut b = FileBuilder::new();
        b.create_dataset("nums").with_i32_data(&data);
        b.write(&p).unwrap();

        let read_bytes = Arc::new(AtomicU64::new(0));
        let mut engine = WriteEngine::open_bounded_counting(&p, Arc::clone(&read_bytes)).unwrap();
        engine
            .stage_dataset_write(&ObjectPathBuf::parse("/nums"), {
                let mut db = crate::type_builders::DatasetBuilder::new("");
                db.with_i32_data(&data);
                db
            })
            .unwrap();
        let before = read_bytes.load(Ordering::Relaxed);
        engine.commit().unwrap();
        let read = read_bytes.load(Ordering::Relaxed) - before;

        assert!(
            read >= payload,
            "the journal must read the whole value it replaces: {read} of {payload}"
        );
        assert!(
            read < payload + (64 << 10),
            "and not much more than it: {read} against {payload}"
        );
    }
}

#[cfg(test)]
mod staged_query_tests {
    //! The queries a handle asks about objects a session has staged and not
    //! committed (issue #392).

    use rstest::rstest;
    use tempfile::tempdir;

    use super::*;
    use crate::object_path::ObjectPath;
    use crate::type_builders::DatasetBuilder;

    /// A session over a file holding one dataset, `existing`.
    fn open_session(path: &Path) -> WriteEngine {
        let mut b = crate::writer::FileBuilder::new();
        b.create_dataset("existing").with_i32_data(&[1, 2, 3]);
        b.write(path).unwrap();
        WriteEngine::open_rw_with_strategy(
            path,
            crate::source::MetadataCacheConfig::disabled(),
            FileLocking::Enabled,
            MemoryStrategy::Mirrored,
        )
        .unwrap()
    }

    fn i32_dataset(data: &[i32]) -> DatasetBuilder {
        let mut b = DatasetBuilder::new("");
        b.with_i32_data(data);
        b
    }

    fn kind(e: &WriteEngine, path: &str) -> Option<StagedKind> {
        e.staged_object(&ObjectPathBuf::parse(path)).map(|o| o.kind)
    }

    fn child_names(e: &WriteEngine, parent: &str) -> Vec<(String, StagedKind)> {
        e.staged_children(&ObjectPathBuf::parse(parent))
            .into_iter()
            .map(|c| (c.name, c.kind))
            .collect()
    }

    #[test]
    fn a_staged_creation_is_named_at_every_level_of_its_path() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.create_group(&ObjectPathBuf::parse("a")).unwrap();
        let mut col = DatasetBuilder::new("");
        col.with_i32_data(&[1, 2])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[2]);
        e.stage_created_dataset(&ObjectPathBuf::parse("a/b/col"), col)
            .unwrap();

        assert_eq!(kind(&e, "a"), Some(StagedKind::Group));
        assert_eq!(kind(&e, "a/b/col"), Some(StagedKind::Dataset));
        // Nothing stages `a/b`, and a commit builds only what the session
        // staged, so a batch reaching through `a/b` needs it staged at its own
        // path.
        assert_eq!(kind(&e, "a/b"), None);
        // The root always exists, an on-disk object is not staged, and neither
        // is a name nothing reaches.
        assert_eq!(kind(&e, ""), None);
        assert_eq!(kind(&e, "existing"), None);
        assert_eq!(kind(&e, "a/missing"), None);

        assert_eq!(
            child_names(&e, ""),
            vec![("a".to_string(), StagedKind::Group)]
        );
        assert_eq!(
            child_names(&e, "a/b"),
            vec![("col".to_string(), StagedKind::Dataset)]
        );
        // `b` under `a` is the unnamed intermediate again.
        assert!(child_names(&e, "a").is_empty());
        assert!(child_names(&e, "existing").is_empty());

        // Named outright, it answers at every level.
        e.create_group(&ObjectPathBuf::parse("a/b")).unwrap();
        assert_eq!(kind(&e, "a/b"), Some(StagedKind::Group));
        assert_eq!(
            child_names(&e, "a"),
            vec![("b".to_string(), StagedKind::Group)]
        );
    }

    #[test]
    fn a_staged_dataset_reports_what_its_builder_settled() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        let mut col = DatasetBuilder::new("");
        col.with_i32_data(&[1, 2, 3, 4])
            .with_maxshape(&[MaxExtent::Unlimited])
            .with_chunks(&[2])
            .with_deflate(4);
        e.stage_created_dataset(&ObjectPathBuf::parse("col"), col)
            .unwrap();
        e.stage_created_dataset(&ObjectPathBuf::parse("plain"), i32_dataset(&[5]))
            .unwrap();

        let meta = e.staged_dataset_meta(&ObjectPathBuf::parse("col")).unwrap();
        assert_eq!(meta.dimensions, vec![4]);
        assert_eq!(meta.maxshape, Some(vec![MaxExtent::Unlimited]));
        assert_eq!(meta.datatype.type_size(), 4);
        assert!(meta.chunked);
        assert_eq!(meta.filters, vec![(1u16, false)]);

        // A fixed-shape, unfiltered dataset is contiguous and reports no
        // maximum, the way `Dataset::maxshape` reports an on-disk one.
        let plain = e
            .staged_dataset_meta(&ObjectPathBuf::parse("plain"))
            .unwrap();
        assert_eq!(plain.dimensions, vec![1]);
        assert_eq!(plain.maxshape, None);
        assert!(!plain.chunked);
        assert!(plain.filters.is_empty());

        // Only datasets answer.
        e.create_group(&ObjectPathBuf::parse("g")).unwrap();
        assert!(e.staged_dataset_meta(&ObjectPathBuf::parse("g")).is_none());
        assert!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("existing"))
                .is_none()
        );
    }

    #[test]
    fn a_deletion_hides_nothing_until_a_creation_replaces_it() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.delete(&ObjectPathBuf::parse("existing")).unwrap();
        // The object is still in the file, and still readable, so a handle must
        // not be told it is staged.
        assert_eq!(kind(&e, "existing"), None);
        assert!(e.staged_children(&ObjectPathBuf::parse("")).is_empty());

        e.stage_created_dataset(&ObjectPathBuf::parse("existing"), i32_dataset(&[9]))
            .unwrap();
        let staged = e.staged_object(&ObjectPathBuf::parse("existing")).unwrap();
        assert_eq!(staged.kind, StagedKind::Dataset);
        assert!(staged.replaces_link, "the same commit removes the link");
        assert_eq!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("existing"))
                .unwrap()
                .dimensions,
            vec![1]
        );
        assert!(e.staged_children(&ObjectPathBuf::parse(""))[0].replaces_link);
    }

    #[test]
    fn a_creation_colliding_with_a_surviving_link_is_refused_where_it_is_staged() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        // No deletion beside it, so this creation collides with a link the
        // commit keeps. That is the commit's own refusal, raised where the call
        // is made so no handle onto the creation is ever handed back: such a
        // handle would have addressed the file's `existing` while claiming to
        // address the new one.
        let err = e
            .stage_created_dataset(&ObjectPathBuf::parse("existing"), i32_dataset(&[9]))
            .unwrap_err();
        assert!(
            matches!(&err, Error::EditUnsupported(m) if m.contains("already exists")),
            "got: {err}"
        );
        // The same rule for a group, which is also the kind change a listing
        // would otherwise misreport as turning a dataset into a group.
        assert!(e.create_group(&ObjectPathBuf::parse("existing")).is_err());

        // Nothing was staged, so the file's own object is still what the name
        // means, and the session has nothing left to refuse at commit time.
        assert_eq!(kind(&e, "existing"), None);
        assert!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("existing"))
                .is_none()
        );
        assert!(e.staged_children(&ObjectPathBuf::parse("")).is_empty());
        e.commit().unwrap();
    }

    #[test]
    fn a_second_creation_at_a_staged_path_is_refused_where_it_is_staged() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.stage_created_dataset(&ObjectPathBuf::parse("fresh"), i32_dataset(&[1, 2, 3]))
            .unwrap();

        // The staged set is indexed by path and keeps the first record there, so
        // a second creation would be staged behind the first and the handle this
        // call hands back would answer for the first one's shape and datatype.
        let err = e
            .stage_created_dataset(&ObjectPathBuf::parse("fresh"), i32_dataset(&[9]))
            .unwrap_err();
        assert!(
            matches!(&err, Error::EditUnsupported(m) if m.contains("already stages")),
            "got: {err}"
        );
        // Refused, not replaced: the first creation is exactly as it was, and it
        // is still the only thing staged at that name.
        assert_eq!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("fresh"))
                .unwrap()
                .dimensions,
            vec![3]
        );
        assert_eq!(
            child_names(&e, ""),
            vec![("fresh".to_string(), StagedKind::Dataset)]
        );

        // The two kinds collide with each other for the same reason: one path
        // cannot name both, and whichever the index found first is what every
        // handle onto it would report.
        assert!(e.create_group(&ObjectPathBuf::parse("fresh")).is_err());
        e.create_group(&ObjectPathBuf::parse("g")).unwrap();
        assert!(
            e.stage_created_dataset(&ObjectPathBuf::parse("g"), i32_dataset(&[9]))
                .is_err()
        );
        assert_eq!(kind(&e, "g"), Some(StagedKind::Group));

        e.commit().unwrap();
    }

    #[test]
    fn a_replacement_is_staged_once_and_a_withdrawal_frees_the_name_again() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        // A creation over a link this session deletes is a replacement, which
        // the file arm admits — but only one of them, since the second would
        // still be staged behind the first.
        e.delete(&ObjectPathBuf::parse("existing")).unwrap();
        e.stage_created_dataset(&ObjectPathBuf::parse("existing"), i32_dataset(&[9]))
            .unwrap();
        assert!(
            e.stage_created_dataset(&ObjectPathBuf::parse("existing"), i32_dataset(&[8, 8]))
                .is_err()
        );
        assert_eq!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("existing"))
                .unwrap()
                .dimensions,
            vec![1]
        );

        // Deleting a staged creation withdraws it, which is what puts the name
        // back within reach of another creation.
        e.delete(&ObjectPathBuf::parse("existing")).unwrap();
        e.stage_created_dataset(&ObjectPathBuf::parse("existing"), i32_dataset(&[8, 8]))
            .unwrap();
        assert_eq!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("existing"))
                .unwrap()
                .dimensions,
            vec![2]
        );
        e.commit().unwrap();
    }

    #[test]
    fn a_group_staged_twice_is_one_group_and_stays_allowed() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        // Two group creations at one path name one node: the commit builds a
        // single group from them, so a handle onto either addresses it and
        // nothing is misreported. Re-staging is how attributes and children are
        // added to a group already staged.
        e.create_group(&ObjectPathBuf::parse("g")).unwrap();
        e.create_group(&ObjectPathBuf::parse("g")).unwrap();
        assert_eq!(kind(&e, "g"), Some(StagedKind::Group));
        assert_eq!(
            child_names(&e, ""),
            vec![("g".to_string(), StagedKind::Group)]
        );
        e.stage_created_dataset(&ObjectPathBuf::parse("g/inner"), i32_dataset(&[7]))
            .unwrap();
        e.commit().unwrap();
    }

    #[test]
    fn a_creation_under_a_deleted_group_the_session_does_not_rebuild_shadows_nothing() {
        // `delete("g")` carries `g/inner` away with it, but only a commit that
        // builds `g` again can put anything back under that path — and this one
        // does not, so the commit refuses the batch. Until then the file's own
        // `g/inner` is what the name means: calling the staged creation beside
        // it a replacement would make a live, readable dataset answer
        // `NotCommitted` and vanish from its group's listing (issue #392).
        let dir = tempdir().unwrap();
        let path = dir.path().join("prefix.h5");
        let mut b = crate::writer::FileBuilder::new();
        let mut g = b.create_group("g");
        g.create_dataset("inner").with_i32_data(&[1]);
        b.add_group(g.finish());
        b.write(&path).unwrap();

        let mut e = WriteEngine::open_rw_with_strategy(
            &path,
            crate::source::MetadataCacheConfig::disabled(),
            FileLocking::Enabled,
            MemoryStrategy::Mirrored,
        )
        .unwrap();
        e.delete(&ObjectPathBuf::parse("g")).unwrap();
        // The name `g/inner` is still taken: a deletion of `g` alone hands it
        // over to nobody, so this is the collision it looks like rather than a
        // replacement.
        let err = e
            .stage_created_dataset(&ObjectPathBuf::parse("g/inner"), i32_dataset(&[9]))
            .unwrap_err();
        assert!(
            matches!(&err, Error::EditUnsupported(m) if m.contains("already exists")),
            "got: {err}"
        );
        assert_eq!(kind(&e, "g/inner"), None, "the file's own dataset");

        // A name the file does not hold stages, and is not a replacement of
        // anything; the commit refuses the batch for the overlap itself.
        e.stage_created_dataset(&ObjectPathBuf::parse("g/other"), i32_dataset(&[9]))
            .unwrap();
        assert!(
            !e.staged_children(&ObjectPathBuf::parse("g"))[0].replaces_link,
            "nothing rebuilds `g`, so its names are not handed over"
        );
        assert!(e.commit().is_err(), "a deletion overlapping an addition");
    }

    #[test]
    fn a_deleted_group_rebuilt_in_the_same_commit_hands_its_names_over() {
        // The other side of that rule: `g` is replaced, so a creation under it
        // owns its name from the moment it is staged and the object the commit
        // removes is no longer what the path means.
        let dir = tempdir().unwrap();
        let path = dir.path().join("replaced.h5");
        let mut b = crate::writer::FileBuilder::new();
        let mut g = b.create_group("g");
        g.create_dataset("inner").with_i32_data(&[1]);
        b.add_group(g.finish());
        b.write(&path).unwrap();

        let mut e = WriteEngine::open_rw_with_strategy(
            &path,
            crate::source::MetadataCacheConfig::disabled(),
            FileLocking::Enabled,
            MemoryStrategy::Mirrored,
        )
        .unwrap();
        e.delete(&ObjectPathBuf::parse("g")).unwrap();
        e.create_group(&ObjectPathBuf::parse("g")).unwrap();
        e.stage_created_dataset(&ObjectPathBuf::parse("g/inner"), i32_dataset(&[9]))
            .unwrap();
        assert_eq!(kind(&e, "g/inner"), Some(StagedKind::Dataset));
        assert!(
            e.staged_object(&ObjectPathBuf::parse("g/inner"))
                .unwrap()
                .replaces_link
        );
        assert!(e.staged_children(&ObjectPathBuf::parse("g"))[0].replaces_link);
        e.commit().unwrap();
    }

    #[rstest]
    #[case("")]
    #[case("/")]
    #[case(".")]
    fn the_root_cannot_be_deleted(#[case] spelling: &str) {
        // Nothing links to the root, so there is no link to remove — and an
        // empty path is a prefix of every other, so a deletion staged there
        // would make every creation in the session look like a replacement.
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        let err = e.delete(&ObjectPathBuf::parse(spelling)).unwrap_err();
        let Error::EditUnsupported(message) = &err else {
            panic!("expected EditUnsupported, got {err:?}");
        };
        assert_eq!(
            *message,
            "cannot delete the root group; delete its members instead"
        );
        e.stage_created_dataset(&ObjectPathBuf::parse("fresh"), i32_dataset(&[1]))
            .unwrap();
        assert!(
            !e.staged_object(&ObjectPathBuf::parse("fresh"))
                .unwrap()
                .replaces_link,
            "the refused deletion staged nothing, so this replaces nothing"
        );
    }

    #[rstest]
    #[case("")]
    #[case("/")]
    #[case(".")]
    fn the_root_group_cannot_be_created(#[case] spelling: &str) {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        let err = e.create_group(&ObjectPathBuf::parse(spelling)).unwrap_err();
        let Error::EditUnsupported(message) = &err else {
            panic!("expected EditUnsupported, got {err:?}");
        };
        assert_eq!(*message, "cannot create the root group");
        assert!(
            !e.has_staged_edits(),
            "the refusal staged nothing for a later commit to find"
        );
    }

    #[test]
    fn an_append_onto_a_staged_dataset_grows_the_pending_creation() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.stage_created_dataset(&ObjectPathBuf::parse("col"), i32_dataset(&[1, 2]))
            .unwrap();

        let mut b = AppendBuilder::new();
        b.append_i32(&[3, 4]);
        e.stage_dataset_append_pending(&ObjectPathBuf::parse("col"), b)
            .unwrap();
        assert_eq!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("col"))
                .unwrap()
                .dimensions,
            vec![4]
        );
        // Folded into the creation rather than queued beside it, so the commit
        // has one dataset to write and no append to apply to it.
        assert!(e.staged.appends.is_empty());

        e.commit().unwrap();
        let addr = crate::group_v2::resolve_path_any_from_source(
            &e.image(),
            AccessMode::ReadWrite,
            e.superblock(),
            &ObjectPath::parse("col"),
        )
        .unwrap();
        assert!(addr > 0);
    }

    #[test]
    fn an_append_through_a_handle_onto_the_replaced_object_is_refused() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.delete(&ObjectPathBuf::parse("existing")).unwrap();
        e.stage_created_dataset(&ObjectPathBuf::parse("existing"), i32_dataset(&[100]))
            .unwrap();

        // `stage_dataset_append` is the entry point a handle onto the object in
        // the file uses. Growing the replacement under it would silently move
        // the elements to another object.
        let mut b = AppendBuilder::new();
        b.append_i32(&[200]);
        assert!(matches!(
            e.stage_dataset_append(&ObjectPathBuf::parse("existing"), b),
            Err(Error::EditUnsupported(_))
        ));
        assert_eq!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("existing"))
                .unwrap()
                .dimensions,
            vec![1],
            "the refusal must leave the replacement alone"
        );
        assert!(e.staged.appends.is_empty());
    }

    #[test]
    fn an_append_the_staged_dataset_cannot_carry_is_refused_without_changing_it() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.stage_created_dataset(&ObjectPathBuf::parse("col"), i32_dataset(&[1, 2]))
            .unwrap();

        let mut wrong_type = AppendBuilder::new();
        wrong_type.append_f64(&[1.0]);
        assert!(matches!(
            e.stage_dataset_append_pending(&ObjectPathBuf::parse("col"), wrong_type),
            Err(Error::AppendUnsupported(_))
        ));

        let mut partial = AppendBuilder::new();
        partial.append_raw(&[0u8, 1, 2]);
        assert!(matches!(
            e.stage_dataset_append_pending(&ObjectPathBuf::parse("col"), partial),
            Err(Error::AppendUnsupported(_))
        ));

        // A finite maximum is a promise the commit would refuse to break.
        let mut capped = DatasetBuilder::new("");
        capped
            .with_i32_data(&[1, 2])
            .with_maxshape(&[MaxExtent::Fixed(3)])
            .with_chunks(&[2]);
        e.stage_created_dataset(&ObjectPathBuf::parse("capped"), capped)
            .unwrap();
        let mut over = AppendBuilder::new();
        over.append_i32(&[3, 4]);
        assert!(matches!(
            e.stage_dataset_append_pending(&ObjectPathBuf::parse("capped"), over),
            Err(Error::AppendUnsupported(_))
        ));

        // Nothing the refusals touched changed.
        assert_eq!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("col"))
                .unwrap()
                .dimensions,
            vec![2]
        );
        assert_eq!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("capped"))
                .unwrap()
                .dimensions,
            vec![2]
        );
    }

    #[test]
    fn an_append_onto_a_committed_dataset_still_stages_an_append() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        let mut b = AppendBuilder::new();
        b.append_i32(&[4]);
        e.stage_dataset_append(&ObjectPathBuf::parse("existing"), b)
            .unwrap();
        assert_eq!(e.staged.appends.len(), 1);

        // And so does one made through a handle whose creation was committed
        // between the caller's check and this lock.
        let mut b = AppendBuilder::new();
        b.append_i32(&[5]);
        e.stage_dataset_append_pending(&ObjectPathBuf::parse("existing"), b)
            .unwrap();
        assert_eq!(e.staged.appends.len(), 2);
    }

    #[test]
    fn deleting_a_staged_creation_withdraws_it() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.stage_created_dataset(&ObjectPathBuf::parse("col"), i32_dataset(&[1, 2]))
            .unwrap();
        e.delete(&ObjectPathBuf::parse("col")).unwrap();
        // Withdrawn, not queued for a deletion the commit could not perform.
        assert_eq!(kind(&e, "col"), None);
        assert!(e.staged.deletes.is_empty());
        assert!(!e.has_staged_edits());
        e.commit().unwrap();
    }

    #[test]
    fn deleting_a_staged_group_withdraws_its_staged_subtree() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.create_group(&ObjectPathBuf::parse("g")).unwrap();
        e.create_group(&ObjectPathBuf::parse("g/inner")).unwrap();
        e.stage_created_dataset(&ObjectPathBuf::parse("g/inner/col"), i32_dataset(&[1]))
            .unwrap();
        e.set_group_attr(&ObjectPathBuf::parse("g"), "kind", AttrValue::I64(1))
            .unwrap();

        e.delete(&ObjectPathBuf::parse("g")).unwrap();
        assert_eq!(kind(&e, "g"), None);
        assert_eq!(kind(&e, "g/inner"), None);
        assert_eq!(kind(&e, "g/inner/col"), None);
        assert!(
            !e.has_staged_edits(),
            "the attribute and the subtree go with the group"
        );
        e.commit().unwrap();
    }

    #[test]
    fn deleting_a_staged_replacement_leaves_the_plain_deletion() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.delete(&ObjectPathBuf::parse("existing")).unwrap();
        e.stage_created_dataset(&ObjectPathBuf::parse("existing"), i32_dataset(&[9]))
            .unwrap();
        // Changing one's mind about the replacement leaves the deletion of the
        // object in the file, which is what was asked for first.
        e.delete(&ObjectPathBuf::parse("existing")).unwrap();
        assert_eq!(kind(&e, "existing"), None);
        assert_eq!(e.staged.deletes.len(), 1);
        e.commit().unwrap();
        assert!(
            crate::group_v2::resolve_path_any_from_source(
                &e.image(),
                AccessMode::ReadWrite,
                e.superblock(),
                &ObjectPath::parse("existing"),
            )
            .is_err()
        );
    }

    #[test]
    fn a_batch_that_fails_leaves_the_staged_index_matching_the_staged_set() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.stage_created_dataset(&ObjectPathBuf::parse("kept"), i32_dataset(&[1]))
            .unwrap();
        let refused: Result<(), Error> = e.stage_atomically(|s| {
            s.create_group(&ObjectPathBuf::parse("gone"))?;
            s.stage_created_dataset(&ObjectPathBuf::parse("gone/col"), i32_dataset(&[2]))?;
            Err(Error::EditUnsupported("refused on purpose"))
        });
        assert!(refused.is_err());
        // The index must forget what the rewind dropped, or a lookup would hand
        // back a handle onto an entry that is no longer there.
        assert_eq!(kind(&e, "gone"), None);
        assert_eq!(kind(&e, "gone/col"), None);
        assert_eq!(kind(&e, "kept"), Some(StagedKind::Dataset));
        assert!(e.staged_children(&ObjectPathBuf::parse("gone")).is_empty());

        // And it must forget them for good: an index entry left behind would be
        // found by the *next* creation at that path and keep it pointing at the
        // position the rewind dropped, so the retry would stage a dataset no
        // lookup could see.
        // Staged behind an unrelated creation, so the retry lands at a
        // *different* position than the one the rewind dropped: an index entry
        // left behind would still name the old one.
        e.create_group(&ObjectPathBuf::parse("other")).unwrap();
        e.stage_created_dataset(&ObjectPathBuf::parse("other/col"), i32_dataset(&[3]))
            .unwrap();
        e.create_group(&ObjectPathBuf::parse("gone")).unwrap();
        e.stage_created_dataset(&ObjectPathBuf::parse("gone/col"), i32_dataset(&[2]))
            .unwrap();
        assert_eq!(kind(&e, "gone"), Some(StagedKind::Group));
        assert_eq!(kind(&e, "gone/col"), Some(StagedKind::Dataset));
        assert_eq!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("gone/col"))
                .unwrap()
                .dimensions,
            vec![1]
        );
        e.commit().unwrap();
    }

    #[test]
    fn an_edit_that_changes_staged_work_is_refused_inside_a_batch() {
        let dir = tempdir().unwrap();
        let mut e = open_session(&dir.path().join("q.h5"));
        e.stage_created_dataset(&ObjectPathBuf::parse("col"), i32_dataset(&[1, 2]))
            .unwrap();
        // `rewind` undoes a batch by truncating, which is exact only while
        // staging appends. Both operations that do otherwise say so here rather
        // than leaving half of themselves behind in a refused batch.
        let folded: Result<(), Error> = e.stage_atomically(|s| {
            let mut b = AppendBuilder::new();
            b.append_i32(&[3]);
            s.stage_dataset_append_pending(&ObjectPathBuf::parse("col"), b)
        });
        assert!(matches!(folded, Err(Error::EditUnsupported(_))));
        let withdrawn: Result<(), Error> =
            e.stage_atomically(|s| s.delete(&ObjectPathBuf::parse("col")));
        assert!(matches!(withdrawn, Err(Error::EditUnsupported(_))));
        assert_eq!(
            e.staged_dataset_meta(&ObjectPathBuf::parse("col"))
                .unwrap()
                .dimensions,
            vec![2]
        );
    }
}

#[cfg(test)]
mod object_header_wrap_tests {
    use super::*;

    /// A region whose messages cannot be walked is reported, not asserted away.
    ///
    /// The `debug_assert!(false)` this replaced split the behavior by build
    /// profile: a test build panicked, and a release build wrote the header with
    /// no Attribute Info message — the zero-`num_attrs` defect the normalization
    /// exists to prevent. Asserted as an `Err` because that is the one answer
    /// both profiles can give.
    #[test]
    fn an_unwalkable_region_is_refused_rather_than_wrapped() {
        // One version 2 header message — type byte, 2-byte size, flags byte —
        // whose size field claims far more body than the region holds.
        let mut region = vec![0x0Cu8]; // Attribute
        region.extend_from_slice(&0xFFFFu16.to_le_bytes());
        region.push(0); // flags
        region.extend_from_slice(&[0u8; 4]); // a body far shorter than declared

        let err = build_v2_object_header(&OhRegion::new(region, OhHeaderProps::PLAIN)).unwrap_err();
        assert!(
            matches!(err, Error::EditUnsupported(_)),
            "an unwalkable region gave {err:?}"
        );
    }

    /// The walkable case still normalizes: a header carrying inline attributes
    /// comes back with the Attribute Info message that declares their count.
    #[test]
    fn a_walkable_region_still_gains_its_attribute_info() {
        let body = [0u8; 8];
        let mut region = vec![0x0Cu8]; // Attribute
        region.extend_from_slice(&(body.len() as u16).to_le_bytes());
        region.push(0); // flags
        region.extend_from_slice(&body);

        let oh =
            build_v2_object_header(&OhRegion::new(region.clone(), OhHeaderProps::PLAIN)).unwrap();
        assert_eq!(&oh[..4], b"OHDR");
        assert!(
            oh.len() > 8 + region.len() + 4,
            "the wrapped header did not grow by an Attribute Info message"
        );
    }
}