# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
> Pre-1.0 caveat: while we are still on `0.x`, every minor version (`0.x.0`)
> may contain breaking API or on-disk changes. Read the **Breaking changes**
> section before upgrading.
## [0.7.0] - 2026-09-06
Follows `hyperfile 0.7`, where `SegmentId` names a stored object rather than a
checkpoint number. Two defects surfaced while adapting, and they are the reason to
read past the breaking-change list: the `wal` feature panicked on any directory
flush, and a directory had begun writing objects it could not then read.
The minor version skips `0.6` to line up with the `hyperfile` release this speaks
to. This crate's public API *is* that crate's types — `SegmentId` and `Cno` in the
flush surface, `Inode` and `InodeRaw` in the dirents — so a reader who knows which
`hyperfile` wrote a container knows which `hyperdir` reads it, and a number that
drifts from it invites the pairing to be guessed. `hyperfile-cleaner` adopted the
same rule. `0.6.0` was never published, so nothing is skipped in substance.
### Breaking changes
- **`hyperfile` requirement moves from `0.6.11` to `0.7`.** `SegmentId` is no
longer an alias for `u64`: it is a struct naming a stored object — a checkpoint
number and, under `BlockPtrFormat::PartedSegment`, which of that checkpoint's
objects. Consumers must move together, since a `SegmentId` from one build is not
the other's, and `hyperfile` and `hyperdir` must still resolve to a single
`btree-ondisk` (`0.20.0`, which both pin).
- **`fs_release` and `fs_flush` return `Cno`.** The value is unchanged and the type
is still `u64`; `Cno` is what `hyperfile` now calls a checkpoint number. The
public surface answers in those because that is what a caller pins and logs,
while the trait beneath answers in `SegmentId`.
- **A directory container records `BlockPtrFormat::Flat`** in its inode's
`i_meta_config`, rather than whatever the caller's config carried. See the fix
below for why. No change for containers created by `0.5.0`, which recorded the
then-default `MicroGroup` — that names its objects identically.
### Fixed
- **The `wal` feature panicked on every directory flush.** Its three methods were
`todo!()`, on the theory that nothing calls them unless a log is configured. That
was false: `wal_drop_superseded_prefixes` runs at the end of `flush_process`,
gated on `#[cfg(feature = "wal")]` and never on whether a log exists, and its
loop runs at least once even with nothing deferred — so it always reached
`wal_spawn_delete_segment`. `fs_compact` on a three-entry directory panicked
there. `hyperfile 0.7` widened the trait to nine wal methods, which forced the
question rather than allowing another three stubs.
All nine now report "no wal is configured", which is both true and the answer
`hyperfile` itself gives when its `wal_mut()` is `None` (see `wal_seal_barrier`
returning `Ok(())` on that branch). `e2e_s3` with `--all-features` went from 21
of 21 panicking to 21 of 21 passing.
**Enabling `wal` on this crate gives directory operations no write-ahead
logging.** The feature exists so a consumer that enables `hyperfile/wal` for its
*file* containers still compiles, since the trait demands these whenever
`hyperfile` has the feature on.
- **A directory wrote objects it could not read back, silently.** After the
adaptation compiled and every unit test passed, 20 of 21 end-to-end cases failed
on `read_dir` answering *empty* for a directory that had just been compacted — no
error, just nothing.
Two things read a container's pointer format, for different purposes, and both
have to agree. Pointer encode/decode reads the bmap's userdata, which this crate
has always hard-coded to `Flat`. Object naming reads the *inode's*
`i_meta_config`, through `Segment::object_of`, and that came from `config.meta`
— whose default `hyperfile 0.7.0` changed to `PartedSegment`. The disagreement
was harmless while the old default was `MicroGroup`, since `Flat` and
`MicroGroup` name a checkpoint's object identically and the micro-group field
takes no part in addressing. `PartedSegment` names it `<segid>.<part>`. So a
directory wrote `0000000001.0` and then asked for `0000000001`.
`HyperDirFile::new` now forces `Flat` into the meta config it records, so all
three agree. That is the right way round rather than adopting parts: parts exist
so a container can stream dirty *data* out under memory pressure, and a directory
has no data blocks — its contents are the bmap's leaves.
### Added
- Two unit tests pinning the format agreement, since its failure mode is silence:
one asserts the inode's recorded format equals the bmap's and that the resulting
object name carries no part, the other records that upstream's default is parted
so the override is not later mistaken for redundant.
### Changed
- `do_open` converts a checkpoint number with `SegmentId::new_from_cno`, the call
`hyperfile`'s own `do_open` makes, in place of an `as` cast that was never a cast.
- `demote_data_blocks_dirty` (new in the trait) is empty, for the same reason the
data-cache methods beside it are: a directory has no data blocks, so
`get_data_blocks_dirty` always answers empty and the flush path never names any.
- `check_writable` (new) is `Ok(())`; `hyperfile` refuses writes once publishing has
failed unrecoverably, which it tracks in wal state this crate does not have.
- `pending_parts_get` / `_advance` / `_clear` (new) get real state mirroring
`hyperfile`'s. Nothing should accumulate there — partials come from dirty data
pressure and a directory has none — but the flush path reads and writes them
unconditionally, so honest state beats a no-op resting on a reachability argument.
That kind of argument is what produced the `wal` panic above.
### Verified
Against real S3: 19 unit tests; `e2e_s3` 21 on default features and 21 with
`--all-features`; `e2e_concurrent` 18 with `--all-features`; `e2e_blob`; `e2e_fault`
under `fault-injection`. `clippy --all-targets` clean with `-D warnings` on default,
`--all-features` and `fault-injection`.
## [0.5.0] - 2026-08-27
Adopts `btree-ondisk 0.20.0` and exposes the reachability enumeration a
directory needs, so a reclaimer can be told which storage a directory still
occupies instead of guessing. Also fixes a latent alignment hazard on the bmap
open path.
Everything here is relative to `0.4.0`.
### Breaking changes
- **`btree-ondisk` `0.18.1` → `0.20.0`, `hyperfile` `0.6.10` → `0.6.11`.**
These move together and consumers must move with them. This crate implements
`hyperfile` traits whose signatures mention `btree-ondisk` types
(`BtreeNodeDirty`, `BMap`) and passes `hyperfile`'s block loader into its own
bmap, so `hyperdir` and `hyperfile` must resolve to **one** `btree-ondisk`
version. Two copies in one dependency graph do not compile ("found a different
`BtreeNodeDirty`"; `DirFileEntryRaw: NodeValue` and `L: BlockLoader<u64>`
unsatisfied). `hyperfile 0.6.11` pins the same `0.20.0`. No on-disk change.
### Added
- **Directory reachability**: `HyperDir::fs_reachable_meta_ptrs()` returns every
external storage location a directory occupies, and
`HyperDir::fs_reachable_segments()` returns the same answer as a set of
staging segment ids (decoded here, since a bare `BlockPtr` cannot be decoded
by a caller). Also on `HyperDirFile` as `reachable_meta_ptrs` /
`reachable_segments`.
A directory's B-tree leaves hold `DirFileEntryRaw` *inline payload*, not
storage locations, so the bmap's metadata nodes are the only external storage
it occupies — that set is the complete reachable set, and the walk stops one
level above the leaves, fetching no leaf node. Backed by `btree-ondisk`'s
`collect_meta_ptrs`; the sibling `collect_ptrs` is not usable here and returns
`InvalidInput`, because this crate's `V` differs from its `P`.
Both entry points are **fail-closed**: an `Err` means reachability is unknown
and a caller must reclaim nothing, never a short result. A generic reclaimer
pointed at a `DIR/<uuid>` container must use these rather than `collect_ptrs`.
### Fixed
- **The bmap root buffer could be rejected for misalignment.** `BMap::read`
parses the root header through `BtreeNode::from_slice_ref`, which requires an
8-byte-aligned buffer and returns `InvalidInput` otherwise. Two call sites
(opening a directory, and reloading to the latest bmap) copied `i_bmap` into a
local `[u8; N]` first — alignment 1, so they were correctly aligned only by
luck of stack layout. Both now pass `&raw_inode.i_bmap`, which sits at an
8-aligned offset inside `repr(C, align(8))` `InodeRaw`. (`BMap::read` copies
the bytes into its own aligned buffer, so passing the field reference retains
no borrow.) Pre-existing since the alignment check predates `0.20.0`.
### Tests
- `tests/e2e_reachability.rs`: a populated, compacted, reopened directory
reports a non-empty duplicate-free set of metadata-node locations resolving to
at least one staging segment (40 children → 7 nodes; 80 → 14); the answer is
stable across repeated calls; and growth never reports *fewer* nodes than
before while every child stays readable — the regression class that motivated
the upstream API.
- Unit tests pin the two properties the implementation depends on:
`collect_ptrs` is refused with `InvalidInput` for this crate's `V != P` shape
while `collect_meta_ptrs` is served, and `InodeRaw` is 8-aligned with `i_bmap`
at an 8-aligned offset.
## [0.4.0] - 2026-08-27
The release that turns this crate from a directory container into a POSIX-shaped
namespace: hard links, cross-directory rename, two-phase delete, advisory locks,
extended attributes, inline small files, and the crash-recovery machinery that
makes those safe when several mounts race on the same tree. Most of the work
landed as fixes to concurrent-access bugs found by multi-client tests against
real S3.
Everything here is relative to `0.3.0`, which is the last published release.
### Breaking changes
- **On-disk layout is not compatible with `0.3.0`.** Directories and files are
addressed by UUID, scatter objects are named suffix-first
(`.inode` / `.tombstone` / `.link` / …) and keyed by name rather than by a
CRC64 hash of the name. There is no migration; this is intended for fresh
trees.
- **`hyperfile` requirement moves from `^0.3` to `0.6.10`.** `hyperfile 0.4.0`
reordered `InodeRaw`'s on-disk layout — `i_size`, `i_blocks`, `i_last_seq`,
`i_last_cno` and `i_bmap` are now a contiguous tail so that tail can double as
the payload of an inlined small file. The struct is still 160 bytes but the
serialized offsets differ, so an inode written by a `0.3.x`-based tool is
misread by `0.4.0` and later, and vice versa. `InodeRaw` is embedded in the
dirents this crate reads and writes, so both sides must be built against the
same `hyperfile`.
- **`fs_unlink` / `fs_unlink_with_interceptor` changed meaning.** They used to
delete a directory container identified by a URI. They now unlink a *name*
within a parent directory:
`fs_unlink(client, layout, bucket, parent_dir_uuid, name, …)`.
- **`read_entry` / `fs_read_entry` take a name, not a hash.** The parameter went
from `hash: &EntryNameHash` to `name: &str`, following the name-first scatter
layout. `read_entry_raw` is the `DirFileEntryRaw` variant.
- **`read_dir` / `fs_read_dir` take `&self`, not `&mut self`.** Reading a
directory no longer needs exclusive access; compaction is the only `&mut`
operation.
- **`path_encode` gained a `uuid: &Uuid` parameter**, since a scatter key is now
scoped to the directory's UUID.
- **`DEFAULT_DIR_INODE_MARKER` is gone.** Scatter objects are named by suffix
now: `DEFAULT_DIR_INODE_SUFFIX` (`.inode`) and
`DEFAULT_DIR_TOMBSTONE_SUFFIX` (`.tombstone`), plus `.link` for a link intent.
### Added
- **Namespace operations.** `fs_create_file` (namespace entry returning a
byte-I/O handle), `fs_create_file_at`, hard links with authoritative `nlink`
kept in the file inode, same-directory and cross-directory rename, rename that
replaces an existing destination, `fs_reclaim`, `fs_exchange` (atomic
same-directory swap, `RENAME_EXCHANGE`), root bootstrap and `rmdir`.
- **Inline small files, two levels.** A tiny file lives entirely inside its
inode, with no separate data object. Second-level inline ("blob") files extend
this to the directory namespace, with `fs_update_inline` and
`fs_update_blob_meta` for metadata-only dirent updates.
- **Two-phase delete (`PreDelete`).** `unlink` publishes an intent and defers the
tombstone work to compaction, so the expensive part is not on the caller's
path.
- **S3-backed advisory file locks** (`lock` module, `_lock` sidecar object,
global lock mode), with a shared clock derived from S3 `Date` response headers
so lease expiry does not depend on client clocks. The lock is an unfenced TTL
lease — documented, with a lease-expiry test.
- **Extended attributes**, stored as sidecar objects with last-write-wins
semantics.
- **Open leases for open-but-unlinked files across mounts.** Layout keys for the
lease, `fs_gc` defers keep-open files that hold a live lease, and the GC TTL is
a settable process-wide value.
- **Tree-wide maintenance.** Directory enumeration, UUID-level orphan GC
(`fs_gc_orphans`), crash-safe reclaim of rename-displaced children via a
reclaim intent, crash-safe `fs_link` via a `.link` intent, and per-key intent
recovery with a key classifier and the `.unref` orphan scheme.
- **`HyperDirLayout::heartbeat_key`** for mount liveness heartbeats.
- **Handle-less name resolution** (`fs_resolve_entry_fast`), which drops the
lookup's redundant inode GET: zero inode GETs while a name still has pending
scatter, one on the compacted fallback, versus two for open-then-resolve.
- **Opt-in S3 fault injection** behind the `fault-injection` feature
(`S3FaultInterceptor`), matching by operation name and firing by request count
so an injected run is reproducible. Absent from a normal build.
### Fixed
- **A rename dropped the blob version pointer** — data loss for blob files.
- **A partial scatter listing was returned as if complete.** A failed
`ListObjectsV2` page was silently skipped, so a name's newest
`PreDelete`/`Delete` could be missed and an unlinked file would reappear.
SQLite's rollback-journal pattern hit this and failed transactions with
"disk I/O error". Page errors now propagate.
- **`read_dir` order was arbitrary and unstable.** Entries were merged into a
`HashMap` and returned via `.into_values()`, so two calls could see different
orders. Callers that paginate by index (a FUSE `readdir` cookie) then re-sent
some entries and skipped others. The merged list is sorted by name, which is
invariant under compaction since folding never renames.
- **`unlink` nlink accounting is now exactly-once**, and **hard link is safe
under concurrent same-name races** (previously an nlink leak). `ctime` advances
on any link/unlink `nlink` change.
- **Cross-directory rename is move-once**, source-scoped — concurrent renames of
the same source could previously lose data.
- **Same-name identity races.** Scatter resolution is identity-aware, so a
delete tombstone cannot shadow a same-name recreate, and displaced same-name
identities are reclaimed so create/rename races never orphan.
- **Staleness against the concurrent compactor** in `readdir`, `lookup` and
`rename`; scatter fetch and `fs_link` tolerate compaction landing underneath
them.
- **Fail-closed recovery.** Orphan GC and reclaim recovery now error out when a
directory's or destination parent's references cannot be read, instead of
proceeding on a partial view.
- **Per-object `DeleteObjects` failures are detected** in
`remove_scatter_inodes`. `DeleteObjects` is a partial-success API, so an `Ok`
response can still carry per-key errors.
- **CRC64 directory-key collisions** are resolved by open addressing.
- **S3 SDK errors map to precise `io::ErrorKind`s** rather than collapsing into
`Other`.
### Changed
- A same-directory rename commits its scatter fold and the rename in **one**
flush instead of two. Measured against real S3 on the write-temp-then-rename
pattern that git, rsync and editors use: median 842ms → 676ms (~20%). A rename
over a directory with no pending scatter is unchanged, since that case never
flushed twice.
- `tokio` features narrowed; indentation normalized; clippy clean with
`-D warnings`.
### Tests
- 15 unit tests plus four end-to-end suites against real S3, all `#[ignore]`d and
run explicitly: `e2e_s3` (21 cases, scatter and crash-recovery heavy),
`e2e_concurrent` (18 cases, 2/4/8-client races over create, link, unlink,
rename, rmdir, compaction, orphan GC, reclaim, xattr and locks), `e2e_blob`,
and `e2e_fault` (needs `fault-injection`).
### Known limitations
- The `wal` feature's three `HyperTrait` methods (`wal_set_mem_segment`,
`wal_clear_mem_segment`, `wal_spawn_delete_segment`) are still `todo!()`, so
enabling `wal` panics on the first flush. It compiles — which is why a consumer
that enables `hyperfile/wal` must also enable `hyperdir/wal` to satisfy the
trait — but it must not be exercised on a write path.
- The advisory lock is an unfenced TTL lease: a holder that stalls past its lease
is not prevented from acting on stale state.
## [0.3.0] - 2026-05-27
### Changed
- `hyperfile` `0.2` → `0.3`, `btree-ondisk` `0.16` → `0.18`.
- Narrowed `tokio` features, typo and indentation cleanups, clippy fixes.
## [0.2.0] - 2025-09-23
First packaged release: directory contents held in a hyperfile B+tree, `stat`
updates, a flush lock, the `wal` feature gate, and crate metadata (README,
license, manifest).