# Testing Guide
This directory holds Hyperfile's test suite. Tests are organized by what
they exercise and what dependencies they require.
## Layout
```
tests/
├── README.md ← this file
├── common/
│ └── mod.rs ← direct-API fixtures, interceptors
├── common_reactor/
│ └── mod.rs ← reactor spawner helper
├── functional_memory_staging.rs ← core on in-memory staging, no S3
├── functional_recovery.rs ← open's roll-forward and abandonment, no S3
├── functional_dirty_memory.rs ← memory held by unflushed writes, no S3
├── functional_write_ref.rs ← writing blocks by reference, no S3
├── functional_split_checkpoint.rs ← a checkpoint larger than one object, no S3
├── functional_compaction.rs ← relocation, FLAG_COMPACTED and content_cno, no S3
├── replay_fsx_log.rs ← replay an fsx log, model-checked (optional)
├── integration_s3_smoke.rs ← happy-path create/write/read/truncate
├── integration_s3_rollback.rs ← rollback (exposure + correctness)
├── integration_s3_contract.rs ← flush contract (invariant) tests
├── integration_s3_concurrent.rs ← multi-instance concurrency
├── integration_s3_segment_open.rs ← segment summary read (`open`)
├── integration_s3_block_api.rs ← block borrow API (`fs_block*`)
├── integration_s3_local_disk_cache.rs ← local-disk data cache tier
├── integration_s3_read_timing.rs ← read-side counters (`read_timing`)
├── integration_s3_open_cno.rs ← read-only checkpoint views (`open_cno`)
├── integration_s3_taken_checkpoint.rs ← a checkpoint past the inode, on S3
├── integration_s3_multipart.rs ← segments uploaded as multipart
├── integration_s3_large_checkpoint.rs ← a checkpoint past 1 GiB, as several objects
├── integration_s3_node_reload.rs ← index nodes read back from S3 while others change
├── integration_s3_write_failure.rs ← a write whose index read fails part-way
├── integration_s3_compaction.rs ← fs_relocate and fs_content_cno on S3
├── integration_s3_error_kinds.rs ← the error kinds callers decide retries by
├── integration_s3_partial_delete.rs ← a DeleteObjects that refuses some keys fails unlink
├── public_error_kind.rs ← the public error classification, no S3
├── integration_reactor_s3_smoke.rs ← reactor smoke (default features)
├── integration_reactor_s3_block_api.rs ← reactor block access (`fh_with_block*`)
├── integration_reactor_s3_placement.rs ← where blocks are (`fh_read_plan`)
├── integration_reactor_s3_contention.rs ← reactor under concurrent use (no stalls)
├── integration_reactor_s3_range_lock.rs ← reactor + range-lock
├── integration_reactor_s3_wal.rs ← reactor + wal
└── integration_reactor_s3_all_features.rs ← reactor + wal + range-lock
```
`common/mod.rs` and `common_reactor/mod.rs` are not themselves test
binaries. They are consumed by each `integration_*_s3_*.rs` file via
`mod common; use common::*;` (and optionally `mod common_reactor; use
common_reactor::*;`). Cargo compiles each integration_*_s3_*.rs as its
own test binary; the common modules are shared source among them.
## Test categories
| Unit tests | inline in `src/**/*.rs` (`#[cfg(test)]` modules) | None | Yes |
| S3 integration tests | `tests/integration_s3_*.rs` | Real S3 bucket + AWS credentials | No (opt-in via `--ignored`) |
All unit tests must pass on every build. Integration tests are marked
`#[ignore]` so they don't run under plain `cargo test`; they must be
explicitly invoked.
## Running unit tests
```bash
cargo test --release --lib
```
No credentials, no network. Should complete in well under a second.
As of the latest commit there are approximately 195 unit tests across
modules like `meta_format`, `ondisk`, `segment`, `config`, `buffer`,
`inode`, `data_cache::mem_cache`, `file::flags`, `file::mode`,
`file::handler`, `file::lock` (under the `range-lock` feature), etc.
## Running S3 integration tests
Integration tests run against a real S3 bucket (or S3 Express One Zone
directory bucket). They exercise end-to-end scenarios that cannot be
meaningfully mocked: optimistic concurrency, conditional writes,
what open does with a checkpoint that reached storage without its inode,
and MPU semantics.
### Prerequisites
- A writable S3 bucket in some region. Tests create / delete objects
under a unique `hyperfile-test/<YYYYMMDD>/<ULID>/` prefix and best-
effort clean up at the end of each test.
- AWS credentials resolvable via the AWS SDK default chain:
environment variables, `~/.aws/credentials`, EC2 IMDS, ECS task
role, or SSO session.
### Configuration (environment variables)
Both variables are **required** — the integration suites panic with a
clear message if either is unset, because picking a default bucket
in library code would either silently hit someone else's account or
hard-code a specific environment's identifier into the repository.
| `HYPERFILE_TEST_BUCKET` | Name of the S3 bucket to write to. Must be an S3 general-purpose bucket or S3 Express One Zone directory bucket you own. |
| `HYPERFILE_TEST_REGION` | AWS region of the bucket. |
The tests use the AWS SDK's default credential provider chain
(env vars → `~/.aws/credentials` → IMDS → SSO). The identity in
use must have `s3:*` on the configured bucket.
### Running all integration suites
```bash
HYPERFILE_TEST_BUCKET=<your-bucket> \
HYPERFILE_TEST_REGION=<your-region> \
cargo test --release --tests -- --ignored --test-threads=1
```
`--test-threads=1` is important: the tests share a bucket prefix and
concurrent execution can cause listing / cleanup interference. Several
tests also deliberately race themselves; adding extra parallelism
obscures their state.
### Running one suite
Each file is its own `--test` target:
```bash
HYPERFILE_TEST_BUCKET=<your-bucket> \
HYPERFILE_TEST_REGION=<your-region> \
cargo test --release --test integration_s3_smoke -- --ignored --test-threads=1
```
Substitute `integration_s3_rollback`, `integration_s3_contract`, or
`integration_s3_concurrent` for the other suites.
### Running one test
Pass the test name (or a substring) before the `--`:
```bash
HYPERFILE_TEST_BUCKET=<your-bucket> \
HYPERFILE_TEST_REGION=<your-region> \
cargo test --release --test integration_s3_concurrent \
concurrent_two_writers_fail_fast_policy \
-- --ignored --test-threads=1 --nocapture
```
`--nocapture` is handy for diagnosing failures — it lets `println!`
output from the test show up even when the assertion passes.
### Compiling only (no run)
```bash
cargo test --release --tests --no-run
```
Useful in CI or when iterating on a test without paying for S3 calls.
### Enabling debug logs
Use `RUST_LOG` to see Hyperfile's internal debug traces — especially
useful when a concurrency / rollback test exposes a subtle ordering
issue.
```bash
RUST_LOG=hyperfile=debug \
HYPERFILE_TEST_BUCKET=... HYPERFILE_TEST_REGION=... \
cargo test --release --test integration_s3_concurrent <name> \
-- --ignored --test-threads=1 --nocapture
```
## Integration test suites
### `functional_memory_staging`
The only suite that needs nothing: no bucket, no credentials, no network,
and not `--ignored`. It runs the core against
[`staging::memory::MemoryStaging`](../src/staging/memory.rs), which keeps
segments and the inode in a map instead of an object store.
Everything below `Hyper` is generic over the staging backend, so the
segment format, the bmap, block pointers, the caches, flush and reopen all
run exactly as they do against a bucket — only where the bytes land
changes. Covers round trips, several flushes across segments, an unaligned
write preserving the bytes around it, holes reading as zeroes, truncate
both ways, `write_zero`, a 600-block file whose bmap outgrows one meta
node, the read counters, `unlink`, and that two separately constructed
handles do not share storage. One test reads back cold a flush whose index
passes 16 nodes, and a checkpoint written as partials, on staging built with
defaults: memory staging used to key published index nodes by a guessed pointer
format that agreed with the container's only below that, and every suite here
read holes past it. Another counts data requests: a batch of whole blocks makes
none, a block written in part one.
What it deliberately does not cover, because these belong to S3 rather
than to hyperfile: conditional writes and the OCC built on them, multipart
upload, and the error kinds a real service returns. Nor the reactor or the
`fs_*` wrappers — `Hyper` is bound to S3 staging and takes a client in
every constructor, so these tests drive `HyperFile` directly.
```bash
cargo test --test functional_memory_staging
```
Runs in about 0.1 s, under every feature combination including
`--no-default-features --features blocking`.
### `functional_recovery`
What open does with checkpoints that reached storage past the persisted
inode, and what a failed flush does about the ones it may have left. Also
on memory staging, so also needs nothing.
Every segment carries the inode of the checkpoint it completes, so a
segment no inode points at is a whole commit whose pointer update was
lost. Open takes it up — a reader in memory, writing nothing and waiting
for nothing; a writer for good — and a chain of them in order. A flush
that fails and rolls back abandons instead, so what its caller was told
failed stays failed, including when its retries run out or a later attempt
fails on something else. Anything under a checkpoint number that is not a
commit is stepped past: junk under each name a checkpoint answers to, a
real segment copied under another number, partials with no part 0, and a
checkpoint built on an inode that is no longer the inode — each segment
records the fingerprint of the state it was built on, and the base has to
follow every publish in the session for a crash after one to be taken up.
`each_check_on_a_commit_refuses_on_its_own` starts every case from a
genuine crashed commit and breaks exactly one thing, with an untouched
control, so each check the walk makes is shown to be the one that refuses.
Removing the counter check makes the walk re-take the same checkpoint for
ever — that check is what makes it terminate — and on memory staging that
loop never yields, so a hang there is a hang of the process. Run
revert experiments on this suite's binary under `timeout`.
```bash
cargo test --test functional_recovery
```
Runs in about 6 s, most of it one test waiting out flush retries.
### `functional_dirty_memory`
What unflushed writes cost in memory. It writes 128 MiB on memory staging
without a flush and reads the process's resident set before and after,
which has to grow by between 0.9 and 1.5 times the data: by the data and
the index over it, 1.04 times when measured. With data blocks allocated on
page boundaries it grew by 2.00 times, because glibc's malloc keeps a
header in front of every allocation and a page-aligned block's header
takes the page before it. The lower bound keeps writes that were never
held from passing.
It is alone in its binary, so nothing else allocates while it reads, and
Linux only, since it reads `/proc/self/status`. Not built under `blocking`.
```bash
cargo test --test functional_dirty_memory
```
Runs in about 0.1 s.
### `functional_write_ref`
Blocks written by reference (`HyperFile::write_ref`), whose bytes are read from
a `BlockSource` as each object uploads. By reference and by value store the
same bytes; later writes, truncates, `write_batch`, `block_mut` and `block`
take a reference out or read through it; a flush that rolls back discards
references with the rest; a partial written at a threshold holds them and they
are not stored again; the source is read on the blocking pool, shown with a
source that can only answer while the runtime's thread is free. Refused with
the WAL, which is the one test that build runs.
```bash
cargo test --test functional_write_ref
```
Runs in about 0.1 s.
### `functional_split_checkpoint`
A checkpoint larger than one object, written as data-only parts and then part 0
with the index and the inode. On memory staging with `segment_max_object_bytes`
at 64 KiB, so a few hundred blocks are many objects: each object within the
limit and part 0 the only one with metadata; the same writes split and unsplit
reading back the same, with a bmap several levels deep; part 0 landing after
the data parts and before the inode; a split after partials continuing their
numbering; a partial larger than one object split too; references split like
any block; a flush failing part-way leaving no commit and, when the checkpoint
already has partials, no reused part name on the next attempt; and the refusals
— `MicroGroup`, an index larger than one object, and the WAL build, which is
what that build runs.
```bash
cargo test --test functional_split_checkpoint
cargo test --features wal --test functional_split_checkpoint
```
Runs in about 0.3 s.
### `functional_compaction`
When a publish is marked as moving blocks only (`InodeRaw::FLAG_COMPACTED`),
so that `content_cno()` stays where it was. A flush of relocated blocks only is
marked, with `last_cno` moving on and `content_cno` not, mtime and ctime
untouched and holes and zero blocks left as they were. Not marked: any content
change in the same checkpoint — a write, a whole-block `write_zero`, a
truncate, a size set through the attributes, a block already dirty — a write
in an earlier partial of it, a failed content flush retried at the same
number, and a relocation published after a skipped number. The flag follows
the newest checkpoint: an attribute-only write keeps it, an extend published
the same way clears it, the next content flush clears it. References are not
relocated. `content_cno` is refused for a device node and an inline file.
`relocate` is refused with the WAL, which is what that build runs.
```bash
cargo test --test functional_compaction
```
Runs in about 0.1 s.
## The build matrix, and how it was chosen
Two rules, and the second exists because the first is not enough on its own.
Every combination below covers at least one `#[cfg]` that no other one reaches, so the
list is derived from the gates in the source rather than from a guess at what is
interesting. **And every feature declared in `Cargo.toml` is turned on by some row**,
which is the half that non-redundancy does not give you: `bench` has exactly one
`#[cfg]` in the whole crate, fell under no row, and had no row of its own, so it
compiled nowhere and therefore failed nowhere until a consumer needed it.
The second rule is checked by `every_declared_feature_is_built_somewhere` in
`src/lib.rs`, which reads `[features]` and fails on anything the matrix does not name.
Adding a feature fails that test until a row covers it.
```bash
cargo build --release --lib # 1
cargo build --release --lib --features wal # 2
cargo build --release --lib --features range-lock # 3
cargo build --release --lib --features wal,range-lock,concurrent-segment-build # 4
cargo build --release --lib --no-default-features --features blocking # 5
cargo build --release --lib --no-default-features --features blocking,wal # 6
cargo build --release --lib --no-default-features --features reactor # 7
cargo build --release --lib --no-default-features --features blocking,bench # 8
```
| 1 | `not(wal)`, `not(range-lock)`, `not(blocking)`, `meta_loader_batch` |
| 2 | `wal`, `all(wal, reactor)` |
| 3 | `range-lock`, `all(reactor, range-lock)` |
| 4 | `all(concurrent-segment-build, wal)` |
| 5 | `not(reactor)` |
| 6 | **`all(wal, blocking)`** |
| 7 | `not(meta_loader_batch)` with a reactor |
| 8 | `bench` |
Rows 6 and 8 are here because they were missing, and each cost a release. Row 8's
side is arbitrary — `bench.rs` has no gate on the running model, so `blocking,bench`
and `reactor,bench` fail and pass together; one row is enough.
Row 6 is here because it was missing. `wal` and `blocking` were each built alone, so
the two functions gated on both — `wal_flush_process_blocking` and
`kick_wal_protected_flush_blocking` — were never compiled, and a change to a return
type three other flush paths shared left that one broken. Nothing failed, because
nothing built it.
`reactor` and `blocking` are mutually exclusive and say so through a `compile_error!`,
so no row combines them.
Zero warnings, not just zero errors. A warning in one combination is often a `#[cfg]`
that is wider than the code it guards — which is how row 6 turned up a counter gated on
`wal` whose only caller is behind `reactor`.
Touch `src/lib.rs` between rows: cargo caches by feature set, and a clean rebuild is
what makes the warning count mean anything.
### The other axis: targets
The eight rows above vary features on one target. A platform gate needs a target
instead, and there are two rows because a library-only check misses half of them:
```bash
cargo zigbuild --lib --target aarch64-apple-darwin # 9
cargo zigbuild --tests --target aarch64-apple-darwin # 10
```
Row 9 reaches every `target_os` and `target_vendor` gate in the crate: the `fallocate`
punch in `local_disk_cache.rs`, the four `open(2)` flags `linux_only` supplies for hosts
that lack them, and the per-target `libc::stat` widths in `Inode::to_stat` together with
the `compile_error!` that refuses a target whose widths are not written down.
**One target, not one per feature row.** Before any of this was fixed, all eight feature
combinations produced the same 35 errors in the same four files — measured, not assumed.
The portability surface is entirely in code that compiles unconditionally, so crossing
the two axes buys nothing.
**Row 10 is the one that is easy to leave out, and leaving it out gives a wrong answer.**
At the baseline `--lib` had 35 errors and `--tests` had 80, because `src/file/mode.rs`
and `src/ondisk.rs` fail *only* inside their `#[cfg(test)]` modules. Check the library
alone and you conclude `mode.rs` needs no changes; it needs ten.
Compiling is all these two rows check. The link step wants the `Security` and
`CoreFoundation` frameworks from a macOS SDK and fails without one, so count compile
errors rather than trusting the exit status:
```bash
Running the suite is a macOS job, and so is anything about `mlock`: `src/buffer.rs`
panics when it fails, macOS has it with its own limits, and no cross build can say
whether those limits hold.
Setting it up on a Linux host:
```bash
rustup target add aarch64-apple-darwin
cargo install cargo-zigbuild # and zig 0.13 on PATH
export CFLAGS_aarch64_apple_darwin="-march=armv8.2-a+crypto+sha3"
```
The `CFLAGS` is for `aws-lc-sys`, whose `cpu_aarch64_apple.c` raises an `#error` without
it. A C cross-compiler is needed at all because `foyer` is an unconditional dependency
and pulls `zstd-sys` and `lz4-sys`, which is why a plain `cargo check --target` dies in a
build script having said nothing about this crate.
### `replay_fsx_log`
Replays an `fsx` operation log against hyperfile, checking every read
against an in-memory model. Optional: with no log to replay it prints a
message and passes.
```bash
# produce a log
fsx -N 1000 -S 1 -P /tmp -d <mountpoint>/fsx.1000 > ops.txt
# replay it
HYPERFILE_FSX_LOG=ops.txt cargo test --release --test replay_fsx_log \
-- --ignored --test-threads=1 --nocapture
```
Worth having because running fsx through a filesystem puts the kernel page
cache between it and hyperfile. A read the kernel answers itself never
arrives, so a read that returns stale bytes can go unnoticed until a later
read is served from the page cache — by which point the operation that
caused it is several steps back. That is how the read-ahead stale-install
bug hid: fsx flagged a read three operations after the one that went
wrong, and tests written against the flagged read all passed. Replaying
the same operations directly has no page cache in the way, so a bad read
is caught where it happens.
Reads are preceded by a read-ahead fired without waiting, the way a
filesystem layer doing its own read-ahead would. That is what leaves a
fetch in flight while later operations run, which is the shape that found
the bug.
### `integration_s3_smoke`
Happy-path end-to-end round trips. Create, write, flush, release,
reopen, read, truncate (extend and shrink). Runs in ~0.5 s.
### `integration_s3_rollback`
Two groups:
1. **Rollback exposure** (`rollback_exposure_*`): always-fail
interceptor, so even the rollback's abandonment cannot be written.
A flush that wrote no segment leaves storage unchanged. One whose
segment landed is left as a crash would leave it: the error says the
next open takes it up, storage is untouched until then, a reader
sees it without writing, and a writer publishes it.
2. **Rollback correctness** (`rollback_correctness_*`): fail-once
interceptor; verifies that a single transient flush failure is not
silently committed by subsequent retries or release-time flushes.
Covers every mutation API (write / truncate extend / truncate
shrink / write_zero / write_aligned_batch / write_batch).
Every size check reads the inode object, never a session: the session is
what the open path made of storage, and the open path is itself under
test. Checked across a read-only and a writable reopen, because a reopen
is exactly what used to publish a failed flush.
Runs in ~15 s, mostly two writable opens waiting out the window a fresh
commit's writer is given.
### `integration_s3_contract`
Uses `#[doc(hidden)]` getters on `Hyper` (dirty_block_count,
is_attr_dirty, is_bmap_dirty, cno tracking) to assert that flush
invariants hold after success, after rollback, after segment-upload
failure, and along the retry path (retries `ResourceBusy`, does not
retry `ErrorKind::Other`).
Runs in ~7 s.
### `integration_s3_concurrent`
Multi-instance concurrency:
- `concurrent_two_writers_optimistic_cc`: default
`RetryLastWriterWins` policy, two writers race on the same URI.
- `concurrent_read_while_writer_flushes`: reader observes
self-consistent checkpoints while writer churns.
- `concurrent_writer_a_fails_writer_b_succeeds`: writer A injected
failure, writer B succeeds.
- `concurrent_two_writers_fail_fast_policy`: `FlushConflictPolicy::
FailFast`; exactly one writer wins, the other returns
`ErrorKind::AlreadyExists`.
See `docs/concurrency.md` for the full behavior model.
Runs in ~27 s.
### `integration_s3_segment_open`
Covers `SegmentReadWrite::open`, which reads a segment's summary. It
speculatively fetches `SEGMENT_HEADER_FETCH_SIZE` (512 KiB) and tops up
only if `s_bytes` says the summary is larger, so its behavior depends on
how the object's real size compares to that constant. The suite writes a
single flush of 4 KiB, 8 KiB, 512 KiB, 1 MiB and 16 MiB and opens the
resulting segment in each case.
`open` has no caller inside hyperfile — reads go through the block map
and the block loaders, which issue exact-length range reads — so no other
suite exercises it. `hyperfile-cleaner` is the consumer.
**Run this one under debug assertions as well as release.** It previously
regressed on an integer underflow that wrapped harmlessly in release and
only panicked with `debug_assertions` on:
```bash
cargo test --test integration_s3_segment_open -- --ignored --test-threads=1
```
Runs in ~1 s.
### `integration_s3_block_api`
Covers `fs_block` / `fs_block_mut` / `fs_block_state`, which let a
caller borrow a cached data block instead of copying it through
`fs_read` / `fs_write`. Asserts the semantics a block-storage
consumer relies on: the borrow is the cache's own buffer, a hole is
distinguishable from a block of zeros (which `fs_read` cannot do),
an in-place modification is persisted by the next flush with no
write-back call, repeated borrows inside one flush window produce a
single checkpoint, `i_size` is untouched, and access mode is
enforced in both directions.
Runs the same round trip under all three data cache configurations,
because the borrow API does materially different work in each:
- default in-memory cache;
- local-disk cache, where clean blocks are views into a backing file,
`Cache::get` mlocks the block it hands out, and eviction punches a
hole;
- data cache disabled (`data_cache_blocks = 0`, which `O_DIRECT`
without `wal` forces), where there is nothing to borrow and
`fs_block` falls back to owning the block it loaded.
**Run this one under debug assertions as well as release.** Both
cache tiers carry `debug_assert!`s about block dirty state, and the
local-disk tier asserts outright that a clean block handed out by
`get` was not already locked — which is what makes the borrow
guards' `Drop` load-bearing.
Note the local-disk case creates a block in a hole *inside* `i_size`.
That tier addresses cached blocks as offsets into a mapping sized
from `i_size` and cannot grow the mapping in place, so a block above
EOF is not representable there; that case runs on the in-memory
cache instead.
Runs in ~2 s.
### `integration_s3_local_disk_cache`
The only suite that selects `HyperFileDataCacheConfig::LocalDisk` for
the **data** cache. That tier keeps blocks in a memory-mapped file;
because nothing exercised it, four separate defects lived there
undetected, including memory corruption on an extending write and a
process abort on release. See the suite's module docs.
Covers: creating an empty file on that tier, releasing it repeatedly
without aborting, writing far past EOF, a 4 TiB sparse address space,
slot recycling under constant eviction, truncate releasing slots
without leaving stale bytes, and exhausting the slot pool so blocks
fall back to heap allocations.
Runs in ~2 s.
### `integration_s3_read_timing`
Checks that the read-side counters count what they claim to. The read
path's cost is dominated by object-store round trips, and wall-clock
timing cannot tell one request for a coalesced range from many, nor a
cache hit from a fetch — so these are the assertions that keep the
counters trustworthy.
Covers: a read spanning 64 contiguous blocks in one segment costing a
single request; a byte read not populating the data cache, so a second
read of the same block fetches again; a block borrow populating it, so
a second borrow is a hit and a later byte read is too; index requests
counted separately from data requests, including that the batched
index fetch reports both of the requests it makes; holes and unflushed
writes costing nothing; and `read_timing_reset` zeroing every counter.
Runs in ~2 s.
`the_plan_agrees_with_a_direct_read` covers the placement queries on the
direct surface (`fs_read_plan`, `fs_block_placement` and their batch forms).
It lives here rather than with the reactor placement suite because it needs
the same counters: the queries are worth something only while they agree with
the read path, and the two surfaces reach it by different routes.
### `integration_s3_open_cno`
Opening a published checkpoint read-only, `Hyper::open_cno`. Each checkpoint
shows its own contents; a view does not move while the container advances;
write access is refused at open; an unpublished cno errors.
The load-bearing one is `a_checkpoint_view_cannot_write`. A checkpoint view
holds a historical inode, so anything of it reaching storage would publish the
past as the present — it asserts the write-class operations fail *and* that the
container is unchanged afterwards, release included. Writing this found that
`release` flushed unconditionally: closing a checkpoint view attempted to
publish, and only the on-disk state check stopped it, at the cost of three
retries and a `ResourceBusy` with no clean way to close.
### `integration_s3_taken_checkpoint`
The S3 half of `functional_recovery`: what memory staging cannot show.
Junk under each name a checkpoint answers to is stepped past with the
inode object's ETag unchanged and no wait. A commit left pending is read by
a reader without the ETag changing and published by a writer. And a
segment write answered with 412 because its own earlier attempt landed —
what a lost answer looks like when the write is tried again — is abandoned
under `FailFast` even though the error that ends the flush says nothing
about a segment: the rollback moves the counter past every number the
session used, rather than trusting the error to know. And the two-writer
order `integration_s3_concurrent` leaves to chance: B publishes, then A,
still on the view before, lands a checkpoint at the next number with its
abandonment refused. Its counters fit a successor; only the base it records
keeps it from being put over B's commit.
Runs in ~20 s.
### `integration_s3_multipart`
Segments larger than `segment_mpu_chunk_size` go up as multipart uploads,
and nothing else in the suites writes one, so this suite sets the part size
to S3's minimum and flushes 12 MiB at once. The baseline checks the object's
ETag says three parts, so a change that quietly stopped using multipart would
fail it rather than pass everything else vacuously.
The rest pin the multipart path to a single put's terms. A complete does not
replace an object already under its name -- something takes the name mid-upload,
the complete is refused, and the flush goes again at the next number. A failed
part aborts the upload, injected through an SDK interceptor that refuses one
part before it is sent. And with the name already holding the flush's own
segment -- a valid commit, which the next open would take up -- the refused
complete is reported as a taken name and, under `FailFast`, abandoned.
Two count the parts in flight, with an SDK interceptor that marks each part
upload's start and end: six parts under `segment_mpu_max_concurrency = 2` have
at most two in flight at once, and exactly two, so the bound is what limited
them; a bound of zero is taken as one. Without the bound every part starts at
once and the count is six.
Every test also checks no upload is left open under the container, and cleans
up any a revert experiment leaves.
Run it in both builds: the default one uploads through the scatter path, `wal`
through the contiguous one.
Runs in ~4 s.
### `integration_s3_large_checkpoint`
The real limit, where offsets used to wrap: 1100 MiB in one flush is two
objects, each within 1 GiB, and every MiB reads back from a cold handle. Before
checkpoints were split this flush succeeded and the data past the first
gigabyte read back wrong; with the split removed the test finds one 1.16 GB
object. A second test under a 40 MiB limit writes 150 MiB as four objects, each
large enough to be a multipart upload of its own.
Needs about 1.2 GB of memory for the written data. Runs in ~15 s.
### `integration_s3_node_reload`
Index nodes evicted and read back from S3 while other nodes of the same stored
segment are changed in memory. S3 staging's default loader returns a segment's
other nodes with each one asked for, and btree-ondisk 0.20.0 installed them over
resident ones, changed or not: the next flush failed assigning index pointers
with "key not found through btree node lookup", and every retry with it. Built as
a downstream container builds its file — `S3Staging`, its loader, `NullNodeCache`
— with a node cache of 16 so every flush evicts; 128 MiB of writes, then eight
flushes of overwrites, small writes and zero writes across 16 GiB, and the data
read back cold. Fails on btree-ondisk 0.20.0, passes on 0.20.1.
Runs in ~4 s.
### `integration_s3_write_failure`
A write whose index lookup fails part-way, as when the network drops. Every
write path inserts into the block map for each block it touches, and an insert
may read an index node back from S3; that used to panic the process. The file
holds two regions flushed separately, so the index for one can be in memory
and the other's still on S3; the test reads the first, refuses reads of the
second's segment with an SDK interceptor, and writes a range across both. For
`write`, `write_zero`, `write_batch`, `write_aligned_batch`, `write_ref` and a
reactor `fh_write`: the write fails, the file reads and stats as before, the
same write repeated succeeds, and what is stored then matches a twin that never
saw the failure. On 0.9.1 the direct paths panic, `write_batch` crashes on a
use after free, and the reactor write never returns.
Runs in ~12 s.
### `integration_s3_compaction`
`fs_relocate` on S3: 2048 blocks in one object are read in a few requests,
not one per block; `fs_content_cno` and `fs_content_cno_fast` agree and stay
where they were across the relocation, and move on with a write; the data
reads back the same from a cold handle.
Runs in ~1 s.
### `integration_s3_error_kinds`
The error kinds a caller decides retries by (see `s3_error_kind`). A data
GetObject whose body an SDK interceptor cuts off after the headers is
`ConnectionReset`, not `Other`, and the read succeeds once it is not cut; a
reactor write whose block fetch the store answers 503 reaches the caller as
`ResourceBusy` within 30 s — with the handler reading the kind as a requeue,
as it did, the write is never answered; and writes racing flushes, which the
handler does requeue, are all answered.
Runs in ~1 s.
### `integration_s3_partial_delete`
A DeleteObjects that succeeds without deleting every key: an SDK
interceptor takes the refused keys out of the request and lists them as
errors in the 200 response, as S3 does. `unlink` returns the first
refusal's kind (`AccessDenied` is `PermissionDenied`, `SlowDown` is
`ResourceBusy`) and names the code, instead of reporting success with
objects left; an unlink with nothing refused still succeeds.
Runs in ~1 s.
### `public_error_kind`
`s3_error_kind`, `s3_code_kind`, `is_condition_failure` and
`is_retryable` called from outside the crate, on SDK errors built in the
test: each per-key code has the kind of its status, a refused condition
is told by its code at status 200, and the retry column of the table on
`s3_error_kind`. Needs no bucket; runs with the ordinary `cargo test`.
### Crate-internal: `wal::abort_tests` (`--features wal`)
An `abort_txn` whose log could not be deleted. In the crate because the
faults go into the log's own S3 client, which nothing outside can reach;
`#[ignore]`d and run against S3 like the suites above:
```bash
cargo test --release --features wal --lib wal::abort_tests -- --ignored --test-threads=1
```
A 503 on the delete fails the abort, leaves the transaction open, and a
second abort succeeds; records refused part-way never come back on a
later open, in either recovery mode, even if the abort is not asked
again.
### `integration_reactor_s3_smoke`
Reactor-mode smoke (`HyperFileHandler` and `HyperFileTokio`). Same
shape as `integration_s3_smoke` but each test uses a `LocalSpawner`
and goes through the request/response channel. Covers read, write,
truncate extend/shrink, flush, getattr, setattr, as well as the
tokio `AsyncRead`/`AsyncWrite`/`AsyncSeek` surface.
`reactor_seek_hole_and_data_match_the_direct_api` asserts the handler's
`SEEK_HOLE`/`SEEK_DATA` against spelled-out expected values *and* against the
direct API on the same file, so neither a wrong answer nor two surfaces wrong
together passes. `reactor_seek_sees_unflushed_writes` covers the case the
direct API once regressed on, a dirty block reported as a hole. Swapping the
two whences in the dispatch arm fails both.
Feature requirement: `reactor` (default).
Runs in ~1 s.
### `integration_reactor_s3_placement`
The read-only placement queries, `fh_read_plan` and `fh_block_placement` with
their batch forms — see [docs/placement.md](../docs/placement.md).
`plan_agrees_with_what_the_read_does` carries the weight: the plan is worth
exposing only because it *is* the planner, so its predicted request count is
asserted against what a real read issues. The layouts are built to make the
two easy to disagree about — alternating flushes so consecutive blocks land in
different segments, and a run longer than `read_get_max_bytes`, which is the
only way two consecutive requests share a segment.
Verified by injecting three ways a hand-rolled cost model would go wrong —
merging across segments, merging across the request-size cap, and reporting a
dirty block as placed. Each fails the suite; the cap case fails only because
that test exists.
Runs in ~2 s, one 20 MiB write included.
### `integration_reactor_s3_block_api`
Closure-scoped block access on the reactor surface,
`fh_with_block` / `fh_with_block_mut`. The reactor cannot hand out
borrow guards — see [docs/block-api.md](../docs/block-api.md#why-the-reactor-surface-takes-a-closure)
— so the caller's action is sent to the block instead.
Covers values moving in and out of the closure, the closure not
running for a hole, in-place modification persisting without a
write-back call, `create`, repeated edits collapsing to one
checkpoint, access-mode errors surfacing through the channel without
killing the reactor task, and a byte-for-byte equivalence check
between the closure form and the direct guard form.
Also covers the reactor counter accessors (`fh_read_timing`,
`fh_flush_timing` and their resets), including that the cache rule
documented for the byte and block APIs holds on this surface too —
which was untestable before those accessors existed.
Runs in ~1 s.
### `integration_reactor_s3_contention`
Concurrent use of one handle must not stall. Every regression this suite
covers was a hang rather than a wrong answer, so each test carries a
watchdog that reports which operation was outstanding when progress
stopped, and asserts before joining the workload.
Covers the reported shape (concurrent `fh_read_owned` while another task
writes and flushes), the same with `fh_truncate`, several writers plus
readers with the resulting data checked block by block, every
permit-taking operation driven at once, and colliding unaligned writes
to one cold block checking that the bytes neither writer touched keep
their staged contents.
Two details matter when extending it. A **block-aligned** write needs no
retrieve, so it never holds the per-file permit across a callback hop and
exercises none of this — the writes here are deliberately unaligned and
walk cold blocks. And concurrent readers can mask a stall in another
operation, because a read put back on the high-priority queue outranks
one queued behind it, so the narrower cases run without readers.
Run it under `range-lock` as well as default features: the mechanisms
differ, and `concurrent_reads_do_not_stall_writes` covers the flush drain
that only exists there. Worth running in debug too — one of the races it
catches showed up only in a debug build.
```bash
cargo test --test integration_reactor_s3_contention -- --ignored --test-threads=1
cargo test --features range-lock --test integration_reactor_s3_contention \
-- --ignored --test-threads=1
```
Runs in ~20 s. On failure the process may hang after reporting, because
a stalled reactor thread cannot be shut down; run it under a timeout.
### `integration_reactor_s3_range_lock`
Multi-task concurrency on a **single** file handle (via
`HyperFileHandler::clone`). Validates that under `range-lock` the
reactor serializes overlapping byte ranges and lets disjoint ranges
proceed in parallel, and that flush waits for in-flight writes.
Feature requirement: `reactor` + `range-lock`.
Run with:
```bash
cargo test --features range-lock --test integration_reactor_s3_range_lock \
-- --ignored --test-threads=1
```
### `integration_reactor_s3_wal`
Smokes the WAL path: create + write + flush + reopen round trip with
`HyperFileWalConfig` pointing at a sibling `<uri>/wal/` prefix. Also
verifies that reopen after a second flush does not observe stale WAL
state. Among the transaction cases: a transaction left open right
after a publish, with nothing before it in its group, does not stop the
file from opening — it panicked every open until 0.10.2.
Feature requirement: `reactor` + `wal`.
**Note**: tests use `#[tokio::test(flavor = "multi_thread")]` because
`S3Wal::from_uri` calls `tokio::task::block_in_place`, which is not
valid on a current-thread runtime.
Run with:
```bash
cargo test --features wal --test integration_reactor_s3_wal \
-- --ignored --test-threads=1
```
### `integration_reactor_s3_all_features`
Maximum-feature smoke: reactor + wal + range-lock +
concurrent-segment-build enabled together. Covers the full feature
matrix in a single binary.
Feature requirement: `reactor` + `wal` + `range-lock` +
`concurrent-segment-build`.
Run with:
```bash
cargo test --features "wal range-lock concurrent-segment-build" \
--test integration_reactor_s3_all_features \
-- --ignored --test-threads=1
```
## Feature matrix
The reactor suites exercise different feature combinations. When
iterating on a feature flag, run only the relevant suite:
| `reactor` only (default) | `integration_reactor_s3_smoke` |
| `reactor` + `range-lock` | `integration_reactor_s3_range_lock` |
| `reactor` + `wal` | `integration_reactor_s3_wal` |
| `reactor` + `wal` + `range-lock` + `concurrent-segment-build` | `integration_reactor_s3_all_features` |
To validate a release candidate, run every combination once:
```bash
# default
cargo test --test integration_reactor_s3_smoke -- --ignored --test-threads=1
# + range-lock
cargo test --features range-lock --test integration_reactor_s3_range_lock \
-- --ignored --test-threads=1
# + wal
cargo test --features wal --test integration_reactor_s3_wal \
-- --ignored --test-threads=1
# + all features
cargo test --features "wal range-lock concurrent-segment-build" \
--test integration_reactor_s3_all_features \
-- --ignored --test-threads=1
```
Plus all the direct-API suites (`integration_s3_*`).
## Adding a new interceptor
Shared interceptors live in `tests/common/mod.rs`. To inject a fault
not yet covered, implement `StagingIntercept<S3Staging>` with overrides
only for the hook you need. All methods have default no-op
implementations, so a minimal interceptor is a few lines:
```rust
#[derive(Clone)]
pub struct FailReadInode {
calls: Arc<AtomicUsize>,
}
impl StagingIntercept<S3Staging> for FailReadInode {
fn after_flush_inode(&self, _: &S3Staging, _: &[u8], _: FlushInodeFlag)
-> Pin<Box<dyn Future<Output = Result<()>> + '_ + Send>>
{ Box::pin(async { Ok(()) }) }
fn after_remove_inode(&self, _: &S3Staging)
-> Pin<Box<dyn Future<Output = Result<()>> + '_ + Send>>
{ Box::pin(async { Ok(()) }) }
// override the specific hook you want
}
```
Available hook points (see `src/staging/mod.rs`):
- `before_flush_inode` — can return `Err` to make the PUT fail before
it hits S3.
- `after_flush_inode` — post-success hook, can't cause the flush to
fail but can observe or trigger side effects.
- `after_remove_inode`
- `before_segment_done` — can return `Err` to make a segment upload
fail.
Put the new interceptor in `tests/common/mod.rs` so every suite can
use it. Install in a test with `Hyper::with_staging_interceptor(i.clone())`
after opening.
## Adding a new integration test
1. Pick the right suite or add a new `integration_s3_<name>.rs` file
(and update this README's Layout table).
2. Top of file: `mod common; use common::*;` plus whatever hyperfile
imports you need.
3. Use `#[tokio::test]` + `#[ignore]` so the test is excluded from the
default `cargo test` run.
4. Start with a `TestFile::new(&client).await` and call
`tf.cleanup(&client).await` at the very end (do NOT rely on Drop —
see the comment on `TestFile` for why).
5. For tests that need a specific runtime config (e.g. a
`FlushConflictPolicy`), construct a `HyperFileRuntimeConfig` and
open via `Hyper::fs_open_opt`.
6. Always call `let _ = hyper.fs_release().await` before dropping the
`Hyper`, even on the error path — `release` runs internal flush
cleanup.
7. If the test is concurrent, use `tokio::join!` over plain futures,
not `tokio::spawn` — `Hyper` is `Send` but not `Sync`, so its
futures can't be scheduled across tokio worker threads.
8. Prefer printing diagnostic info with `println!` and `--nocapture`
over stuffing detail into assertion messages.
## Adding a new suite
If a new category of tests doesn't fit into the existing four,
create `tests/integration_s3_<category>.rs`:
1. Start with the same header pattern as existing suites (module-level
doc comment explaining the group, `mod common; use common::*;`,
imports).
2. Add a row to the "Layout" table and a section under "Integration
test suites" describing what it covers.
3. A new test binary is automatically picked up by `cargo test --tests`;
no Cargo.toml changes required.
## Cleanup and orphaned objects
`TestFile::cleanup` uses `Hyper::fs_unlink`, which issues a prefix list
plus batch delete. If a test crashes between `TestFile::new` and
`cleanup`, the created objects remain in the bucket. They are harmless
(each run uses a unique ULID-based prefix) but can accumulate. Nuke
them with:
```bash
aws s3 rm s3://$HYPERFILE_TEST_BUCKET/hyperfile-test/ --recursive \
--region $HYPERFILE_TEST_REGION
```
## Related reading
- [`docs/concurrency.md`](../docs/concurrency.md) — how
`FlushConflictPolicy` and S3 OCC interact; reference for the
concurrency suite.
- [`docs/flush.md`](../docs/flush.md) — what a flush does and what
the front end may do while it happens; reference for the
contention suite and the reads and writes that overlap a flush.
- [`docs/posix.md`](../docs/posix.md) — POSIX semantics, sync mode
behavior.