# Write-Ahead Log (WAL)
This doc describes Hyperfile's optional write-ahead log and what
durability / consistency guarantees it offers. It complements the
`range-lock` / conflict-policy discussion in `concurrency.md` by
covering the crash path.
WAL is gated behind the `wal` Cargo feature and configured via
`HyperFileWalConfig`.
## Why WAL
Without WAL, a write goes to an in-memory cache. It is **not**
durable until a flush successfully persists the inode to S3. If
the process crashes between the write and the flush, the write is
lost.
With WAL enabled, every write is persisted to a separate S3 prefix
before its ack. On crash-and-reopen, the recovery path replays
those entries into the file. The caller observes a simple
"`fh_write` returned `Ok` = durable" contract.
## Durability contract
**A write that has returned `Ok` is recoverable with no further
flush.** Its bytes are in the log, and opening the container again
replays them.
Spelled out, because "durable" is otherwise read four different
ways:
- **Scope** — the bytes each `Ok` return of `write` or `write_zero`
reports as written. Nothing is promised about a write that
returned an error, and nothing about ordering between separate
writes: without a flush between them a caller should not assume
one landed before another.
- **When** — from the moment the call returns. `fh_write` /
`fh_write_zero` / `HyperFileTokio::write` return `Ok` only after
the log object PUT has completed. A crash *before* `Ok` may or
may not have landed, like any pending `Result`.
- **Independent of publishing** — it holds regardless of
`commit_bytes` and `commit_interval_ms`, and does not require any
flush or segment publish to have happened. Callers may therefore
skip flushing on this basis.
- **Recovery** — reopening the same container with the same log
configuration replays it, with no manual step.
It says nothing about a checkpoint existing for those bytes.
Reading them back needs the container reopened, not a checkpoint
published, so this guarantee does not make a cno openable.
This is stronger than the no-WAL baseline, where "`fh_write`
returned `Ok`" only means the data is in the in-memory cache and a
crash erases it.
### Ask for it, do not infer it
```rust
if hyper.writes_durable_on_ack() { /* may skip flushing */ }
```
A caller that skips flushing on this basis should read the
guarantee from `Hyper::writes_durable_on_ack` rather than deciding
for itself whether a log is configured. Inferring it — checking
that the WAL URI looks like `s3://`, say — copies a judgement this
crate owns, and the copy can only diverge in the unsafe direction:
add any path where a log is configured but the guarantee does not
hold, and the caller skips a flush it needed and says nothing about
it.
The name is deliberate. It reports the guarantee, not the
mechanism, so if this crate ever keeps a log while not offering the
property — batching log writes so a write can return before its
bytes are down — it must start answering `false`, and callers
degrade rather than lose data quietly.
`reactor_wal_writes_are_durable_on_ack_without_any_flush` pins all
of this: publish thresholds set out of reach so nothing incidental
can produce a segment, a write, a simulated process death with no
flush, and a byte comparison after reopening — alongside a no-log
control that must lose the write, since without it the first half
would also pass if writes were simply never lost.
## On-disk layout
```text
<wal_root>/
0000000000/
0_0_4096 # seq 0, offset 0, length 4096
1_4096_4096 # seq 1, offset 4096, length 4096
2_8192_512 # seq 2, offset 8192, length 512
0000000001/
0_0_4096
1_16384_4096
```
- **Top-level directory**: zero-padded segment id (the inode's
`last_seq` at the time of the WAL write, **not** the next flush's
segid — see "Indexing" below).
- **Object name**: `<seq>_<offset>_<len>`. `seq` is monotonic per
segid, reset to zero whenever the segid changes.
- **Body**: the raw bytes of the write. A zero-length object
represents `fh_write_zero` (hole punch).
`seq` is tracked in memory (`S3Wal::seq`, an `AtomicU64`). It is
not recovered from S3 — the invariant is that the writer owning
the S3Wal is the only process writing to this prefix, so the
in-memory counter is authoritative for the lifetime of that
handle.
### Indexing nuance
WAL chunks for flush N are indexed under **segid N - 1** (the
inode's `last_seq` at the time of the write). On flush, the code
calls `get_next_seq()` which bumps `last_seq` to allocate the new
segment's segid = N; the WAL chunks that produced segment N
remain under segid N - 1. This is the reason
`HyperFile::wal_flush_done` deletes `segid.saturating_sub(1)`.
If you grep for `delete_segment`, this is the subtle detail to
keep in mind.
## Barriers: which log records belong together
A flush seals the records it is about to publish by writing a **barrier** for
that checkpoint. The barrier carries the `(seq, offset, len)` of every record it
covers.
It exists to answer a question the records cannot. A crash mid-flush leaves a
partial set that is indistinguishable from a complete one by looking at the
objects, and replaying it hands the layer above half of its own transaction.
The manifest is a list rather than a count so that recovery can check it against
what is stored without depending on a listing being complete or on the order the
records went out in.
Two orderings make it mean something:
- **After** every record it covers is confirmed written, because it claims the
group is whole.
- **Before** the segment is built, which is also before `get_next_seq()` moves
past the checkpoint the records are filed under.
Together those give: *a barrier exists for this checkpoint* ⟹ *the group is
complete and was never published*.
Sealing nothing writes nothing. That is load-bearing rather than tidiness — an
empty barrier would sit in front of records written afterwards under the same
checkpoint, and recovery would drop them as out-of-manifest.
The barrier object is named `barrier`, not `<seq>_<offset>_<len>`, so the record
decoder rejects it and `list_chunks` cannot mistake it for data. Its body
carries a magic and a version, and a body this build cannot parse reads as
absent — which stops recovery rather than letting it vouch for a group it cannot
check.
## Recovery modes
Where recovery stops is a choice, set by
`HyperFileWalConfig::with_recovery_mode`:
| `Latest` (default) | whatever it can | may be mid-transaction | `true` |
| `Barrier` | only sealed, complete groups | a declared state | **`false`** |
`Latest` keeps every acknowledged write, which is what the durability contract
promises, so it is the default. `Barrier` stops at the last seal, so it needs no
repair — at the cost of everything acknowledged after that seal.
Those two cannot both hold, and `Barrier` is the case the durability query was
named for: a write with no flush behind it is in an unsealed group, so that mode
discards it by design, and `writes_durable_on_ack` reports `false` there. A
caller skipping flushes on the strength of it then degrades instead of losing
data quietly. Making `Barrier` the default would have withdrawn a shipped
guarantee silently on upgrade.
Under `Barrier`, a group that is unsealed or incomplete **stops** the replay
rather than being skipped. Skipping one and applying a later one would produce a
state that never existed, which is worse than a torn one: that at least was some
moment's truth.
## What recovery reports
```rust
let r = hyper.wal_recovery_report();
if !r.landed_on_barrier { /* repair */ }
```
`replayed`, `landed_on_barrier`, and `records_dropped`. The middle one decides
whether a caller that can repair its own contents needs to: when it is true the
contents are a state whoever wrote them declared consistent, so work
proportional to the whole container can be skipped. Records counted as dropped
are still in the log; nothing is deleted.
"No recovery ran" reports `landed_on_barrier: true`, because a container that
needed no replay is at its last published checkpoint and a publish only happens
at a seal.
## What may publish
With a log configured, **time alone does not publish**. A timer's purpose
without a log is to bound how long an acknowledged write sits only in memory,
and a log already bounds that. All it would add is a checkpoint at a moment
nobody declared — and since recovery has to reach the newest published
checkpoint, one landing mid-transaction puts "reach the newest checkpoint" and
"stop at a declared state" in conflict, which `Barrier` cannot satisfy.
The memory bound stays. `data_cache_dirty_max_bytes_threshold` and
`..._blocks_threshold` still publish when they are crossed, because dropping
them would trade a bounded cache for an unbounded one. A caller that needs
publishes only at its own boundaries raises those thresholds and knows it is
choosing that.
`data_cache_dirty_max_flush_interval` set to `0` does **not** disable the timer.
`elapsed() >= Duration::from_millis(0)` is always true, so `0` publishes on
every check. Without a log, raising it is how to slow the timer down.
## Recovery
`HyperFile::do_open` checks the WAL on every open:
```text
let wal_max_segid = wal.list_segments().iter().max();
if wal_max_segid >= inode.last_seq {
trigger wal_flush_recovery
}
```
`wal_flush_recovery` walks all segids `>= inode.last_ondisk_cno`
(anything whose data has not yet been persisted in the inode),
calls `wal_replay_chunks` for each in ascending order. Replay
reads each chunk's bytes from the WAL, calls the equivalent of
`fs_write` / `fs_write_zero` on the reopening `HyperFile`, then
does a force flush to produce a new segment. After the force flush
succeeds, the replayed WAL segment is scheduled for deletion (as
a fire-and-forget `tokio::spawn`).
That flush runs with the log detached. An ordinary publish tidies the
log by position — it deletes the prefix one below the checkpoint it
wrote — and a replay's checkpoint number is not tied to the prefix it
replayed: open moves the counter past numbers already taken on storage
(see below), so prefix 0 can land as checkpoint 2, and deleting "one
below" would remove prefix 1 before its turn. The replay deletes exactly the prefix it
consumed, and nothing else.
Recovery is idempotent: if it runs and is interrupted partway
through, the next open starts from scratch and replays the same
set of segids again. The replay is deterministic because `seq`
gives a total order of writes within a segid.
A segment object is written create-only, so a replay whose segment
name is already taken fails the conditional put with 412, surfaced as
`AlreadyExists`. That is treated as progress rather than failure: the
name is taken, and its log entries are deleted as they would be after a
replay that did the work — otherwise every open lists them again and
repeats the skip. It is logged each time.
It is no longer the ordinary case. The ordinary one was a segment that
landed and an inode that did not, and open deals with that before any
replay, read-only opens included (see `roll_forward`): a segment that is
a whole commit is taken up — every segment carries its own inode — and
anything else under a checkpoint number, such as partials with no part
0, is stepped past without being read further. Either way the counter
ends past every taken number, so a replay gets a free name, and what it
replays is only what no taken-up commit already holds, since recovery
filters by the checkpoint the inode now points at. What still reaches
the 412 is a name taken between that walk and the replay's own write —
another writer, which this single-writer design does not support — and
there the inference below is wrong, not merely unverified: the stored
object is someone else's, and deleting the log loses what it held. See
the TODO below.
Recovery is what a failed publish falls back to, and it can fail
for the same reason the publish did — an object store refusing
writes. It retries a bounded number of times with backoff. When
those are spent, the file stops accepting modification and goes on
serving reads: every acknowledged write is in the log, so nothing
is lost and offline repair has what it needs, whereas continuing
to accept writes would pile more data behind a publish that is not
happening. Reads through that handle may not show the newest data,
because the flush had already repointed the map at the segment it
could not publish; durability is unaffected, and reopening replays
the log.
This case previously panicked. For a server built on this crate
that means the whole process, including the reads it was still
serving correctly.
## Flushing without stopping the front end
Because the WAL makes a flush's completion a given, the flush does not
have to block the front end: the segment stays pinned in memory while it
is written out, and reads and writes are served from it rather than
waiting. That is the main thing the WAL buys beyond durability, and it is
described in [flush.md](flush.md#with-wal-the-segment-is-pinned-and-the-front-end-carries-on)
together with what a write has to do differently and what it costs a
concurrent reader.
### Which flush paths do this
Worth stating plainly, because the two surfaces differ and the
difference is easy to get backwards:
- **Reactor.** `fh_flush` / `fh_fdatasync` reach a WAL-specific
dispatch arm that calls `kick_wal_protected_flush_reactor`, which
hands the upload to a spawned task and returns. Both the threshold
flush and the explicit one take this path.
- **Direct API.** `fs_flush` goes through `HyperTrait::flush` to
`flush_process`, which has no WAL branch: it waits for the segment
and then for the inode.
So `flush_process` being synchronous says nothing about what a
reactor-mode caller experiences.
What the asynchronous publish does *not* change is how many objects a
run produces. Every flush still builds a segment, whether or not the
caller waits for it. Measured over 50 writes each followed by an
explicit flush, 64 KiB apiece:
| no wal | 51 | 752 ms |
| with wal | 51 | 783 ms |
51 is 50 segments plus the inode. The time barely moves either,
because each flush takes the flush lock and the spawned upload holds
it until it finishes — so the next flush waits for the previous
upload regardless of having returned early. That serialization is
deliberate; see the TODO.
## Cleanup (WAL object delete)
After a successful flush (in the reactor WAL path,
`wal_flush_done`), the WAL chunks for the flushed segid are no
longer needed and are deleted from S3 asynchronously via
`tokio::spawn`. Three properties follow:
- **Non-blocking**: the caller's `fh_flush` ack does not wait for
the delete to complete.
- **Tolerant**: a dropped or failed delete does not affect
correctness. Recovery filters by `last_ondisk_cno`, so stale
WAL chunks are skipped rather than replayed twice.
- **Best-effort**: if a lot of flushes happen and one spawned
delete task lags, storage usage temporarily grows. It is
eventually consistent with "one WAL prefix per unflushed
segment".
## Create-time invariant
`Hyper::create` with `wal` enabled returns
`ErrorKind::AlreadyExists` if the WAL prefix is not empty. The
rationale: creating a fresh file while the WAL holds stale entries
would cause `open` to replay old data into the new file, which is
almost certainly not what the caller wants.
Either unlink or explicitly empty the WAL prefix first. Typical
application code uses `Hyper::fs_unlink(uri)` which removes the
whole `<uri>/...` tree including WAL.
## Performance impact
- **Write**: extra S3 PUT per write. On S3 Express One Zone this
adds roughly 5-10 ms per write depending on payload size.
Large writes may be more costly (the WAL PUT is one S3 object
per write, regardless of payload).
- **Flush**: no direct overhead. The delete that happens after
flush runs off the critical path, and the segment upload does
too — reads and writes carry on against the pinned segment while
it happens, see [flush.md](flush.md).
- **Open**: adds a `list_segments` call. On a recently-flushed
file the list is empty. On crash recovery, the open pays the
replay cost proportional to the number of unflushed writes.
If your workload is dominated by many small writes with rare
flushes, WAL amortizes the flush overhead in exchange for the
per-write PUT. If your workload is dominated by large bulk writes
with synchronous flushes, the extra PUT may be pure cost —
measure.
## Current limitations
- **Single-writer**: Two `Hyper` instances open on the same URI
with WAL both write into the same `<wal_root>` prefix and
allocate `seq` independently. Key collisions are possible. Use
WAL from a single writer at a time.
## TODO
- **Verify an already-published checkpoint instead of assuming it.**
Recovery currently treats a 412 on a segment put as "this
checkpoint is already stored" and moves on. That is an inference
from the key, not a check of the contents, and since open steps
past taken numbers before replaying, the only 412 left is one that
is *not* this replay's work — so the safer reading of it now is
"the name was taken under us", which calls for stepping past it and
replaying again rather than deleting the log. The stricter form is to compare the candidate
segment against the stored one and to treat a difference as a
consistency violation to be fixed with offline tools rather than
worked around at runtime.
Two things stand in the way, both worth recording because they
are why this is not a small change:
- `s_chksum` in the segment header **is always zero**. The field
exists and `realize_ss` takes a checksum argument, but both
production call sites pass `0` and the code says as much
(`don't actually need checksum now`). So every segment written
to date carries no checksum, and a newly computed one would
never match an old segment — the comparison would report a
mismatch on the most ordinary reopen. Whatever is adopted needs
an answer for segments that cannot be verified at all.
- **Comparing the bytes will not do.** The segment header embeds
`s_inode`, and a replay updates mtime and ctime, so a replay of
identical data produces different bytes. A meaningful checksum
has to name which bytes it covers — the data and metadata
blocks, not the header — which is likely why the field was left
unused.
The comparison also has to happen where the candidate bytes still
exist, which is the staging put path rather than the recovery
loop; by the time recovery sees `AlreadyExists` the bytes are
gone. Doing it there would extend the check to every segment
put, not only replays.
- **Satisfy a flush from the log, and replay to storage in the
background.** The remaining cost of a flush is not the waiting —
the reactor path already returns before the upload — it is that
every flush builds and stores a segment at all. A consumer
measured 200 objects for a hundred fsync'd files either way, and
1 object when the flush was suppressed entirely at their layer:
4059 ms without the log, 3333 ms with it, 1904 ms suppressed. The
gap between the second and third rows is the segment per flush,
not the wait.
In principle the log makes it safe to satisfy a flush without
producing a segment: the data is durable the moment the log holds
it, and recovery replays it. The question is what then bounds the
log, since a fsync-heavy workload would otherwise grow it without
limit and recovery time with it.
The commit thresholds turn out to be the answer, and a better
lever than publishing on every flush. The same consumer measured,
on the same workload:
| thresholds | container objects | log peak | after clean stop |
|---|---|---|---|
| 256 MiB / 1 h (deliberately unreachable) | 3 | 869 | 883 |
| 16 MiB / 2 s | 10 | 86 | 5 |
Tight thresholds give few objects *and* a bounded log, which is
what publishing per flush was wanted for — at 200 objects. So a
background replay is an optimization after all rather than a
prerequisite; what it would add is bounding the log by time since
the last publish without a publish having to be triggered by
traffic.
It would need the serialization reconsidered either way. A flush
takes the flush lock and the spawned upload holds it to
completion, so flushes complete one at a time — which is what
keeps `is_flushing` meaningful across the upload window, and what
a background replay would have to work with rather than around.
Until then, a caller who wants this suppresses the flush itself
when `writes_durable_on_ack` reports true, which is sound for the
same reason and is what the consumer above does, with the two
thresholds set tight enough to bound the log.
## Related tests
- `reactor_wal_smoke_write_flush_reopen` — happy path round trip.
- `reactor_wal_flush_and_reopen_is_idempotent` — reopen after
flush does not pick up stale WAL.
- `reactor_wal_delete_after_flush` — WAL objects are actually
removed from S3 after a reactor flush.
- `reactor_wal_flush_seals_the_group_it_publishes` — the barrier appears only
when a flush says so, and its manifest names exactly the records stored.
- `reactor_wal_recovery_mode_decides_the_landing_point` — the two modes land in
different places, on a container each, and the durability query reports which.
- `reactor_wal_open_reports_what_recovery_did` — the three outcomes a caller has
to tell apart.
- `reactor_wal_time_alone_does_not_publish` — with a log, idling does not
publish; without one it still does.
- `reactor_wal_batch_write_reaches_the_log` and
`reactor_wal_batch_write_is_durable_on_ack` — the batch write paths log, and
their records are coalesced across adjacent blocks.
- `reactor_wal_repeated_crash_without_publish_keeps_the_container_sound` — crash,
recover, crash again with no publish in between.
- `reactor_wal_writes_are_durable_on_ack_without_any_flush` — a
write survives a simulated death with no flush of any kind, with
publish thresholds out of reach and a no-log control that loses it.
- `reactor_wal_publish_failure_turns_read_only_without_panicking` —
a publish that keeps failing exhausts recovery's retries, after
which writes are refused and reads are still served; reopening
recovers the acknowledged write from the log.
- `direct_api_wal_delete_after_flush` — same, but via
`Hyper::fs_*` direct API.
- `reactor_wal_crash_recovery_replays_unflushed_write` — drop
handler without flush, reopen, verify the write is restored.
- `reactor_wal_crash_recovery_multiple_disjoint_writes` — two
sequential writes survive crash.
- `reactor_wal_crash_recovery_multiple_handles` — multiple
handler clones with concurrent writes via `tokio::join!`,
survive crash.
- Tests covering reads and writes during a flush are listed in
[flush.md](flush.md#related-tests).
- S3Wal unit tests in `src/wal/s3.rs` cover `next_seq`,
`reset_seq`, `encode`, and `decode`.