Module worker 

The backup task: ships WAL segments, snapshots, archives and side files, and writes the manifests that name consistent restore points.

What it reads, and what it never touches

It reads the store’s files through ordinary OS reads on Tokio’s blocking pool, and one value from the engine: WalSource::durable_wal_position, which says how much of wal.bin is fsynced, in which generation, and which snapshot.bin and wal.bin files the engine’s state is on. Reading it takes no engine lock, so it cannot slow a write. The one exception: when the files on disk show a snapshot or commits the engine has not seen (another process wrote them), the worker asks the handle to catch up — the same metadata check and refresh every read makes. It writes one file into the store directory, once: backup.lineage, on the first backup of a store (see Lineage).

Consistency

A restore point is a snapshot plus the WAL that starts beside it, and which ones those are comes from the files on disk:

• Nothing is read while snapshot.intent holds anything — a snapshot, in this process or another, is between writing snapshot.bin and replacing wal.bin. The step is deferred, not failed.
• Every read is bracketed by two looks at the directory, and kept only if neither the snapshot, the WAL file, nor the intent changed and the WAL did not shrink between them (DiskState::unchanged_until).
• A restore point is built only when the engine’s position names the snapshot and WAL file on disk; a new generation starts whenever either changes. A snapshot that kept its WAL carries a mark of the prefix it holds, and the open of a restored store skips that prefix by itself, so shipping the whole WAL from offset 0 beside it is always correct.
• A manifest’s commit_seq is computed from the objects it lists — the wal.floor it carries plus the frames of each archive and segment — so it is what the restored store reports, by construction.

Objects and ordering

Every object a manifest names is immutable (see crate::manifest); a manifest is written only after every object it names exists, and LATEST only after its manifest. Retention deletes only what no retained manifest names. So at any point — a crash between any two requests included — every manifest in the bucket verifies, and LATEST names one of them.

Lineage

A prefix holds the backups of one store lineage. The first backup of a store gives it a random lineage id (backup.lineage, a side file, so a store restored from the prefix carries it on). Every manifest and every object entry names its lineage, and resume checks it before using anything in the prefix. The worker refuses the prefix — uploads nothing, deletes nothing, reports unhealthy with the reason — when its newest restore point is another lineage’s, when this store has no lineage but the prefix has restore points, or when this store holds fewer commits than that restore point (it was restored from an older backup, or rewound). Archives found in the prefix are reused only when their number, length and CRC all match the local file. Retention reads only this lineage’s manifests and deletes only objects of this lineage’s generations, never the manifest it just wrote or the one LATEST names, and never an object another lineage’s manifest names.

One exception lets a store restored on boot (crate::boot) back up to the prefix it came from. It has a lineage of its own and a parent record (disk::PARENT_FILE) naming the lineage and manifest it was restored from. While LATEST still names exactly that manifest, the worker adopts the prefix: it writes its own lineage’s objects beside the parent’s, under generation numbers past every one in the prefix, moves LATEST to its own manifests, and never overwrites or deletes anything of the parent’s. A manifest key the parent already holds (the same commit count) is not written; the next restore point, one commit later, is. If LATEST has moved on since the restore, another instance wrote to the prefix: refused like any other lineage.

Writes the bucket can refuse

Every immutable object (segment, side file) is written with If-None-Match: *; an object already there is accepted only with the same content. Snapshots and archives (multipart) are checked absent first. Manifests and LATEST are written with If-Match against the version this worker last wrote or read. A refusal means another writer is active: the worker drops its lease and starts over through the lease. A client without conditional PUT (MUSHROOMDB_BACKUP_LEASE_CONDITIONAL=off) falls back to plain writes.

The bucket lease

A spawned worker uploads nothing until it holds the prefix’s BucketLease. While another instance holds a fresh one the worker uploads nothing, reports itself unhealthy naming the holder, and tries again every third of a TTL — so a successor waiting on a holder that died, or on one shutting down in a rolling deploy, takes over within a TTL. A separate task renews the lease every third of a TTL; every request that changes the bucket runs only while BucketLease::may_write holds and is cancelled when it stops holding. A lease found lost (taken over) is dropped, and a lease taken again later starts from the bucket’s state afresh: its previous run’s view of the bucket is stale. On shutdown, after the flush, the lease is deleted. The lease is advisory; see crate::lease for what it does and does not guarantee.

BackupWorker::run_once and BackupWorker::step take no lease: they are for tests and callers that hold one themselves.

Health

A supervisor beside the worker re-evaluates health every wal_interval_ms, whatever the worker is doing, and reports a worker that exits or panics as unhealthy with the reason. A dead backup never looks healthy, and neither does one kept out of the bucket by its lease.

Structs

BackupCounters

The worker’s counters, shared with whoever reports them.

BackupHandle

The running worker, as the serve loop holds it.

BackupWorker

The backup worker. BackupWorker::spawn runs it as a task; tests drive it a step at a time with run_once.

Enums

BackupTrigger

What the serve loop tells the worker, over a watch channel.

Constants

FIRST_BACKOFF

First retry delay after a failure; doubles per failure up to MAX_BACKOFF.

MAX_BACKOFF

Longest retry delay.

MIN_UNHEALTHY_LAG

The shortest lag that makes the worker unhealthy: durable data unshipped for longer than max(MIN_UNHEALTHY_LAG, 10 × wal_interval).

RENEW_RETRY

Retry delay after a failed lease renewal (capped at the renewal interval).

SIDE_FILES

The store’s small files that travel with every restore point.

UNHEALTHY_AFTER_FAILURES

Consecutive failed steps after which the worker reports itself unhealthy.

Traits

WalSource

Where the worker learns how much of the WAL is durable.