Module lease 

The bucket lease: an advisory, remote single-writer lock on a backup prefix.

One object, {prefix}/LEASE, names the instance that uploads to the prefix:

{ "instance_id": "…", "acquired_at": "…", "renewed_at": "…", "ttl_s": 30 }

The holder renews it every ttl_s / 3 and deletes it on shutdown. Another instance that finds a lease renewed less than its ttl_s ago (by the reader’s clock) uploads nothing. A lease older than that — its holder died or lost the network — may be taken over.

Advisory, not a distributed mutex

This catches the common operator error: two machines configured with one backup bucket. It does not make two writers safe. What it does and does not promise:

• Acquisition and renewal are compare-and-swap where the service allows it. Taking the lease is one conditional PUT — If-None-Match: * on an empty key, If-Match: <etag> over a stale lease — and so is every renewal. Two instances that start together get exactly one lease between them, and a holder whose lease was taken over learns so at its next renewal (LeaseError::Lost). Where the client cannot send conditional PUT (MUSHROOMDB_BACKUP_LEASE_CONDITIONAL=off), each step is a read followed by an unconditional write (Exclusion::BestEffort, logged at WARN): two instances that start together may both pass the check, and the later writer wins at the next renewal.
• A request whose answer was lost is not a lost lease. An instance keeps one id across attempts (BucketLease::acquire_as). A renewal or acquire that timed out on the client but landed on the service leaves a lease naming that id; the next attempt recognises it as its own instead of waiting out a TTL behind itself.
• A holder fences itself. It may start an upload only until two thirds of a TTL after its last successful renewal (BucketLease::may_write, measured on both the monotonic and the wall clock), and the worker cancels any upload still in flight at that moment. Nobody else may take the lease before a full TTL has passed, so a holder that cannot renew — partitioned from the bucket, or stalled — stops writing a third of a TTL before its lease can be taken, and writes nothing after a takeover, the partition healed or not.
• Clock skew up to that third of a TTL (10 s at the default 30 s) is absorbed: a contender whose clock runs ahead of the holder’s sees the lease expire that much early, still after the holder stopped. A lease stamped in the future by a holder whose clock runs ahead counts as fresh; such a holder that dies delays a takeover by its skew.
• The bucket fences too, where it can. The worker writes every immutable object with If-None-Match: * and LATEST and the manifests with If-Match against the version it last wrote or read, so on a conditional service a stale holder cannot overwrite an object, and its late LATEST is refused once a successor has written its own.
• What no lease can stop. A stale holder’s request that the service has already received completes, and a process paused between its fence check and its request (a VM frozen mid-call) sends it when it wakes. On a conditional service such a write can only add an object no manifest of the successor names (an orphan) or a manifest LATEST does not point at; retention deletes nothing a lineage’s kept manifests name. Without conditional writes, two overlapping writers can overwrite each other’s objects and corrupt a restore point.

Which services give real exclusion

Service	Acquire	Renew	Exclusion
AWS S3	If-None-Match / If-Match	If-Match	conditional
Cloudflare R2	If-None-Match / If-Match	If-Match	conditional
MinIO	If-None-Match / If-Match	If-Match	conditional
A service that ignores either header	refused at acquire (LeaseError::PreconditionsIgnored)	—	none: backup does not run
MUSHROOMDB_BACKUP_LEASE_CONDITIONAL=off	read, then write	read, then write	best-effort

A service that accepts a conditional PUT but ignores its precondition is caught by a probe at every acquire: the lease is written once more with If-None-Match: * and once with If-Match on a made-up ETag, and both must be refused. If either is accepted the acquire fails closed, naming the cause and the switch that runs best-effort instead. Exclusion is never weakened silently. The lease reports its kind through BucketLease::exclusion, and the worker logs it.

A lease stamped far in the future

A record renewed more than its own TTL in the future, or with a TTL over MAX_SANE_TTL_S, is suspect: a holder with a badly wrong clock, or a hand-written object. It would otherwise block the prefix for as long as the clock is wrong. A contender that sees the same suspect record unchanged for one of its own TTLs (LeaseWatch) takes it over, and logs that it did.

Structs

BucketLease

A held lease on {prefix}/LEASE. See the module docs.

LeaseRecord

The LEASE object’s contents.

LeaseWatch

What a contender has seen of a suspect foreign lease across attempts: which version, and since when, on its own clock.

Enums

Exclusion

How strongly the lease excludes a second writer on this service.

LeaseError

Why a lease could not be taken, renewed or released.

Constants

MAX_SANE_TTL_S

The longest TTL a lease record is trusted with (one hour). A record with a longer one is suspect; see the module docs.

Functions

lease_key

{prefix}/LEASE.

new_instance_id

A new random instance id: 16 bytes as 32 hex digits.