Module cluster 

SS-5b automatic multi-node failover detection (cluster supervisor). Only meaningful for a distributed haematite boot, so gated behind the backend. SS-5b: automatic multi-node failover detection.

[ClusterSupervisor] is the production counterpart to the manual Engine::adopt_shards trigger proven in the SS-5 demo. It runs a background task that watches the liveness of every peer that owns shards and, when a peer’s replication link drops and stays down past a debounce threshold, calls adopt_shards for that peer’s shards ITSELF — no human in the loop.

How peer-down is detected

The liveness signal is the haematite distribution link state ([HaematiteStore::peer_connected]): beamr’s OTP distribution tears the connection down (read-loop EOF → deregister) the instant the peer’s process dies, so peer_connected flips to false on a real kill -9 exactly as it does on a graceful drop. It is a true socket-liveness signal, not a heartbeat heuristic.

Debounce

A single missed poll is not a death: a transient blip must not trigger a disruptive shard adoption. The supervisor requires confirmations CONSECUTIVE polls observing the peer disconnected before it acts. Any single reconnect observation resets the counter. Once a peer’s shards are adopted it is marked handled and not re-adopted while it stays down (adoption is itself idempotent, but re-running it every tick would be wasteful); a later reconnect clears the handled mark so a flapping peer that genuinely dies again is re-adopted.

Scope

Behind the haematite-backend feature and only ever constructed for a distributed ([store.cluster]) boot. A single-node / non-clustered server never spawns it, so default behaviour is unchanged.

Structs

ClusterSupervisor

Watches peer liveness and auto-adopts a dead peer’s shards (SS-5b).

OutboxSettlingAdopter

ShardAdopter over the live engine that, when the durable outbox is commissioned, re-runs the terminal-workflow outbox settlement sweep (#253) after each successful adoption.

SupervisorConfig

Tuning for the supervisor’s poll loop.

WatchedPeer

One peer the supervisor watches: its distribution name and the shards it owns (which this node will adopt if the peer dies).

Traits

PeerLiveness

The liveness signal the supervisor polls. Implemented by [HaematiteStore] in production and by a fake in tests, so the debounce/adopt logic is verified without standing up a real cluster every time.

ShardAdopter

The failover action the supervisor invokes when a peer is confirmed down. Implemented by Engine in production and by a fake in tests.