Expand description
Controller-side lifecycle (ADR-099 section 7): mirror each instance’s
state from its node, notice a Dead node, mark its instances Lost and
reschedule them through the same gate and placement engine as any
placement, and settle with a node that comes back.
Delivery is at-least-once, with no fencing. While a node is cut off
but still running, its copy keeps running (and a sensor cog keeps
producing) until the controller notices, replaces it and the node
returns to be told to unload the old one. For the length of the
partition the workload may run twice. The opt-in node lease
(WorkloadHostService::with_lease) narrows that window: a node that has
heard no controller for the lease stops its continuous instances itself.
See ADR-099 section 7.
PlacementControlPlane::lifecycle_tick is one pass:
- re-describe every target (reachability) and note how long each has been down;
- a node is
Deadwhen it has been unreachable forPlaneConfig::dead_afterand membership, when it knows the peer, does not still hold it as healthy. Direct contact is the authority: membership never makes a node dead that answers; - each instance on a
Deadnode becomesLostand is rescheduled (plane_reschedule) unless it is pinned, non-migratable (operator, kind, or needing attached hardware), has no stored order, or moved or failed to move too often; those raise an alert (chained) and stayLost; - the old record of a rescheduled instance is kept as
Rescheduled. When its node returns, that copy is unloaded there (signed, gated, chained), with backoff and one chained report per distinct failure; - a node that is reachable but no longer holds an instance is told apart: it reports explicitly what a revocation or an unload took down, and anything else was lost with the node (a reboot) and is placed again;
- an instance that was
Lostand is still on its node when it returns is adopted back (a lease-stopped one is restarted).
Every step is chained: workload.lifecycle for state changes and alerts,
workload.migrate for a reschedule.
Structs§
- Controller
Life - What the controller keeps per instance beyond its placement record.
- Lifecycle
Event - Something one
PlacementControlPlane::lifecycle_tickdid.