Skip to main content

Module plane_lifecycle

Module plane_lifecycle 

Source
Expand description

Controller-side lifecycle (ADR-099 section 7): mirror each instance’s state from its node, notice a Dead node, mark its instances Lost and reschedule them through the same gate and placement engine as any placement, and settle with a node that comes back.

Delivery is at-least-once, with no fencing. While a node is cut off but still running, its copy keeps running (and a sensor cog keeps producing) until the controller notices, replaces it and the node returns to be told to unload the old one. For the length of the partition the workload may run twice. The opt-in node lease (WorkloadHostService::with_lease) narrows that window: a node that has heard no controller for the lease stops its continuous instances itself. See ADR-099 section 7.

PlacementControlPlane::lifecycle_tick is one pass:

  1. re-describe every target (reachability) and note how long each has been down;
  2. a node is Dead when it has been unreachable for PlaneConfig::dead_after and membership, when it knows the peer, does not still hold it as healthy. Direct contact is the authority: membership never makes a node dead that answers;
  3. each instance on a Dead node becomes Lost and is rescheduled (plane_reschedule) unless it is pinned, non-migratable (operator, kind, or needing attached hardware), has no stored order, or moved or failed to move too often; those raise an alert (chained) and stay Lost;
  4. the old record of a rescheduled instance is kept as Rescheduled. When its node returns, that copy is unloaded there (signed, gated, chained), with backoff and one chained report per distinct failure;
  5. a node that is reachable but no longer holds an instance is told apart: it reports explicitly what a revocation or an unload took down, and anything else was lost with the node (a reboot) and is placed again;
  6. an instance that was Lost and is still on its node when it returns is adopted back (a lease-stopped one is restarted).

Every step is chained: workload.lifecycle for state changes and alerts, workload.migrate for a reschedule.

Structs§

ControllerLife
What the controller keeps per instance beyond its placement record.
LifecycleEvent
Something one PlacementControlPlane::lifecycle_tick did.