#[non_exhaustive]pub enum CoordinationEvent {
Gained {
split: SplitSpec,
epoch: LeaseEpoch,
progress: Option<SplitProgress>,
},
Lost {
split: SplitId,
},
RevokeRequested {
split: SplitId,
},
Quarantined {
split: SplitId,
attempts: u32,
},
AllComplete,
Stalled {
completed: u64,
quarantined: u64,
},
}Expand description
Ownership or job-state change surfaced by SplitCoordinator::poll.
Events for one split are ordered: a split is Gained before it can be
Lost, and a re-Gained split always carries a higher LeaseEpoch
than the tenancy it replaces.
Variants (Non-exhaustive)§
This enum is marked as non-exhaustive
Gained
This instance now holds the lease for the split and must start (or resume) processing it.
Fields
split: SplitSpecThe split now owned, descriptor included, because the gaining worker never saw the planner run.
epoch: LeaseEpochFencing token for this tenancy.
progress: Option<SplitProgress>The last fenced-committed progress to resume from. None for
a split that has never committed.
Lost
The lease was lost, either seized by a peer after expiry, stolen for
balance, or self-fenced after renewals could not reach the backend.
The source must stop the split promptly and must not commit it
again (a late commit is rejected as
CoordinationErrorKind::Fenced regardless; this event is the
cooperative fast path).
RevokeRequested
The leader has stopped assigning this split to this instance, and wants it back. The owner should stop intake at a safe boundary, chase the split’s tail to a final fenced commit, and release it. The next owner then resumes from a point covering everything this one emitted, so the transfer replays nothing.
The split is leaving either way. A revocation is a decision
rather than a proposal. A source that declines (through the driver’s
SplitSource::begin_revoke,
which defaults to declining) or that does not finish inside
drain_deadline has the release forced instead, and its
uncommitted tail replays under the next owner. Declining is
therefore still safe; it is the expensive way to comply.
The one exception is not the source’s to take. A backend may cancel a
revocation its leader took back (the split is named for this instance
again while it still holds it), and then nothing is forced. A source
cannot observe that and must not wait for it. It also changes little
for a source that already accepted: intake stays stopped, the drain
still ends by handing the split back, and this instance is the
one that gains it again, through a fresh
Gained with a new lane. Only a source
that declined keeps the split without interruption.
Idempotent: the event may be re-emitted for a split already draining, and a revocation for a split this instance does not hold is a silent no-op.
Quarantined
The split exhausted its delivery attempts (repeated owner deaths or
explicit fail reports) and was parked.
It will not be re-offered; it stays visible in the store and in the
spate_coordination_splits_quarantined gauge, and it blocks
AllComplete.
AllComplete
Final plan and every split committed completed. Bounded sources
translate this into
SourceEvent::Drained. An
instance that owns no splits must keep polling until this arrives;
it stands by to cover an owner dying at the finish line.
Stalled
Final plan, nothing left runnable or running, but quarantined
splits remain, so the job cannot finish cleanly. Surfaced to every
instance exactly where AllComplete would have been. The source
decides whether this is fatal (the default in the driver) or a
drain-with-warning.