Enum RecoveryDecision 

pub enum RecoveryDecision {
    Nothing,
    Clean,
    Adopt,
    Wait,
    Defer,
    Observe(AttemptState),
    Conclude(AttemptOutcome),
    Terminate(AttemptOutcome),
}

What to do about one attempt found in the journal at startup.

Variants

Nothing

Already cleaned; there is nothing left to do.

Clean

Terminal but not yet cleaned: remove the runtime directory and mark it cleaned.

Adopt

The process is alive and this attempt is still ours. e3: “an attempt whose process still runs is adopted, not duplicated.”

Wait

Nothing is decidable yet; look again next cycle.

Defer

GitHub is unreachable. Decide nothing destructive during an outage (flow 3.3).

Observe(AttemptState)

Move the attempt to this state to match what was observed.

Where the observed state is live, the caller must adopt supervision of the process independently of this decision. RecoveryDecision has no way to express “adopt and observe” — the two are separate variants — so Starting with process_alive and Registered { busy: false } returns Observe(Idle) and never Self::Adopt, because GitHub is authoritative for the remote status while the local process is authoritative for supervision, and both facts are true at once. Idle with process_alive and Registered { busy: true } returns Observe(Busy) for exactly the same reason; it did not always, and the arm’s own comment records what that cost.

An Observe decision must be applied and persisted, or it repeats forever. The decision is a pure function of the journalled state and the observation, so a caller that moves the in-memory attempt without writing the new state back will read the same stale state on the next pass and be handed the same decision, indefinitely.

Conclude(AttemptOutcome)

Conclude the attempt with this outcome.

Terminal, and therefore a capacity release. An attempt is only concluded where the thing it was supervising is already gone; see Self::Terminate for the case where it is not.

Terminate(AttemptOutcome)

The process is still alive but the attempt cannot go on: stop the process, and only once it is gone record this outcome.

Two producers, and the payload is what separates them. starting past its registration timeout carries FailureReason::RegistrationTimedOut; idle past its idle timeout, with GitHub still reporting the runner registered and unassigned, carries AttemptOutcome::ExitedIdleWithoutWork — flow 2.7’s surplus runner, which is a normal outcome and not a failure at all. The two share this variant because they need the identical sequence — signal, confirm the process is gone, then record — and differ only in what is recorded at the end. A caller that hardcodes either reason will mislabel the other, so the payload is the caller’s instruction, not decoration.

Why this is not a Self::Conclude. Conclude moves the attempt to a terminal state, and a terminal attempt no longer counts against capacity — so concluding one whose process is still running hands the host back a slot it is still using. The agent then starts a replacement runner beside a live, unregistered one that may yet register and take a job, for an attempt the journal already calls failed. There is no RecoveryDecision that would have expressed the fix: Self::Adopt means take over supervision and Self::Clean means delete a runtime directory, and neither stops anything.

What happens to the capacity slot. Nothing, until the process is actually gone. The attempt stays in its current, non-terminal state and keeps holding its slot for as long as the runner it started is running, which is the honest answer — the resources are genuinely occupied. The slot comes back at the moment the caller applies the payload through RunnerAttempt::conclude, which it does only after the process has exited.

It is safe to re-derive in one direction, and owes a debt in the other. The decision is a pure function of the journalled state and the observation, so an agent that dies before terminating sees a live process and a larger elapsed on the next pass and is handed this same decision again. That half costs nothing and needs nothing.

The other half is a known defect, and it is not harmless. An agent that terminates the process and then dies before writing the outcome observes process_alive: false next time and reaches the ordinary Self::Conclude arm, which records FailureReason::ProcessExitedUnexpectedly. The process did not exit unexpectedly; this agent killed it. That is precisely the diagnosis FailureReason::RegistrationTimedOut was split out to prevent — the two reasons send an operator to different places, and this window sends them to a crash investigation (logs, exit code, a corrupt package) for what was a registration failure.

It cannot be fixed here, and the alternative was weighed rather than waved off. This function’s inputs are the journalled state and a RecoveryObservation, and neither carries the fact that separates the two cases: a process this agent killed and a process that crashed on its own present the same observation. Using RegistrationTimedOut in the dead-process arm once elapsed >= startup would close this window at the price of a wider one — every genuine early crash first observed after a restart longer than the startup window would then be reported as a runner that “is running but did not register”, to an operator who can see that it is not running. That trades a rare wrong reason for a common false claim about liveness, in the one direction FailureReason::ProcessExitedUnexpectedly’s own documentation says spends the credibility of every other message this product prints. tests::no_decision_calls_a_dead_process_live is what stops that trade being made by accident later.

So the obligation is e3’s, and it is a persistence one. The distinguishing fact exists only at the moment terminate-intent is formed, and the only way to carry it across a crash is to write it down. e3 journals the intent before it signals the process, and on a later pass concludes an attempt it finds so marked with FailureReason::TerminatedAfterRegistrationTimeout rather than with whatever this function derived from an observation that could not know. Until e3 does that, the window stands.

Why that closure needs a reason of its own, and not RegistrationTimedOut. On the pass where e3 reads the mark back, the process is dead — e3 killed it, which is the whole reason the mark is there. RegistrationTimedOut renders as “the runner process is running but did not register”, so concluding with it would print exactly the false liveness claim the paragraph above rejects option A for: the same sentence, about a process an operator can see is gone, moved one pass later. Rewording that string instead is not open either — tests::the_two_starting_failures_read_differently_to_an_operator pins it on “running”, which is correct for the live case it names.

FailureReason::TerminatedAfterRegistrationTimeout is that reason. It is true of a dead process, it says who stopped it and why, and it sends an operator to the networking and configuration fix rather than to a crash investigation — which is the whole distinction FailureReason::RegistrationTimedOut was split out to draw. Nothing in this function derives it, and nothing should: it is a claim about an action this agent took, not about anything a RecoveryObservation reports. tests::no_decision_calls_a_dead_process_live covers it alongside the other two liveness-claiming reasons, so the day something here does start deriving it — which it legitimately might, once the mark is journalled where this function can read it — it may only do so beside a process the observation says is gone.

The one thing this does not bound is a process that refuses to die. The slot is held until it does. That is a worse outcome than concluding early only if the runner was never going to register, and a better one in every case where it was — and unlike the early conclusion it cannot oversubscribe the host.

Trait Implementations

impl Clone for RecoveryDecision

fn clone(&self) -> RecoveryDecision

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RecoveryDecision

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for RecoveryDecision

impl PartialEq for RecoveryDecision

fn eq(&self, other: &RecoveryDecision) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for RecoveryDecision