Module reconcile

Expand description

The loop that turns GitHub demand into a decision to start runners — and that refuses to start them when it should not.

Every ceiling in this product is enforced from here, so the module is organised around the four things that can go wrong silently:

PollSchedule — the budget-aware interval. Demand shares one 5,000 requests/hour ceiling with inventory and workflow counts, so this loop polls on a bounded interval (default 60 s, hard floor 30 s per target) and increases the delay under a rate-limit signal, never decreases it to catch up.
RepositoryCache — the per-organization repository list, refreshed on an interval materially slower than the demand poll. Re-listing an organization at demand-poll frequency is what exhausts the shared budget the paragraph above exists to protect.
Reconciler::reconcile — the allocation pass. It re-reads the attempt set under the host-wide allocation lock, once per runtime created, so two policies reconciling concurrently cannot both spend the same headroom.
LifecycleEvent — what g2 and the local log sink see. Every field is an identifier, a count, an enumerated state or a duration; nothing free text, and nothing that came off the wire.

§There is no acquisition step, and none may be added

The scale-set model called AcquireJobs to reserve an assignment before scaling. The REST path has no equivalent (01-current-architecture.md, edge case 6), so demand is advisory. Two consequences are load-bearing here and neither is a defect:

A surplus runner is an accepted outcome. Another host serving the same labels may take the job first; this host’s runner then finds no work and exits on its idle timeout, having cost one capacity slot and one cold start. That terminal outcome is AttemptOutcome::ExitedIdleWithoutWork, is cleaned like any other, and is counted apart from a failure — see ReconcileReport::idle_exits.
The same job is still queued on the next poll while its runner starts. The - active_owned_runners term in HostAllocator::allocate is what stops that from starting a second runner, and then a third. This module’s only job in that arithmetic is to hand the allocator the attempt set the host actually holds — which is why RunnerLauncher supplies both the attempts and the launch, from one supply point, for the reason b1 gives at HostAllocator::from_attempts.

tests::nothing_in_this_module_reserves_or_claims_a_job is a tripwire on the obvious shape of a reservation being added back.

§Demand is measured in JOBS, filtered by this policy’s routing labels

02-target-architecture.md writes the formula as “queued jobs whose runs-on matches this policy’s routing labels”, and that is now exactly what this module clamps. It was not always: an earlier owner decision priced the per-run job listing out and left this module clamping a count of runs, unfiltered. crates/github/src/demand.rs records that decision, why it was reversed, and what the reversal costs in requests.

What the reversal means here is two changes to one line:

A run of eight jobs is now eight units of demand, not one. Under the run count a matrix filled one runner per poll while the rest of the matrix waited, so a host configured for ten concurrent runners served an eight-job matrix nearly serially. That was the defect that forced the decision back.
A job this host cannot serve is no longer demand. A repository whose jobs target ubuntu-latest, or another host’s rm-<host>-… label, used to drive its policy toward max_capacity and start runners that idled until they timed out. The gateway now returns each queued job’s runs-on, so b1’s predicate finally has its input.

The predicate is still b1’s and the input is still c4’s. This module calls runner_manager_domain::policy::RoutingLabels’s tally and implements no label comparison of its own; tests::the_label_predicate_is_b1s_and_this_module_only_applies_it scans this file’s own source and fails if a second implementation grows here, which is the same tripwire c4 carries one layer down.

§The filtering happens here rather than in the gateway, on purpose

One target can be watched by more than one policy, each with its own routing labels, and Reconciler’s poll_targets deliberately polls a target once for all of them. A gateway that filtered would have to be told whose labels to filter by, which would make the poll per-policy and multiply its request cost by the number of policies sharing the target — the budget model prices a target, not a policy. So the gateway returns the jobs and each policy tallies them against its own labels.

§What is still approximate

The surplus-runner path above is narrowed by this change and not closed. A runs-on: ${{ matrix.runner }} cannot be resolved without evaluating the workflow, so b1 reports it as unresolvable: never counted as demand, never silently dropped, and surfaced through LifecycleEvent::DemandObserved::unresolvable so that an operator can see a workflow this host will never serve sitting in the queue. And demand remains advisory — another host may still take a job this one started a runner for — which is what the two ceilings bound.

§What is testable without a network, a filesystem, or a process

All of it. DemandSource, RunnerLauncher, AllocationLock, RepositoryDirectory, Jitter and EventSink are ports; GatewayDemand, FileAllocationLock, RandomJitter and TracingEvents are the production adapters, and every one of them is a thin shell over a decision made in this file.

Structs§

AllocationGuard: The lock is held for as long as this value lives.
AllocationLockBusy: The host-wide allocation lock could not be taken.
EventLog: Keeps every event, in order.
FileAllocationLock: d1’s file lock, which is host-wide across processes as well as across tasks.
FixedJitter: A fixed fraction, for tests and for the acceptance suite.
GatewayDemand: DemandSource over c4’s DemandGateway.
InProcessAllocationLock: The lock every task inside one agent process contends for.
LaunchFailure: Why one runner could not be started.
LaunchRequest: What this loop asks e3 to create.
NextPoll: When to poll next, and why then.
NoEvents: Discards everything. For callers that only want the report.
NoJitter: No jitter at all: the back-off is exactly what the schedule computed.
OfflineState: What an operator is told while GitHub is unreachable.
PollSchedule: The bounded, budget-aware poll interval.
RandomJitter: The production source.
ReconcileReport: What one reconciliation pass did.
Reconciler: The reconciliation loop.
ReconcilerPorts: Everything one reconciler needs, written down at the call site.
ReplacementIntent: A lifecycle conclusion that must return through ordinary demand and capacity allocation before another runner may start.
RepositoryCache: The per-organization repository list, refreshed far more slowly than demand.
ScaleDownReport: What one scale-down request did.
TeeEvents: Both sinks at once: the log sink for the operator’s file, the buffer for g2’s screen.
TracingEvents: The local log sink, through d1’s redacting layer.
WslRecoveryAllocationLock: Adds the Windows/WSL recovery fence to an ordinary allocation lock.

Enums§

LifecycleEvent: What g2’s activity view and the local log sink see.
OutcomeKind: Which terminal thing happened, as a closed vocabulary.
PollOutcome: One target’s demand poll, as a value this module can decide from.
PollPace: Why the next poll is when it is.

Constants§

ALLOCATION_LOCK_WAIT: How long FileAllocationLock waits for the host-wide allocation lock before reporting contention.
GITHUB_CANCELS_QUEUED_JOBS_AFTER: GitHub cancels a queued job after this long.
MAX_OFFLINE_BACKOFF: The longest the unjittered offline back-off may grow to.
REPOSITORY_LIST_REFRESH_MULTIPLE: How much slower than the demand poll the per-organization repository list is refreshed.

Traits§

AllocationLock: Flow 2.4’s “takes the host-wide allocation lock before creating each local runtime”, as a port.
DemandSource: Where this loop gets its demand from.
EventSink: Where lifecycle events go.
Jitter: The randomness in the offline back-off, as a port.
RepositoryDirectory: Which repositories an organization installation reaches.
RunnerLauncher: The seam between the decision to start a runner and the act of starting one.

Functions§

failure_reason_kind: A FailureReason’s variant name, with no detail.