Module pool 

Multiplayer farm-out: one Foreman run spread over several CAR instances.

run_farm_out takes a single WorktreeAgent and runs every subtask through it. FleetPool is a WorktreeAgent, backed by several — the local coding CLI plus a worker on each reachable peer — so the harness, the gate, and the integration step are untouched. Distribution is a placement decision underneath an unchanged interface, which is the whole reason it costs no new soundness argument:

• The worktree, the patch capture, the AST containment check, the build/test leg, the policy consult, and the union integration all still happen on the orchestrating host.
• A remote worker’s only output is edits in that worktree. A broken or hostile one produces a patch the local gate then rejects, exactly like a local agent having a bad day.

Placement

Least-loaded first, ties broken by declaration order, so a caller expresses preference by ordering its workers. Each worker has a capacity — the number of subtasks it will run at once — enforced by a semaphore, because a machine that accepts eight parallel coding CLIs when it can serve two turns a speed-up into a thrash.

Failover, and why it must reset the worktree

A worker that errors (network dropped, CLI missing, peer declined) hands the subtask to the next candidate. But a half-finished attempt leaves edits behind, and the next worker would then be editing someone else’s partial work and the gate would attribute the mess to the subtask. So every failover resets the worktree to the commit it was provisioned at before retrying. A run that cannot reset does not retry: silently continuing from a dirty tree is the one outcome worse than failing the subtask.

Quarantine

Failing over per-subtask is not enough on its own. A worker that fails never takes a permit, so it keeps maximum availability and candidates ranks it FIRST again for the next subtask — a peer that dies mid-run is then tried, and times out, once for every subtask left (car#1323).

So a REMOTE worker that returns ForemanError::Worker is excluded for the rest of the run. That error means the failure is a fact about the machine rather than the subtask, which is exactly the condition under which retrying it buys nothing; a subtask that merely failed returns ForemanError::Agent and changes nothing about the worker. The local worker is never quarantined — it is the one guaranteed-reachable machine, and the pool must not be able to empty itself.

The bound is the worker’s CAPACITY, not one attempt. A level runs under futures::future::join_all, so several subtasks can be inside a dying peer before any of them sets the flag — and acquire_owned WAITS on the top-ranked candidate rather than skipping to a free one, so the rest of the level parks on its semaphore. The flag is therefore re-read on the far side of the acquire, which is what turns “one dispatch per remaining subtask” into “one per permit”. Wall-clock cost is one timeout, not N.

Run-local and one-way: a peer that comes back stays out until the next run. Re-probing liveness mid-run is a different feature, and the cost this fixes is already paid by then. The one member of the set with a shorter natural life is a rate limit, whose window is hourly — a run longer than that loses a peer that would have been served again. Still not worth a re-probe.

Nothing produces ForemanError::Worker locally today: ForemanExternalAgent never returns it. The remote-only guard is there because a local one would mean this host’s own coding CLI vanished, which quarantining cannot route around and which could leave the pool with nothing to run.

Structs

FailedAttempt

One failed attempt at a subtask, kept so a run that eventually succeeded still shows which workers dropped it.

FleetPool

A WorktreeAgent that spreads subtasks over a set of workers.

FleetWorker

One place subtasks can run.

Placement

Where a subtask ended up running.