Module cluster 

THE MEMBERSHIP HANDSHAKE (D9 — zero control plane, the leader is a worker): a run’s workers rendezvous through the leader and open the exchange mesh. No scheduler, no etcd:

• Rendezvous. The leader listens on a control port; each --join worker connects, registers its exchange address, and the leader — once every worker has registered — broadcasts the sorted member set and the topology fingerprint. Every worker then derives the same placement (crate::placement) from that set, so no worker is told its slice (D9).
• Mesh. Every worker opens exactly one Link to every peer: it connects to peers with a lower id and accepts from peers with a higher id (the peer-id preamble on Link::connect_as lets one shared listener demultiplex the higher peers). One Link per pair, full-duplex, handed to Graph::start_worker as a WorkerExchange.

Control messages are length-prefixed postcard (§7.2 — tiny, schema-evolvable). This is the transport-agnostic handshake; failure detection and re-assignment on worker loss are 4c.

Structs

ClusterCfg

How this process participates in a run: solo (no clustering), or one worker of a cluster — either the leader (waits for n_workers to register) or a joiner (registers with a leader).

LeaderCtl

The leader’s live control connections to the joined workers (worker id → its stream), kept past the handshake so the leader can drive epochs, commits and shutdown across processes. The leader is itself a worker (D9); it coordinates its own slice in-process and the joined workers here.

WorkerCtl

A worker’s live control connection to the leader.

Enums

Coord

This process’s coordination role once the handshake settles: the leader holds connections to the joined workers; a joiner holds its connection to the leader.

ToLeader

A worker’s report back to the leader. Acked = every local task reported epoch e, this worker’s objects/files are durable, and here is its manifest slice (source offsets, objects and files for the tasks it owns — keyed by global task id, so the leader merges every worker’s slice into the one manifest it writes). Finished = this worker’s whole graph drained. The leader waits for every worker before it commits an epoch or completes shutdown.

ToWorker

Runtime control from the leader to a worker over the persistent control connection (D9 — the leader coordinates the run; barriers travel in-band on the data edges, but the decision to issue an epoch, commit it, or stop is the leader’s). Barrier(e) injects epoch e at this worker’s sources (aligned across all workers, so an operator sees the same epoch on every edge); Commit(e) releases the sources that recorded a position at e; Stop begins the ordered shutdown.

Functions

connect

Settle this worker’s membership: bind its exchange listener, then lead or join. base_fp is the topology fingerprint without the member clause — the leader broadcasts it and every joiner checks its own against it, so a worker whose steps/vnodes differ is rejected before it runs. Returns the settled Members, the exchange listener (for mesh), and this process’s coordination role (Coord) with its live control connection(s).

join

The worker side: connect to the leader’s control address, register my_exchange_addr, and receive the membership. Returns this worker’s Members and the leader-declared fingerprint (the caller checks it against its own local fingerprint — a mismatch is a topology disagreement).

lead

The leader side of the rendezvous: wait for n_workers - 1 registrations on control, then broadcast the sorted member set + fingerprint to every worker. Returns the leader’s own Members (its me is my_exchange_addr’s index in the sorted set). The leader is worker 0 only if its address sorts first — placement depends on the addresses, not on who leads.

mesh

Open the exchange mesh from a settled membership: one Link per peer — connect to lower ids, accept from higher ids on exchange — returning the WorkerExchange start_worker consumes. Accept and connect run concurrently so no worker deadlocks waiting on a peer that is waiting on it.