Skip to main content

Module join

Module join 

Source
Expand description

Self-deployed worker dial-in (fdl join): tunnel bring-up, agent bootstrap-spec synthesis, training-binary supervision. fdl join — self-deploy this box as a dial-in worker.

The worker-side walk-in for a discovery window (see docs/ddp/01-reference.md): dials the controller’s mux port, offers this host’s GPUs, and — once admitted — the training binary takes over in agent role (it joins, then spawns and supervises this host’s relay and rank children; see flodl’s distributed::launcher::agent). fdl-cli stays protocol-blind: its whole job is orchestration —

  1. resolve settings (flags over the fdl.yml join: block),
  2. prepare this box (crate::prepare): the GPU gate, the dataset source root, the node-local directories the data plane writes — all of it BEFORE the dial, because admission starts a window deadline,
  3. optionally bring up an ssh -L forward of the controller port (the guardrailed-sshd trust path: reachability = authentication),
  4. synthesize the agent bootstrap spec into the binary’s environment (FLODL_INTERNAL_AGENT_JSON, hex-encoded JSON — the same envelope cluster fan-out ships),
  5. run + supervise the binary, and in --persist mode re-dial with backoff when it exits (the systemd / golden-image loop).

The spec (which may carry the pre-shared session token) rides the child’s ENVIRONMENT, never argv — owner-readable via /proc/<pid>/environ instead of world-readable via ps, the same salt hygiene as the launcher’s fan-out.

Constants§

EXIT_PERMANENT
Exit code for a failure retrying cannot fix: no usable GPU, a spec that does not parse, a directory that cannot be created, a missing binary. Distinct from 1 (a transient failure, one-shot) so a fleet can act on the difference without parsing stderr:

Functions§

run
Run fdl join. bin_tail is everything after the command line’s standalone --: the training binary’s own arguments, forwarded verbatim (rank children re-enter the binary with them). None = no -- was given (the config block’s args: applies); a present but empty tail is an explicit “no arguments”.