Module cluster 

Cluster-mode env preparation. fdl-cli sets FLODL_INTERNAL_FULL_CLUSTER_JSON

• FLODL_INTERNAL_FDL_CMD + FDL_ENV on its process env so the user binary inherits them and detects launcher role via flodl::distributed::launcher::dispatch. Fan-out, log fan-in, and ClusterController all live on the flodl side. Entry point cluster::prepare_cluster_env; recursion guard via cluster::should_dispatch. Cluster-mode env preparation.

Process-per-rank model: flodl owns fan-out and controller orchestration (flodl::distributed::launcher::run_launcher). fdl-cli’s job here is purely to ship the parsed cluster topology to the launcher via env vars, then let the normal RunScript / ExecCommand dispatch invoke the user binary. The user binary’s flodl::distributed::launcher::dispatch reads the env, detects launcher role, and fans out (ssh for remote hosts, fork+exec for local hosts). All log fan-in + ClusterController + exit-code propagation happen on the flodl side.

fdl @cluster train
  ↓ fdl-cli parses fdl.yml + fdl.cluster.yml overlay
  ↓ fdl-cli calls prepare_cluster_env: sets FLODL_INTERNAL_FULL_CLUSTER_JSON,
    FLODL_INTERNAL_FDL_CMD, FDL_ENV on its own process env
  ↓ fdl-cli falls through to normal RunScript / ExecCommand path
  ↓ resolved command (e.g. `cargo run --release --bin my-trainer`) runs
  ↓ my-trainer inherits env, flodl::launcher::dispatch detects Launcher
  ↓ launcher fans out: ssh per remote host, fork+exec per local rank
  ↓ each rank child has FLODL_INTERNAL_CLUSTER_JSON + FLODL_INTERNAL_LOCAL_RANK set
  ↓ rank-side flodl::launcher::dispatch returns Role::Rank, training runs

Recursion guard: the launcher’s ssh fan-out invokes fdl <cmd> on the remote, which re-enters fdl-cli with FLODL_INTERNAL_CLUSTER_JSON set (not FLODL_INTERNAL_FULL_CLUSTER_JSON). should_dispatch returns false in that case so the remote fdl-cli skips cluster setup and just runs the user binary normally — the user binary’s launcher dispatch then detects Role::Rank (because FLODL_INTERNAL_LOCAL_RANK is also set).

Constants

ENV_CLUSTER_EXTRA_HOSTS

Pre-resolved name:ip pairs (space-separated) for every cluster host, written by prepare_cluster_env using the controller’s NSS resolution. Consumed by run/prebuild/schema-cache when they build docker compose run --rm commands: each pair is injected as a --add-host name:ip flag so the containerized launcher can SSH into cluster hosts without depending on the container’s own resolver (which lacks libnss-libvirt etc.).

ENV_CLUSTER_JSON

Env var name carrying the slim per-rank envelope. Set by the launcher (not fdl-cli) on each rank child. Kept here so the recursion guard can reference it by name. Mirrors flodl::distributed::cluster::ENV_CLUSTER_JSON.

ENV_FDL_CMD

Env var name carrying the original fdl command name (e.g. train). Read by the launcher when it needs to invoke fdl <cmd> over ssh on remote hosts. Mirrors flodl::distributed::launcher::ENV_FDL_CMD.

ENV_FDL_ENV

Env var name picking the overlay env name (e.g. cluster). Set by fdl-cli at env-selector parsing time; propagated through to remote hosts by the launcher so they see the same overlay-merged view.

ENV_FULL_CLUSTER_JSON

Env var name carrying the full multi-host topology (hex-encoded JSON of ClusterConfig). Set by fdl-cli on its own process env so the spawned user binary inherits it and detects launcher role. Mirrors flodl::distributed::launcher::ENV_FULL_CLUSTER_JSON.

ENV_HOST_OVERRIDE

Env var name overriding the OS hostname for cluster lookups. Mirrors flodl::distributed::cluster::ENV_HOST_OVERRIDE.

ENV_HOST_USER

Controller’s OS user name (resolved on the host by fdl-cli, before any docker spawn). The launcher in the container reads it as the default ssh -l target when the per-host ssh.user: is unset. Bridges the container-vs-host user mismatch (containers ship a stock ubuntu UID-1000 user, but ubuntu@<remote> is rarely the account the user actually uses on cluster hosts).

ENV_LOCAL_RANK

Env var name picking this rank’s local-rank index within its host. Set by the launcher on rank children. Mirrors flodl::distributed::cluster::ENV_LOCAL_RANK.

ENV_NET_TIMEOUT_SCALE

Env var scaling every flodl cluster network deadline together (connect budget, write-stall, heartbeat staleness, coord-liveness, CPU reduce read). Mirrors flodl::distributed::wire’s constant + validation rule (kept in lockstep — flodl-cli is decoupled from the library crate). The library reader warns-and-defaults on a bad value; the cluster fan-out path calls validate_net_timeout_scale first so an explicit-but-invalid value errors loudly BEFORE any host is touched.

Functions

cluster_compose_overlay_arg

Write a temporary docker-compose overlay (under project_root) that populates extra_hosts: for the cluster-capable services (cuda, dev, bench) from the controller-resolved cluster hosts in ENV_CLUSTER_EXTRA_HOSTS, then return the -f flag sequence to splice in front of docker compose run.

hex_encode

Hex-encode raw bytes (lowercase, no separators). Companion to the library’s hex_decode in flodl::distributed::cluster. Kept here so prepare_cluster_env doesn’t pull in a flodl runtime dep.

is_recursive_invocation

Whether this fdl invocation is itself a spawned child of a launcher’s ssh fan-out (FLODL_INTERNAL_CLUSTER_JSON already set in env). Used as the recursion guard everywhere cluster dispatch is evaluated.

is_reserved_cluster_env_key

True if key must not appear in a user-supplied cluster/host env map. Mirrors flodl::distributed::cluster::is_reserved_cluster_env_key (kept in lockstep — flodl-cli is decoupled from the library crate, so the reserved rule is duplicated, not imported). The launcher applies user env after its own built-ins (shell last-wins), so a launcher- owned key set via env would silently override device mapping / rank identity / the HMAC envelope. Reserved: the loud FLODL_INTERNAL_ prefix (all launcher-private vars, future-proof) plus three names that are user-facing elsewhere but launcher-owned per-rank here (CUDA_VISIBLE_DEVICES + CUDA_DEVICE_ORDER, ENV_HOST_OVERRIDE, ENV_FDL_ENV). User-facing knobs (FLODL_VERBOSITY, FLODL_DASHBOARD_BIND, NCCL tuning, LD_LIBRARY_PATH) are deliberately allowed.

prepare_cluster_env

Prepare the env vars needed for the user binary’s flodl launcher to detect launcher role and fan out. Caller continues to normal dispatch (RunScript / ExecCommand); the spawned subprocess inherits these env vars and the launcher takes over.

prepare_test_cluster_env

Prepare the testing-cluster envelope (FLODL_TESTING_CLUSTER_JSON).

resolve_local_hostname

Resolve the local OS hostname. Used by gpus::synthesize_local_cluster (the --gpus single-host shorthand) and by prebuild to skip the controller from the remote-host fan-out. Test/override seam via ENV_HOST_OVERRIDE; falls back to the hostname(1) command.

resolve_local_user

Resolve the controller’s OS user name. Used to pre-populate ENV_HOST_USER before docker spawn, so the launcher inside the container can default ssh -l <user> to the host’s identity. Falls through USER then whoami; None when both fail — the caller then leaves ENV_HOST_USER unset and the launcher omits -l entirely, so ssh applies its own defaults (ssh_config User directives, the effective uid). Never fabricate a name: a made-up -l unknown-user produced a bare “Permission denied (publickey)” with no hint the username was invented.

should_dispatch

Top-level cluster-dispatch decision.

validate_net_timeout_scale

Validate FLODL_NET_TIMEOUT_SCALE from the process env: unset is fine (scale 1.0); a set value must parse as a finite float ≥ 0.1. Mirrors the library’s parse rule.