Module prebuild 

Pre-flight build for cluster commands. Builds the target binary locally (in Docker on the controller) for each remote host’s libtorch ABI before fan-out, delivering it via the shared project-root mount so the remote can exec it directly without a cargo / rustc toolchain. Pre-flight build for cluster commands.

Heterogeneous-rig pain: source lives on a shared mount (NFS / virtiofs / S3-FUSE) so editing on the controller is visible to remote hosts, but each host needs its own libtorch-linked binary (cu128 for a Blackwell host, cu126-pt27 for a Pascal host). Without pre-flight, the first fdl @cluster <cmd> after an edit hits stale remote binaries — confusing runtime errors or worse, silent wrong behaviour.

This module runs cargo build LOCALLY on the controller, once per remote host, into per-(host, arch) target/cluster/<host>/<arch>/ directories with the right libtorch bind-mounted. The shared mount delivers the resulting binary to the remote, which execs it directly (no cargo, no rust toolchain on remote). Per-(host, arch) target dirs isolate cargo’s fingerprint cache so a libtorch swap on one host doesn’t invalidate anyone else’s incremental build – and so changing a host’s own arch: builds fresh rather than reusing a binary linked against the old libtorch (in docker mode the container’s LIBTORCH_PATH is a fixed mount point, invisible to cargo’s cache).

Convention: command name == binary name. fdl @cluster ddp-bench builds --bin ddp-bench. Features derive from the host’s libtorch .arch (cuda=12.x → --features cuda, cuda=none → no features).

Builds run in parallel across hosts (per-host target dirs ⇒ zero contention). All builds run to completion — there is no early abort: every host is joined and every failure is collected, then surfaced together so the user sees every host’s diagnostic in one place.

Structs

PerHostEnvelope

Per-host pre-flight build artifact descriptor — exactly what the launcher needs to substitute the remote dispatch with a direct binary exec. Mirrors flodl::distributed::launcher::PerHostPrebuild on the consumer side; the two structs share an on-the-wire JSON schema but are independent types because the crates can’t share declarations without a circular dep.

Constants

ENV_PREBUILD_PER_HOST

Env var carrying the per-host pre-flight build envelope (a JSON map) from fdl-cli’s prebuild phase to flodl’s launcher. The launcher reads it on the controller side just before fan-out and substitutes the direct-binary form for each host whose entry is present.

Functions

prebuild_remotes

Run pre-flight builds for every remote host in cluster. The controller itself is skipped — its build is handled by the normal dispatch path (cargo run in Docker against the local .active).

preflight_hosts

Pre-fan-out readiness probe for every host, run BEFORE any build and in BOTH prebuild and --no-prebuild modes (mount-readiness is orthogonal to binary freshness). One ssh per remote host: always verifies the shared data_path is mounted + readable; when prebuilding, also runs the controller-vs-remote ABI gate (arch / libc / pkill). The controller is checked LOCALLY (no ssh — it is the build/dispatch host, so ABI is trivially satisfied).