Skip to main content

Module prepare

Module prepare 

Source
Expand description

Training preparation for a dial-in worker: the GPU gate, the dataset source root, libtorch, the training binary, and the node-local directories the data plane writes. Runs before this box dials in, once per attempt. Training preparation — get this box ready before it dials in.

fdl join runs this once per attempt, strictly BEFORE the tunnel and the dial: admission starts a window deadline, so anything acquired after it burns the deadline. Every step is idempotent, which is what makes --persist a provisioning loop for free — a box picks up a changed source on its next re-dial, with no reprovisioning.

Five steps, in an order that is itself load-bearing. First the cheap ones: gate on the GPU stack, put the dataset source root where the ranks will look for it, prove the node-local directories the data plane writes are writable. Then what a box may not have yet — the training source, a libtorch variant, and the binary built from both — because those take minutes, and discovering an unwritable stage directory after a cold build has wasted the build.

Failures split two ways, and the split is the point. --persist re-dials forever with backoff, which is right for a controller that is not up yet and wrong for a box with no GPU: without the distinction a misprovisioned instance hot-loops instead of stopping.

Structs§

DataSpec
The data half of the join recipe, as resolved from flags over the join: block.
PrepareSpec
Everything this box has to settle before it dials.
Prepared
What preparation settled, for the join that follows.
SourceSpec
The source half of the join recipe.

Enums§

Fail
Why preparation stopped, and whether trying again could help.

Functions§

prepare
Prepare this box. notes collects everything worth telling the operator that is not a failure (a reused mount, a tmpfs stage, a nearly-full volume); the caller prints them.
print_notes
Print what preparation found, under the name of the command that found it. Notes are advisory by construction — every fatal condition already returned a Fail.