Expand description
Training preparation for a dial-in worker: the GPU gate, the dataset source root, libtorch, the training binary, and the node-local directories the data plane writes. Runs before this box dials in, once per attempt. Training preparation — get this box ready before it dials in.
fdl join runs this once per attempt, strictly BEFORE the tunnel and
the dial: admission starts a window deadline, so anything acquired
after it burns the deadline. Every step is idempotent, which is what
makes --persist a provisioning loop for free — a box picks up a
changed source on its next re-dial, with no reprovisioning.
Five steps, in an order that is itself load-bearing. First the cheap ones: gate on the GPU stack, put the dataset source root where the ranks will look for it, prove the node-local directories the data plane writes are writable. Then what a box may not have yet — the training source, a libtorch variant, and the binary built from both — because those take minutes, and discovering an unwritable stage directory after a cold build has wasted the build.
Failures split two ways, and the split is the point. --persist
re-dials forever with backoff, which is right for a controller that
is not up yet and wrong for a box with no GPU: without the
distinction a misprovisioned instance hot-loops instead of stopping.
Structs§
- Data
Spec - The data half of the join recipe, as resolved from flags over the
join:block. - Prepare
Spec - Everything this box has to settle before it dials.
- Prepared
- What preparation settled, for the join that follows.
- Source
Spec - The source half of the join recipe.
Enums§
- Fail
- Why preparation stopped, and whether trying again could help.
Functions§
- prepare
- Prepare this box.
notescollects everything worth telling the operator that is not a failure (a reused mount, a tmpfs stage, a nearly-full volume); the caller prints them. - print_
notes - Print what preparation found, under the name of the command that
found it. Notes are advisory by construction — every fatal condition
already returned a
Fail.