Expand description
fdl nccl build – compile libnccl from NVIDIA’s NCCL source.
Why this exists: libtorch wheels statically link NCCL into
libtorch_cuda.so. On heterogeneous-arch rigs (Pascal sm_61 + Blackwell
sm_120, etc.), that static NCCL can corrupt the shared CUmodule loader
state during cross-host bootstrap, which surfaces as
cudaErrorNoKernelImageForDevice on otherwise-working kernels. Building
libnccl as a standalone .so and LD_PRELOAD-ing it gives NCCL its own
CUmodule scope and sidesteps the issue.
Output layout (mirrors fdl libtorch build):
libtorch/nccl/builds/v<version>-<archs>/
include/nccl.h
lib/libnccl.so -> libnccl.so.2
lib/libnccl.so.2 -> libnccl.so.<X.Y.Z>
lib/libnccl.so.<X.Y.Z>
lib/libnccl_static.a
lib/pkgconfig/