Expand description
CUDA backend (Linux / Windows, NVIDIA): the driver API and NVRTC are loaded at run time with
dlsym, so the binary has no CUDA dependency and runs unchanged on machines without a GPU.
The GEMM kernels are compiled from source by NVRTC for the detected compute capability.
Only prefill-sized matmuls are offloaded (see wants): weights are copied to device memory
the first time a layer is used (skipped when VRAM is short, that layer then stays on the CPU),
activations are copied up and results down per batch. Discrete GPUs pay for those transfers,
so the threshold is higher than Metal’s. Decode stays on the CPU.
This backend is compile-checked but has not been run on NVIDIA hardware yet.