Crate kime_cuda 

The NVIDIA backend, from spec/08-cuda.md.

This first version runs the compat graph with FP16 weights and activations and FP32 accumulation. The residual stream, the head’s inputs and everything after the scorer stay in FP32. The GEMMs go through cuBLASLt, and the other ops are our own kernels in kernels/compat.cu, compiled with NVRTC for the device when the backend starts, so the build needs no CUDA toolkit and the binary runs on any machine with a driver.

Every launch is sized for the plan’s bucket and every kernel reads the batch’s real counts from a device buffer, so a run is a fixed sequence of launches that a CUDA graph can capture.

One of the six crates where unsafe is allowed. Every block carries a // SAFETY: comment that names the invariant which makes it sound.

Modules

energy

Energy per decision from NVML’s total energy counter, as spec/13-benchmarks.md asks.

Structs

CudaBackend

One GPU, with its stream, kernels and cuBLASLt handle.

CudaPlan

A graph lowered for one bucket on one GPU.

Weights

Every weight of a checkpoint on one GPU.

Enums

Precision

How much of the graph runs in FP16.

Functions

executor

The compat graph of model on GPU ordinal at precision, with the default bucket table.