Struct Cuda 

pub struct Cuda;

The CUDA backend (ferrox-cuda).

Trait Implementations

impl BackendCaps for Cuda

const GEMM_FALLBACK: &'static str = "CUDA per-position matvec"

apply_batch_with_acts decomposes a CUDA prefill into one matvec per position for every kind off Cuda::gemm_supported.

fn matvec_kernel(kind: QuantKind) -> Option<&'static str>

The decode path, and the arm that has actually run on a GPU – for six of its nine kinds.

DERIVED from ferrox_cuda::matvec_kinds::KINDS, not restated. That table is compiled on every build (it is CUDA C text and three strings per row; nothing in it needs cudarc), and ferrox-cuda is an unconditional dependency for exactly this reason. The set used to be written out here as a matches! and checked against the kernel table by a test that only ran under --features cuda; over-claiming there sends a decode to an NVRTC module that does not exist, and under-claiming leaves a kernel nothing ever calls. Both have happened.

The name is returned only to share BackendCaps::matvec_kernel’s shape with Metal; nothing on the CUDA path reads it, because ferrox-cuda’s launchers are named functions rather than entries in a string-keyed table.

fn gemm_supported(kind: QuantKind) -> bool

The mul_mm prefill path.

DERIVED from ferrox_cuda::mul_mm::KINDS, for the same reason and by the same mechanism as Cuda::matvec_kernel above. It equals that set, and the_matvec_table_and_the_mul_mm_table_name_the_same_kinds in ferrox-cuda is what keeps it equal – a kind with one kernel and not the other splits a forward pass across two devices.

It did not until 2026-09-04: only Q8_0 and Q4_0 had a matrix-matrix product, so a K-quant prefill decomposed into one matvec launch per position. Measured on a GTX 1080, that cost Llama-3.2-3B Q4_K_M 4.88 tok/s of pp512 against llama.cpp’s 1586.80, a 325x gap on the most common quantization in circulation (#131). IQ4_XS is still absent: it is a codebook lookup rather than an affine dequant.

Q5_0 joined on 2026-09-05, matvec and GEMM together, because a_cuda_kind_with_a_matvec_also_has_a_gemm makes half of it fail the suite – and because half of it is the shape that cost Metal a Q5_0 decode path: GPU prefill with every decode step on the host.

IQ4_NL, IQ4_XS and MXFP4 joined on 2026-09-09, matvec and GEMM together. They are CODEBOOK formats: the stored 4-bit code is an index into a sixteen-entry table, not a magnitude, so the kernel carries that table in __constant__ memory. gpt-oss ships MXFP4 and no GPU backend had it at all, so every expert decoded on the host with the device idle.

UNRUN ON HARDWARE. The kernel is checked against a scalar twin and by executing the emitted CUDA C on the host, and has never executed on a GPU. See crates/ferrox-cuda/src/mul_mm.rs.

const ID: Backend = Backend::Cuda

How crate::kernel_registry reports this backend. Dispatch and observability read the same constant, so a backend cannot be dispatched to under one name and reported under another.

const NAME: &'static str = "CUDA"

Human-readable name, for the one message a dispatch failure prints.