Module gpu 

Facade for GPU backends: a single call entry point for qtensor/pipeline/ linear_core. Job types and the threshold are canonical HERE; behind the facade dispatch goes to a platform backend:

• gpu_metal (Apple Silicon, unified memory + no-copy buffers);
• gpu_wgpu (C1: Vulkan/DX12/Metal — NVIDIA/Radeon/Intel/Apple, weights resident in VRAM), available under --features gpu.

Runtime selection via CMF_GPU: 1 — native Metal (macOS) or wgpu (other OSes); wgpu — force wgpu (including for the local Metal-via-wgpu parity test). Any backend refusal — false and the honest CPU path, no partial results.

Structs

BatchJob

A single independent batch matvec (GDN projections of one input).

MoeJob

A single MoE-FFN job (an expert with its own weight), executed in one submission: (rows, cols, idx, row_scale) for gate/up/down + prescaled inputs + the down θ-field + the blending weight.

Constants

GPU_MIN_ROWS

Default row threshold: the GPU takes only larger matrices (lm_head class). Below it, the dispatch/readback cost does not pay off on unified memory.

Functions

enabled

GPU enabled and initialized on the selected backend?

enabled_here

GPU allowed FOR THE CURRENT LAYER: backend is initialized AND the layer falls within CMF_GPU_LAYERS (GPU/CPU layer-split). Op gates call this.

matvec_batch

Independent matvecs of one input in a single submission (GDN projections).

min_rows

Effective threshold: CMF_GPU_MIN_ROWS overrides the default — on a discrete card it is worth lowering it (VRAM bandwidth pays off even for FFN), on unified memory — raising it. A «squeeze out the maximum» tuning on the server.

moe_block

A layer’s MoE-FFN in one submission (amortizing the dispatch cost).

q8_matmat

GEMM of a prefill batch: pre — prescaled inputs row-major [b, cols], out — row-major [b, rows].

q8_matvec_range

q8_row/q8_2f matvec, rows [row0, row0+rows). xs — prescaled by the θ-field.

set_layer

Pipeline: mark the current layer (or −1 outside layers) for layer-split.