Expand description
The CPU backend: graphs lowered to a list of steps over one arena, run on a persistent pool.
Lowering resolves every value to an arena offset, every weight to its converted tensor and every RoPE base to its table, and checks every shape, so running a step is a match and a call. The arena, the per worker attention scratch and the per batch index tables are sized for the bucket when the plan is built, and nothing on the run path allocates.
Each step runs on the rows the batch has rather than the bucket’s padded count. Padding only
matters to a backend that captures a fixed shape, and on the CPU it would be wasted work. The
kernels are the reference ones, so a plan gives the same bits as Compat.
A backend made with_int8 runs the GEMMs over token rows in INT8
instead, which is every GEMM of the encoder and the decision head. The scorer and the act head
run on a row per option or per question and stay in FP32, as spec/10-cpu.md has it.
Structs§
- CpuBackend
- The CPU backend, which owns its threads.
- CpuPlan
- A graph lowered for one bucket.
- Dump
- What one step wrote in one run, from
CpuPlan::dumps. - Tensor
- A weight converted to f32.
- Weights
- Every weight of a checkpoint, shared by all the plans built from it.