Module placement 

Which MoE layers decode on the CPU.

The q* split (crate::qstar) decides how a step divides its misses. This module decides something coarser and more permanent: which whole layers never use the GPU expert path at all, because their weights could not be page-locked for DMA in the first place.

That is a host-memory question, not a bandwidth one. Pinning host memory so the GPU can DMA from it is a scarce, OS-wide resource; on some systems it is capped near half of RAM. A model whose expert banks exceed that cap cannot have every layer pinned, so some layers must be served the other way – read as ordinary pageable memory by CPU threads.

Head and tail, not a contiguous block

auto_cpu_layers picks from both ends. Expert-cache miss rates across a transformer’s layers are U-shaped: the first and last layers route more diffusely (their residuals carry the least task-specific structure), so they hit least and benefit least from GPU residency. Handing the middle layers to the GPU cache and the ends to the CPU therefore costs the least throughput per byte of pinning saved. A contiguous prefix would give up the middle layers, which are exactly the ones the cache serves well.

Ported 1:1 from FreeToken’s engine/engine.py (_parse_cpu_layers_spec, _auto_cpu_layers) (Apache-2.0); see docs/THIRD_PARTY_NOTICES.md.

Enums

CpuLayerSpecError

A CPU-layer spec that does not name a valid set of layers.

Functions

auto_cpu_layers

The layers to serve on the CPU when the expert banks do not fit the host’s page-locking budget.

parse_cpu_layers_spec

Read a --moe-cpu-layers spec.

round_half_even

Python’s round: halves go to the nearest even integer.

strided_layers

count layers spread evenly across num_layers.