Expand description
EDGE-1 §B: SLRU GPU expert-residency cache (MOE-SLRU-PLAN §B).
Stage-1 moe_ffn re-stages EVERY routed expert EVERY token over PCIe into one scratch slot.
The same ~15-20% of experts recur (the “hot expert” mass), so an SLRU residency cache makes
the steady-state re-stage count -> ~0. The cache holds N fixed-address GPU slots (never
re-allocated, never fragmented), a BlockId -> slot residency table, an SLRU eviction policy
(probation + protected segments; the second-miss “ghost” admission filter was measured a net
loss in both regimes and removed 2026-07-08 — first-miss admit is the policy) so a one-off cold
expert can never evict a genuinely hot one.
THE bit-identity property (MOE-SLRU-PLAN §B.3): a cache HIT and a MISS feed qmatvec_view the
same block bytes — the only difference is whether the memcpy_htod ran. So the cache-hit
weight path is byte-for-byte identical to stage-every-token. TWO gates pin this, and they
cover different classes: src/bin/kernel_check.rs d2-cache-bit-identity pins ONE block of a
real GGUF checkpoint (dtypes IQ3_S | IQ4_XS | Q6_K | Q8_0 — everything else, NVFP4 included,
takes its cells.skip arm), and tests/glm5_moe_residency_gpu.rs pins it END TO END on a
glm5_next fixture for the safetensors NVFP4 macro-carrying class, in CI, without a checkpoint.
QUANT-FORMAT AGNOSTIC, and that is load-bearing for the safetensors NVFP4 class (glm5_next /
GLM-5.3-Flash, Step-3.7-Flash-NVFP4, the unsloth 35B-A3B ST class). A slot is max_block_bytes
of opaque bytes keyed by BlockId; nothing here reads a qtype, a block stride, or a scale.
The loader has already repacked modelopt NVFP4 (weight + per-16 weight_scale) into ONE
contiguous per-expert block in memra’s internal block_nvfp4 layout
(nvfp4_repack::repack_modelopt_to_gguf, row_bytes = in_f / 64 * 36), so an NVFP4 block is
staged and hit exactly like a k-quant GGUF block. The per-expert weight_scale_2 MACRO scale
is NOT in the block — it rides HostExps::macros and is folded post-matmul by the MoE forward
— so residency can never move it, and hit/miss stay bit-identical for macro-carrying banks too.
Gated behind MEMRA_MOE_CACHE, default ON since 2026-07-08 (docs/FLAGS.md: the row is
spelled MEMRA_MOE_CACHE=0 = stage-every-token, i.e. =0 is the ROLLBACK, not the default).
Engine::moe_cache_enabled() is var("MEMRA_MOE_CACHE") != Ok("0"). This line previously read
“default off => current stage-every-token behavior”, which was the pre-2026-07-08 state and had
been stale for seven weeks; it was corrected in the glm53-flash bring-up lane (2026-08-28) after
it was quoted as the live default in a placement plan.
Structs§
- BlockId
- Residency key: expert
exof layerlayerprojectionprojis a distinct block. - MoeSlot
Cache - SLRU GPU expert-residency cache. Slots remain fixed-address for the cache lifetime. Uniform models use one class; mixed-layout models may preallocate several exact-capacity classes.
Enums§
- Dispatch
Slot - Where a dispatched block landed (always a retained resident slot since the first-miss-admit policy, 2026-07-08 — the transient staging tier went with the ghost filter).