Struct MoeSlotCache 

pub struct MoeSlotCache {
    pub hits: u64,
    pub misses: u64,
    pub staged_bytes: u64,
    /* private fields */
}

SLRU GPU expert-residency cache. Slots remain fixed-address for the cache lifetime. Uniform models use one class; mixed-layout models may preallocate several exact-capacity classes.

Fields

hits: u64

misses: u64

staged_bytes: u64

Implementations

impl MoeSlotCache

pub fn new(e: &Engine, max_block_bytes: usize) -> Result<Self, Box<dyn Error>>

Build the cache sizing N from free VRAM (MOE-SLRU-PLAN §B.4): probe free VRAM AFTER residents are loaded; N is shared across ALL layers so it must hold the WHOLE-MODEL hot set, not one layer’s. The 35B-A3B keeps its 256 experts HOST-resident, so the GPU has ~20+ GB free at decode — empirically a 256-slot cache thrashes (~2-7% hit) while a few-thousand-slot cache reaches ~85%+ steady-state. So the DEFAULT auto-sizes N to fill MEMRA_MOE_VRAM_FRAC (default 0.85) of free VRAM, clamped to [256, ~hot-set]. MEMRA_MOE_SLOTS forces an exact N.

pub fn n_slots(&self) -> usize

pub fn is_frozen(&self) -> bool

pub fn freeze(&mut self)

pub fn max_block_bytes(&self) -> usize

pub fn resident(&self, id: BlockId) -> Option<usize>

O(1) residency check (the ktransformers generate_gpu_experts_masks analog).

pub fn dispatch( &mut self, id: BlockId, host_bytes: &[u8], e: &Engine, ) -> Result<DispatchSlot, Box<dyn Error>>

The dispatch decision for one (BlockId, host_bytes). Returns where the block landed; resolve the device buffer with buf(). On the bit-identity-critical path the buffer holds EXACTLY host_bytes either way (a HIT skipped the copy; the prior stage wrote the same bytes).

Policy (MOE-SLRU-PLAN §B.2, first-miss admit since 2026-07-06):

• HIT (table[id] = s): promote, return s. ZERO PCIe.
• MISS: admit (stage into a retained slot, evicting an SLRU victim when full).

pub fn prefetch( &mut self, id: BlockId, host_bytes: &[u8], keep: &[BlockId], e: &Engine, ) -> Result<bool, Box<dyn Error>>

Deterministically stage a known-future block on the copy stream. The slot is reserved but is not considered resident until dispatch inserts a compute-stream wait for the returned copy event. Before overwriting a reused slot, the copy stream waits for all compute work already queued at this call site; the caller issues prefetch before the current expert’s kernels, so the transfer can overlap those kernels without racing any earlier consumer of the victim.

keep is the current expert’s gate/up/down ids. If no safe victim exists, return false and let the normal synchronous miss path handle the block.

pub fn force_admit( &mut self, id: BlockId, host_bytes: &[u8], e: &Engine, ) -> Result<usize, Box<dyn Error>>

Pre-warm: force-admit a block (used by the §D.2 bit-identity gate to make all blocks resident).

pub fn export_residency(&self) -> Vec<(u16, u8, u16)>

STAGE 3 one-shot PREWARM: force-admit every block of layer while FREE slots can hold it (never evicts — a spill rig whose cache can’t fit the layer just skips; organic residency still applies). Runs at most once per layer (success or not). The H2D copies are the SAME stage_expert bytes the miss path would issue — bit-identity unchanged; this only front-loads them so the device-dispatch fast path fires from token 0 instead of after the SLRU fill. Frozen residency as (layer, proj, ex) triples in slot order, for the freeze-profile sidecar. Slot order keeps the restage admit sequence close to the original placement.

pub fn restage_block( &mut self, id: BlockId, m: &MoeWeights, e: &Engine, ) -> Result<bool, Box<dyn Error>>

Admit one specific block from a saved freeze profile, reading through the layer’s established expert source (the same recipe as prewarm_layer, but id-targeted so a persisted residency set restages without a profiling warmup). Returns false for ids that no longer resolve (changed plan, pruned expert) — the caller counts and reports.

pub fn prewarm_layer( &mut self, layer: u16, m: &MoeWeights, e: &Engine, ) -> Result<(), Box<dyn Error>>

pub fn layer_dev_row( &mut self, layer: u16, n_expert: usize, e: &Engine, ) -> Result<Option<&CudaSlice<u64>>, Box<dyn Error>>

STAGE 3: device pointer row for a FULLY-RESIDENT layer. Returns the [3, n_expert] u64 slot base-address table (proj-major: gate row, up row, down row) if EVERY block of layer is cache-resident, else None (caller falls back to host routing). The row is built+uploaded on first full residency and reused until an eviction touches the layer. n_expert is the layer’s expert count (the full-residency threshold is 3*n_expert blocks).

pub fn buf(&self, d: DispatchSlot) -> &CudaSlice<u8>

Resolve a DispatchSlot to the device buffer to feed qmatvec_view.

pub fn slot(&self, s: usize) -> &CudaSlice<u8>

Read-only access to a slot’s device buffer (the qmatvec_view source on a HIT).

pub fn hit_rate(&self) -> f64

Hit rate over this cache’s lifetime (for the §D.4 print).

pub fn reset_counters(&mut self)

Reset the per-window perf counters (lets the run print steady-state vs warmup separately).

Trait Implementations

impl Drop for MoeSlotCache

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.