pub struct MoeWeights {Show 15 fields
pub gate_inp: GpuTensor,
pub gate_inp_shexp: Option<GpuTensor>,
pub exp_probs_b: Option<Vec<f32>>,
pub exp_probs_b_dev: CudaSlice<f32>,
pub active_experts: Option<Vec<bool>>,
pub active_experts_dev: CudaSlice<u8>,
pub gate_exps: HostExps,
pub up_exps: HostExps,
pub down_exps: HostExps,
pub gate_shexp: Option<GpuTensor>,
pub up_shexp: Option<GpuTensor>,
pub down_shexp: Option<GpuTensor>,
pub dev_exps: Option<DevExps>,
pub dev_macros: CudaSlice<f32>,
pub has_macros: bool,
}Expand description
MoE weights for one layer. Router + shared expert stay GPU-RESIDENT (tiny); the routed experts stay HOST-RESIDENT (HostExps) and are staged per-token (EDGE-1).
The shared-expert fields are Option: qwen35moe carries a shared expert, but OLMoE (and most
vanilla MoE) have none (shared_expert_intermediate_size absent) — those layers load_opt the
shexp tensors to None (ST-MOE-PLAN §1.3, §3.2). When None the shared-expert branch is skipped.
Fields§
§gate_inp: GpuTensor§gate_inp_shexp: Option<GpuTensor>§exp_probs_b: Option<Vec<f32>>DeepSeek-V3/MiniMax-M3 e_score_correction_bias [n_expert]: added to the sigmoid scores
for expert SELECTION only; the routing weights use the un-biased scores. The host row is
the rollback oracle; the device row is zero-filled when the checkpoint carries no bias.
exp_probs_b_dev: CudaSlice<f32>§active_experts: Option<Vec<bool>>Original-width router mask for physically pruned expert overlays. Inactive ids never enter top-k, so their absent weight files cannot be dispatched. The device row is all ones when no overlay mask exists.
active_experts_dev: CudaSlice<u8>§gate_exps: HostExps§up_exps: HostExps§down_exps: HostExps§gate_shexp: Option<GpuTensor>§up_shexp: Option<GpuTensor>§down_shexp: Option<GpuTensor>§dev_exps: Option<DevExps>FITS-VRAM RESIDENT EXPERTS (2026-07-06): when the WHOLE model’s expert bytes fit the VRAM
budget, each (proj) slab is uploaded once as a contiguous device buffer and the fused
_dev kernels take base+ex*stride pointers — no SLRU, no dispatch, no residency checks
(llama’s full-offload regime; measured 169.55 vs memra’s cache path 28.5 on the local 35B).
None => the SLRU host-expert machinery (the spill regime, where it WINS vs llama’s
CPU-offload degradation). Decided at load in load_ffn (MEMRA_MOE_RESIDENT=0 forces off).
dev_macros: CudaSlice<f32>Per-expert post-matmul macro-scales on DEVICE: [3n_expert] f32 in (gate, up, down)
order — all 1.0 unless the checkpoint carries compressed-tensors NVFP4 global scales
(unsloth qwen3.6 class). The _dev gate_up epilogues multiply unconditionally (x1.0f
is bit-exact — zero change for macro-free artifacts); the down fold is one
moe_w_scale_by_expert launch gated on has_macros.
has_macros: bool