Struct MoeGateParams 

pub struct MoeGateParams {
    pub hidden_dim: usize,
    pub n_experts: usize,
    pub top_k: usize,
    pub seq_len: usize,
    pub rms_eps: f32,
}

Parameters for MoE gate routing.

Fields

hidden_dim: usize

Hidden state dimension (e.g. 2816 for Gemma 4).

n_experts: usize

Total number of experts (e.g. 128 for Gemma 4).

top_k: usize

Number of experts to select (e.g. 8 for Gemma 4).

seq_len: usize

Number of tokens in the sequence (seq_len >= 1).

rms_eps: f32

RMS norm epsilon (e.g. 1e-6).