Struct MoEExpert 

pub struct MoEExpert {
    pub gate_proj: Tensor,
    pub up_proj: Tensor,
    pub down_proj: Tensor,
}

Helper struct for a standard MoE expert (simple MLP)

Fields

gate_proj: Tensor

Gate projection: hidden_size -> intermediate_size

up_proj: Tensor

Up projection: hidden_size -> intermediate_size

down_proj: Tensor

Down projection: intermediate_size -> hidden_size

Implementations

impl MoEExpert

pub fn new(gate_proj: Tensor, up_proj: Tensor, down_proj: Tensor) -> Self

pub fn forward(&self, hidden_states: &Tensor) -> Result<Tensor>

Forward pass: SwiGLU activation

Trait Implementations

impl Debug for MoEExpert

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.