Skip to main content

sparse_ffn_forward

Function sparse_ffn_forward 

Source
pub fn sparse_ffn_forward(
    hidden_states: &[f32],
    gate_proj_full: &[f32],
    up_proj_full: &[f32],
    down_proj_full: &[f32],
    hidden_size: usize,
    intermediate_size: usize,
    active_indices: &[u16],
    pool: Option<&Pool>,
) -> Vec<f32>
Expand description

Sparse SwiGLU FFN: compute only active_indices neurons.

out = down_projᵀ[·, active] · (silu(gate_proj[active, ·]·h) ⊙ up_proj[active, ·]·h)

With a full index list this is bit-identical to the dense path — masking is an execution schedule, not an approximation.