pub fn sparse_ffn_forward(
hidden_states: &[f32],
gate_proj_full: &[f32],
up_proj_full: &[f32],
down_proj_full: &[f32],
hidden_size: usize,
intermediate_size: usize,
active_indices: &[u16],
pool: Option<&Pool>,
) -> Vec<f32>Expand description
Sparse SwiGLU FFN: compute only active_indices neurons.
out = down_projᵀ[·, active] · (silu(gate_proj[active, ·]·h) ⊙ up_proj[active, ·]·h)
With a full index list this is bit-identical to the dense path — masking is an execution schedule, not an approximation.