Struct FusedGateUpOp 

pub struct FusedGateUpOp {
    pub hidden_size: usize,
    pub intermediate_size: usize,
}

Fused Gate+Up FFN projection with SiLU activation.

Computes gate and up projections in a single pass:

• gate = x * W_gate
• up = x * W_up
• output = SiLU(gate) * up (SwiGLU activation)

Performance Impact

Fusing 2 separate matmuls + activation provides:

• 2x reduction in kernel launches (GPU)
• Fused SiLU avoids intermediate memory traffic
• Expected speedup: 1.5-2x for decode phase

Five-Whys Root Cause (PMAT-PERF-009)

Why 1: Why is FFN phase slow?
→ 3 kernel launches: gate_proj, up_proj, SiLU activation

Why 2: Why separate kernels?
→ Traditional implementation pattern from training frameworks

Why 3: Why does this matter for inference?
→ Inference is memory-bound; kernel launch overhead dominates

Why 4: Why not fuse earlier?
→ Requires custom kernel development

Why 5: ROOT CAUSE
→ SwiGLU requires gate*up pattern that naturally fuses
→ FIX: Fuse gate+up+SiLU into single operation

Fields

hidden_size: usize

Hidden dimension size

intermediate_size: usize

Intermediate FFN dimension

Implementations

impl FusedGateUpOp

pub fn new(hidden_size: usize, intermediate_size: usize) -> Self

Create a new fused gate+up operation.

Arguments

• hidden_size - Hidden dimension (e.g., 3584 for Qwen 3B)
• intermediate_size - FFN intermediate dimension (e.g., 18944)

pub fn silu(x: f32) -> f32

SiLU activation: x * sigmoid(x)

ONE PATH: Delegates to crate::activations::silu_scalar (UCBD §4).

Trait Implementations

impl Clone for FusedGateUpOp

fn clone(&self) -> FusedGateUpOp

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl ComputeOp for FusedGateUpOp

type Input = (Vec<f32>, FusedGateUpWeights)

Input type for this operation

type Output = Vec<f32>

Output type for this operation

fn name(&self) -> &'static str

Operation name for identification

fn execute( &self, input: Self::Input, _backend: Backend, ) -> Result<Self::Output, TruenoError>

Execute the operation on the given backend

fn tokens(&self, _input: &Self::Input) -> usize

Number of tokens this operation processes (for budget calculation)

fn clone_input(&self, input: &Self::Input) -> Option<Self::Input>

where Self::Input: Clone,

Clone the input for verification (if needed)

impl Debug for FusedGateUpOp

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.