Struct EncoderFeedForward 

pub struct EncoderFeedForward {
    pub w_up: Tensor,
    pub b_up: Tensor,
    pub w_down: Tensor,
    pub b_down: Tensor,
    /* private fields */
}

Encoder feed-forward network with GELU activation (BERT/RoBERTa/CodeBERT).

Unlike decoder SwiGLU (3 projections), encoder FFN uses 2 projections: FFN(x) = W_down * GELU(W_up * x + b_up) + b_down

Contract (ENC-004)

• Uses GELU activation (not SiLU/SwiGLU)
• 2 projections (up + down), not 3
• Supports bias terms (BERT convention)

Fields

w_up: Tensor

Up projection (hidden → intermediate)

b_up: Tensor

Up projection bias

w_down: Tensor

Down projection (intermediate → hidden)

b_down: Tensor

Down projection bias

Implementations

impl EncoderFeedForward

pub fn new(config: &TransformerConfig) -> Self

Create new encoder FFN with random normal initialization (C-INIT-001).

pub fn from_params( config: &TransformerConfig, params: &HashMap<String, Tensor>, prefix: &str, ) -> Option<Self>

Create from pre-trained parameters (BERT/RoBERTa weight names)

Expected keys:

• {prefix}.intermediate.dense.weight (hidden → intermediate)
• {prefix}.intermediate.dense.bias
• {prefix}.output.dense.weight (intermediate → hidden)
• {prefix}.output.dense.bias

pub fn forward(&self, x: &Tensor, seq_len: usize) -> Tensor

Forward pass: FFN(x) = W_down * GELU(W_up * x + b_up) + b_down

pub fn parameters(&self) -> Vec<&Tensor>

Get all parameters