Struct TransformerEncoderLayer 

pub struct TransformerEncoderLayer<T>
where
    T: Float + SimdUnifiedOps,
{
    pub attention: MultiHeadAttention<T>,
    pub feed_forward: PositionwiseFeedForward<T>,
    pub norm1_gamma: Array1<T>,
    pub norm1_beta: Array1<T>,
    pub norm2_gamma: Array1<T>,
    pub norm2_beta: Array1<T>,
    pub norm_eps: T,
}

Transformer Encoder Layer

A single layer of the transformer encoder consisting of:

1. Multi-head self-attention
2. Add & Norm (residual connection + layer normalization)
3. Position-wise feed-forward network
4. Add & Norm (residual connection + layer normalization)

Architecture

Input
  │
  ├───> Multi-Head Attention ───> Add & Norm ───┐
  │                                              │
  └──────────────────────────────────────────────┘
  │
  ├───> Feed-Forward Network ───> Add & Norm ───┐
  │                                              │
  └──────────────────────────────────────────────┘
  │
Output

Fields

attention: MultiHeadAttention<T>

Multi-head attention

feed_forward: PositionwiseFeedForward<T>

Position-wise feed-forward network

norm1_gamma: Array1<T>

Layer normalization parameters for attention (gamma)

norm1_beta: Array1<T>

Layer normalization parameters for attention (beta)

norm2_gamma: Array1<T>

Layer normalization parameters for feed-forward (gamma)

norm2_beta: Array1<T>

Layer normalization parameters for feed-forward (beta)

norm_eps: T

Layer normalization epsilon

Implementations

impl<T> TransformerEncoderLayer<T>

where T: Float + SimdUnifiedOps + ScalarOperand,

pub fn new( d_model: usize, num_heads: usize, d_ff: usize, dropout: f64, ) -> TransformerResult<Self>

Creates a new transformer encoder layer

Arguments

• d_model - Model dimension
• num_heads - Number of attention heads
• d_ff - Feed-forward dimension
• dropout - Dropout probability

pub fn forward( &self, x: &ArrayView2<'_, T>, mask: Option<&ArrayView2<'_, T>>, training: bool, ) -> TransformerResult<Array2<T>>

Forward pass through encoder layer

Arguments

• x - Input tensor (seq_len, d_model)
• mask - Optional attention mask
• training - Whether in training mode

Trait Implementations

impl<T> Clone for TransformerEncoderLayer<T>

where T: Float + SimdUnifiedOps + Clone,

fn clone(&self) -> TransformerEncoderLayer<T>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T> Debug for TransformerEncoderLayer<T>

where T: Float + SimdUnifiedOps + Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.