Struct PointwiseFeedForward

pub struct PointwiseFeedForward { /* private fields */ }

Point-wise feed-forward layer (Vaswani et al., 2017).

This layer is applied pointwise, meaning that the same transformation is applied to each sequence element. This transformation is:

g(xW_1 + b_1)W_2 + b_2

W_1 and b_1 transform the input to an intermediate width, g is a non-linear activation function and W_2 and b_2 transform the output of the activation back to the input width.

Gated Linear Units (Dauphin et al., 2016; Shazeer, 2020) are also supported. Gating applies the following transformation:

(g(xW_g + b_g) * (xW_1 + b_1))W_2 + b_2

W_g and b_g are the affine transformation for the gate.

• Vaswani et al., 2017: https://arxiv.org/abs/1706.03762
• Dauphin et al., 2016: https://arxiv.org/abs/1612.08083
• Shazeer, 2020: https://arxiv.org/abs/2002.05202

Trait Implementations

impl ModuleT for PointwiseFeedForward

fn forward_t(&self, xs: &Tensor, train: bool) -> Result<Tensor, Error>