Module feed_forward 

Feedforward network layer for transformer blocks.

This module implements the position-wise feedforward sub-layer that appears in every transformer encoder/decoder block (Vaswani et al. 2017). The standard two-layer structure is:

output = Linear2( Activation( Linear1( input ) ) )

where Linear1 projects from input_dim → hidden_dim (typically 4 × input_dim) and Linear2 projects back to output_dim.

Supported activations

Variant	Description
ReLU	max(0, x)
GELU	Gaussian Error Linear Unit (tanh approximation)
SiLU	x · σ(x) (also known as Swish)
Linear	Identity — no activation applied

Weight initialisation

Weights are initialised with He (Kaiming) normal initialisation using a custom xorshift64 PRNG with Box-Muller transform — zero external crate dependency.

Example

use ipfrs_tensorlogic::{FeedForwardConfig, FeedForwardActivation, FeedForwardNetwork};

let cfg = FeedForwardConfig {
    input_dim: 8,
    hidden_dim: 32,
    output_dim: 8,
    activation: FeedForwardActivation::GELU,
    use_bias: true,
    dropout_rate: 0.1,
};

let mut net = FeedForwardNetwork::new(cfg, 42);

let token = vec![1.0_f64; 8];
let out = net.forward(&token);
assert_eq!(out.len(), 8);

Structs

FFLayer

A single affine (linear) layer: weight matrix and optional bias.

FFStats

Lightweight counters accumulated across FeedForwardNetwork::forward calls.

FeedForwardConfig

Configuration for FeedForwardNetwork.

FeedForwardNetwork

Two-layer feedforward network suitable for use as the FFN sub-layer of a transformer block.

Enums

FeedForwardActivation

Activation function applied between the two linear projections.