Struct ActivationFunction 

pub struct ActivationFunction { /* private fields */ }

Neural network activation layer with forward pass, derivative, in-place application, and call statistics.

Implementations

impl ActivationFunction

pub fn new(config: ActivationConfig) -> Self

Create a new activation layer with the given configuration.

pub fn forward(&mut self, input: &[f64]) -> Vec<f64>

Apply the configured activation to every element of input and return a newly allocated Vec<f64>.

For Softmax the operation is applied across the entire slice.

pub fn derivative(&self, output: &[f64]) -> Vec<f64>

Compute the element-wise derivative of the activation with respect to the pre-activation input.

Argument semantics

The output slice is interpreted differently per activation:

• Sigmoid — output should be the post-activation value σ(x). The derivative is σ(x) * (1 − σ(x)) which avoids re-computing the expensive exponential.
• Tanh — output should be the post-activation value tanh(x). The derivative is 1 − tanh(x)².
• All others — output is treated as the pre-activation value x and the derivative is computed from first principles.

pub fn apply_inplace(&mut self, data: &mut [f64])

Apply the configured activation in place, mutating data.

Statistics are updated identically to forward.

pub fn stats(&self) -> &ActivationStats

Return a reference to the accumulated statistics for this instance.

pub fn config(&self) -> &ActivationConfig

Return a reference to the current configuration.

pub fn relu(x: f64) -> f64

Rectified Linear Unit: max(0, x).

pub fn leaky_relu(x: f64, slope: f64) -> f64

Leaky ReLU: x if x >= 0, else slope * x.

pub fn elu(x: f64, alpha: f64) -> f64

Exponential Linear Unit: x if x >= 0, else alpha * (exp(x) - 1).

pub fn sigmoid(x: f64) -> f64

Logistic sigmoid: 1 / (1 + exp(-x)).

pub fn tanh_activation(x: f64) -> f64

Hyperbolic tangent activation.

pub fn softmax(input: &[f64]) -> Vec<f64>

Numerically stable softmax over the entire input slice.

Subtracts max(input) before exponentiating to prevent overflow.

pub fn gelu(x: f64) -> f64

GELU activation (tanh approximation).

0.5 * x * (1 + tanh(sqrt(2/π) * (x + 0.044715 * x³)))

pub fn swish(x: f64) -> f64

Swish / SiLU: x * sigmoid(x).

pub fn mish(x: f64) -> f64

Mish: x * tanh(softplus(x)) where softplus(x) = ln(1 + exp(x)).

Uses the numerically stable form ln(1 + exp(x)): for large positive x, softplus(x) ≈ x; for large negative x, softplus(x) ≈ exp(x).

pub fn hard_swish(x: f64) -> f64

Hard Swish: x * relu6(x + 3) / 6 where relu6(t) = min(max(t, 0), 6).