Skip to main content

Module activations

Module activations 

Source
Expand description

激活函数实现

Functions§

gelu
GELU 激活函数:f(x) = x * Φ(x) ≈ x * 0.5 * (1 + tanh(√(2/π) * (x + 0.044715 * x³)))
leaky_relu
Leaky ReLU 激活函数:f(x) = x if x > 0 else alpha * x
relu
ReLU 激活函数:f(x) = max(0, x)
sigmoid
Sigmoid 激活函数:f(x) = 1 / (1 + exp(-x))
softmax
Softmax 函数(沿指定轴)
tanh
Tanh 激活函数:f(x) = tanh(x)