Enum ActivationFunction 

pub enum ActivationFunction {
    ReLU,
    GELU,
    Swish,
    GLU,
    GeGLU,
}

Activation functions for feed-forward networks

Variants

ReLU

ReLU activation

GELU

GELU activation

Swish

Swish/SiLU activation

GLU

GLU (Gated Linear Unit): splits the input and gates one half by the sigmoid of the other, halving the width.

GeGLU

GeGLU (GELU variant of GLU): splits the input and gates one half by the GELU of the other, halving the width.

Implementations

impl ActivationFunction

pub fn is_gated(self) -> bool

Whether this activation halves the feature width (gated variants do).

pub fn output_width(self, input_width: usize) -> usize

Width produced by this activation given input_width.

Trait Implementations

impl Clone for ActivationFunction

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for ActivationFunction

impl Debug for ActivationFunction

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ActivationFunction

fn default() -> Self

Returns the “default value” for a type.

impl Eq for ActivationFunction

impl PartialEq for ActivationFunction

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for ActivationFunction