Enum Activation

pub enum Activation {
    gelu,
    relu,
    swish,
    mish,
    gelu_new,
    tanh,
    identity,
}

Activation function used in the attention layer and masked language model head

Variants

gelu

Gaussian Error Linear Unit (Hendrycks et al., 2016,)

relu

Rectified Linear Unit

swish

Swish (Ramachandran, 2017)

mish

Mish (Misra, 2019)

gelu_new

Gaussian Error Linear Unit (New) (Hendrycks et al., 2016,)

tanh

Tanh

identity

Identity

Implementations

impl Activation

pub fn get_function(&self) -> TensorFunction

Trait Implementations

impl Clone for Activation

fn clone(&self) -> Activation

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Activation

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for Activation

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for Activation

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for Activation