Trait ActivationOps 

pub trait ActivationOps<R>
where
    R: Runtime,
{
    // Provided methods
    fn relu(&self, a: &Tensor<R>) -> Result<Tensor<R>, Error> { ... }
    fn sigmoid(&self, a: &Tensor<R>) -> Result<Tensor<R>, Error> { ... }
    fn silu(&self, a: &Tensor<R>) -> Result<Tensor<R>, Error> { ... }
    fn gelu(&self, a: &Tensor<R>) -> Result<Tensor<R>, Error> { ... }
    fn leaky_relu(
        &self,
        a: &Tensor<R>,
        negative_slope: f64,
    ) -> Result<Tensor<R>, Error> { ... }
    fn elu(&self, a: &Tensor<R>, alpha: f64) -> Result<Tensor<R>, Error> { ... }
    fn softmax(&self, a: &Tensor<R>, dim: isize) -> Result<Tensor<R>, Error> { ... }
    fn log_softmax(&self, a: &Tensor<R>, dim: isize) -> Result<Tensor<R>, Error> { ... }
    fn softmax_bwd(
        &self,
        grad: &Tensor<R>,
        output: &Tensor<R>,
        dim: isize,
    ) -> Result<Tensor<R>, Error> { ... }
    fn softplus(&self, a: &Tensor<R>) -> Result<Tensor<R>, Error> { ... }
    fn silu_mul(&self, a: &Tensor<R>, b: &Tensor<R>) -> Result<Tensor<R>, Error> { ... }
    fn gelu_mul(&self, a: &Tensor<R>, b: &Tensor<R>) -> Result<Tensor<R>, Error> { ... }
    fn relu_mul(&self, a: &Tensor<R>, b: &Tensor<R>) -> Result<Tensor<R>, Error> { ... }
    fn sigmoid_mul(
        &self,
        a: &Tensor<R>,
        b: &Tensor<R>,
    ) -> Result<Tensor<R>, Error> { ... }
    fn silu_mul_bwd(
        &self,
        grad: &Tensor<R>,
        a: &Tensor<R>,
        b: &Tensor<R>,
    ) -> Result<(Tensor<R>, Tensor<R>), Error> { ... }
    fn gelu_mul_bwd(
        &self,
        grad: &Tensor<R>,
        a: &Tensor<R>,
        b: &Tensor<R>,
    ) -> Result<(Tensor<R>, Tensor<R>), Error> { ... }
    fn relu_mul_bwd(
        &self,
        grad: &Tensor<R>,
        a: &Tensor<R>,
        b: &Tensor<R>,
    ) -> Result<(Tensor<R>, Tensor<R>), Error> { ... }
    fn sigmoid_mul_bwd(
        &self,
        grad: &Tensor<R>,
        a: &Tensor<R>,
        b: &Tensor<R>,
    ) -> Result<(Tensor<R>, Tensor<R>), Error> { ... }
    fn softmax_with_bias(
        &self,
        a: &Tensor<R>,
        bias: &Tensor<R>,
        dim: isize,
    ) -> Result<Tensor<R>, Error> { ... }
    fn dropout(
        &self,
        a: &Tensor<R>,
        p: f64,
        training: bool,
    ) -> Result<Tensor<R>, Error> { ... }
}

Activation operations

Provided Methods

fn relu(&self, a: &Tensor<R>) -> Result<Tensor<R>, Error>

Rectified linear unit: max(0, a)

fn sigmoid(&self, a: &Tensor<R>) -> Result<Tensor<R>, Error>

Sigmoid: 1 / (1 + e^(-a))

fn silu(&self, a: &Tensor<R>) -> Result<Tensor<R>, Error>

SiLU (Swish): a * sigmoid(a) = a / (1 + e^(-a))

Used in LLaMA, Mistral, and other modern transformer architectures.

fn gelu(&self, a: &Tensor<R>) -> Result<Tensor<R>, Error>

GELU (Gaussian Error Linear Unit): 0.5 * a * (1 + tanh(sqrt(2/pi) * (a + 0.044715 * a^3)))

Uses the tanh approximation. Used in GPT, BERT, and other transformer architectures.

fn leaky_relu( &self, a: &Tensor<R>, negative_slope: f64, ) -> Result<Tensor<R>, Error>

Leaky ReLU: max(negative_slope * a, a)

Allows small gradients for negative inputs, helping prevent “dying ReLU” problem. Default negative_slope is typically 0.01.

fn elu(&self, a: &Tensor<R>, alpha: f64) -> Result<Tensor<R>, Error>

ELU (Exponential Linear Unit): a if a > 0, else alpha * (exp(a) - 1)

Smooth approximation to ReLU with negative values saturating to -alpha. Default alpha is typically 1.0.

fn softmax(&self, a: &Tensor<R>, dim: isize) -> Result<Tensor<R>, Error>

Softmax along a dimension

fn log_softmax(&self, a: &Tensor<R>, dim: isize) -> Result<Tensor<R>, Error>

Log-softmax along a dimension: log(softmax(x, dim))

Computed as x - logsumexp(x, dim) for numerical stability. Used in log-probability calculations, Bayesian inference, categorical distributions, and information theory.

fn softmax_bwd( &self, grad: &Tensor<R>, output: &Tensor<R>, dim: isize, ) -> Result<Tensor<R>, Error>

Softmax backward pass: computes gradient w.r.t. input given output gradient and softmax output.

Formula: d_input = output * (grad - sum(grad * output, dim, keepdim=true))

This is the Jacobian-vector product for softmax, used in training backward passes.

Arguments

• grad - Upstream gradient (same shape as output)
• output - The softmax output from the forward pass
• dim - The dimension along which softmax was computed

fn softplus(&self, a: &Tensor<R>) -> Result<Tensor<R>, Error>

Softplus: log(1 + exp(a))

A smooth approximation to ReLU that is always positive and differentiable. Used in Mamba2 for dt (step size) processing via softplus(dt_proj(x)) + dt_bias.

Gradient: sigmoid(a)

fn silu_mul(&self, a: &Tensor<R>, b: &Tensor<R>) -> Result<Tensor<R>, Error>

Fused SiLU-Mul: silu(a) * b in a single pass.

Computes (a / (1 + exp(-a))) * b element-wise with one memory pass instead of two (activation + multiply). Used in SwiGLU and similar gated architectures.

fn gelu_mul(&self, a: &Tensor<R>, b: &Tensor<R>) -> Result<Tensor<R>, Error>

Fused GELU-Mul: gelu(a) * b in a single pass.

Computes (0.5 * a * (1 + tanh(sqrt(2/pi) * (a + 0.044715*a^3)))) * b element-wise. Used in GeGLU gated architectures.

fn relu_mul(&self, a: &Tensor<R>, b: &Tensor<R>) -> Result<Tensor<R>, Error>

Fused ReLU-Mul: relu(a) * b in a single pass.

Computes max(0, a) * b element-wise. Used in ReGLU gated architectures.

fn sigmoid_mul(&self, a: &Tensor<R>, b: &Tensor<R>) -> Result<Tensor<R>, Error>

Fused Sigmoid-Mul: sigmoid(a) * b in a single pass.

Computes (1 / (1 + exp(-a))) * b element-wise. Used in SiGLU gated architectures.

fn silu_mul_bwd( &self, grad: &Tensor<R>, a: &Tensor<R>, b: &Tensor<R>, ) -> Result<(Tensor<R>, Tensor<R>), Error>

Fused SiLU-Mul backward: computes gradients for output = silu(a) * b.

Returns (d_a, d_b) where:

• d_a = grad * b * silu'(a) with silu'(x) = sigmoid(x) * (1 + x - silu(x))
• d_b = grad * silu(a)

Backends may implement this as a single fused kernel for better performance.

fn gelu_mul_bwd( &self, grad: &Tensor<R>, a: &Tensor<R>, b: &Tensor<R>, ) -> Result<(Tensor<R>, Tensor<R>), Error>

Fused GELU-Mul backward: computes gradients for output = gelu(a) * b.

Returns (d_a, d_b) where:

• d_a = grad * b * gelu'(a)
• d_b = grad * gelu(a)

fn relu_mul_bwd( &self, grad: &Tensor<R>, a: &Tensor<R>, b: &Tensor<R>, ) -> Result<(Tensor<R>, Tensor<R>), Error>

Fused ReLU-Mul backward: computes gradients for output = relu(a) * b.

Returns (d_a, d_b) where:

• d_a = grad * b * relu'(a) with relu'(x) = 1 if x > 0, else 0
• d_b = grad * relu(a)

fn sigmoid_mul_bwd( &self, grad: &Tensor<R>, a: &Tensor<R>, b: &Tensor<R>, ) -> Result<(Tensor<R>, Tensor<R>), Error>

Fused Sigmoid-Mul backward: computes gradients for output = sigmoid(a) * b.

Returns (d_a, d_b) where:

• d_a = grad * b * sigmoid'(a) with sigmoid'(x) = sigmoid(x) * (1 - sigmoid(x))
• d_b = grad * sigmoid(a)

fn softmax_with_bias( &self, a: &Tensor<R>, bias: &Tensor<R>, dim: isize, ) -> Result<Tensor<R>, Error>

Fused softmax with additive bias: softmax(a + bias, dim) in a single memory pass.

Semantically equivalent to softmax(a + bias, dim) but avoids materializing the intermediate a + bias tensor. The bias is added element-wise (with broadcasting) during the max/exp/sum scan so that only one pass over a is needed.

Arguments

• a - Input tensor (e.g. [B, H, S, S] attention scores)
• bias - Additive bias broadcastable to a (e.g. [B, 1, 1, S] attention mask)
• dim - Dimension along which softmax is computed

Semantics

Must produce results numerically identical (within FP tolerance) to softmax(add_broadcast(a, bias), dim). Accumulation is always done in F32 even when a and bias are F16/BF16.

fn dropout( &self, a: &Tensor<R>, p: f64, training: bool, ) -> Result<Tensor<R>, Error>

Dropout: randomly zero elements with probability p during training.

When training is true, each element is independently zeroed with probability p, and remaining elements are scaled by 1/(1-p) to maintain expected values. When training is false, returns the input unchanged.

Used in regularization, Monte Carlo dropout, and Bayesian approximation.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl ActivationOps<CpuRuntime> for CpuClient

ActivationOps implementation for CPU runtime.