Struct SdpaOutputGate 

pub struct SdpaOutputGate {
    pub w_gate: Vec<f32>,
}

Head-specific sigmoid gate after SDPA, before Wo. Zero-initialized → starts at sigmoid(0) = 0.5 (neutral multiplicative identity).

Fields

w_gate: Vec<f32>

Gate weights: [n_heads * head_dim, dim]. Zero-init so gate starts at sigmoid(0) = 0.5.

Implementations

impl SdpaOutputGate

pub fn new(n_heads: usize, head_dim: usize, dim: usize) -> Self

Allocate zeroed gate weights.

pub fn forward(&self, attn_out: &mut [f32], dim: usize, temp: &mut [f32])

Apply sigmoid-gated projection to attention output.

Computes: gate[i] = sigmoid(W_gate[i] · attn_out), then attn_out[i] *= gate[i]. Zero-init weights produce sigmoid(0) = 0.5 for all (neutral half-pass). Paper reference: +0.3–0.5 avg points on zero-shot benchmarks.

Trait Implementations

impl Clone for SdpaOutputGate

fn clone(&self) -> SdpaOutputGate

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.