Skip to main content

gated_delta_rule_preprocess

Function gated_delta_rule_preprocess 

Source
pub fn gated_delta_rule_preprocess(
    b: &[f32],
    a: &[f32],
    a_log: &[f32],
    dt_bias: &[f32],
    rows: usize,
    hidden: usize,
) -> (Vec<f32>, Vec<f32>)
Expand description

Prepares Gated Delta Rule recurrent inputs after projection.

b and a are [rows, hidden]; a_log and dt_bias are [hidden]. The returned beta is sigmoid(b). The returned gate is -exp(a_log) * softplus(a + dt_bias), using the same stable softplus threshold as the Triton reference.