Struct RDrop 

pub struct RDrop { /* private fields */ }

R-Drop regularization (Liang et al., 2021).

Forces consistency between two forward passes with different dropout masks. Adds bidirectional KL divergence loss between the two outputs.

Loss = CE(p1, y) + CE(p2, y) + α * (KL(p1||p2) + KL(p2||p1)) / 2

Reference

Liang, X., et al. (2021). R-Drop: Regularized Dropout for Neural Networks.

Implementations

impl RDrop

pub fn new(alpha: f32) -> Self

Create R-Drop with regularization weight alpha.

pub fn alpha(&self) -> f32

pub fn kl_divergence(&self, p: &[f32], q: &[f32]) -> f32

Compute KL divergence: KL(p || q) = Σ p * log(p / q)

pub fn symmetric_kl(&self, p: &[f32], q: &[f32]) -> f32

Compute bidirectional KL divergence (symmetric).

pub fn compute_loss(&self, logits1: &[f32], logits2: &[f32]) -> f32

Compute R-Drop regularization loss between two forward passes.

Trait Implementations

impl Clone for RDrop

fn clone(&self) -> RDrop

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RDrop

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.