Enum ClippingStrategy 

pub enum ClippingStrategy {
    GlobalNorm {
        max_norm: f64,
    },
    PerTensorNorm {
        max_norm: f64,
    },
    ValueClip {
        min: f64,
        max: f64,
    },
    Adaptive {
        target_norm: f64,
        momentum: f64,
    },
}

Strategy for clipping gradients.

Variants

GlobalNorm

Clip all gradients so that their global L2 norm is at most max_norm.

Fields

max_norm: f64

Maximum allowed global L2 norm.

PerTensorNorm

Clip each gradient tensor independently so its L2 norm is at most max_norm.

Fields

max_norm: f64

Maximum allowed per-tensor L2 norm.

ValueClip

Clamp every scalar value in every tensor to the range [min, max].

Fields

min: f64

Minimum allowed value.

max: f64

Maximum allowed value.

Adaptive

Running EMA of the global norm; clip when current norm > EMA * 1.5.

The EMA is updated as: ema = momentum * ema + (1 - momentum) * global_norm. On the very first call the EMA is bootstrapped to the current global norm so no spurious clipping occurs on the initial step.

Fields

target_norm: f64

Target norm used to scale clipping (the clip threshold is ema * 1.5).

momentum: f64

EMA momentum coefficient (should be in [0, 1)).

Trait Implementations

impl Clone for ClippingStrategy

fn clone(&self) -> ClippingStrategy

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ClippingStrategy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for ClippingStrategy

fn eq(&self, other: &ClippingStrategy) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for ClippingStrategy