Module gradient_processing 

Gradient Processing Enhancements

This module provides advanced gradient processing techniques that can improve training stability, convergence speed, and final model performance.

Available Techniques

• Gradient Centralization: Removes the mean of gradients to improve convergence
• Gradient Standardization: Normalizes gradients to unit variance
• Adaptive Gradient Clipping: Dynamically adjusts clipping based on gradient history
• Gradient Noise Injection: Adds controlled noise to escape local minima
• Gradient Smoothing: Applies exponential moving average to gradients
• Hessian-based Preconditioning: Uses second-order information to precondition gradients

Structs

AdaptiveClippingConfig

Configuration for adaptive gradient clipping.

GradientProcessedOptimizer

Wrapper for optimizers that automatically applies gradient processing.

GradientProcessingConfig

Configuration for gradient processing techniques.

GradientProcessor

Gradient processor that applies various enhancement techniques.

HessianPreconditioningConfig

Configuration for Hessian-based preconditioning.

NoiseInjectionConfig

Configuration for gradient noise injection.

SmoothingConfig

Configuration for gradient smoothing.

Enums

HessianApproximationType

Types of Hessian approximation methods.

NoiseType

Types of noise for gradient noise injection.