Struct NaturalGradient 

pub struct NaturalGradient { /* private fields */ }

Natural gradient optimizer

Implementations

impl NaturalGradient

pub fn new(config: NaturalGradientConfig) -> Self

Create new natural gradient optimizer

pub fn step_logits(&self, logits: &[f32], grad_logits: &[f32]) -> Vec<f32>

Compute natural gradient step for logits Returns updated logits

pub fn step_diagonal( &self, params: &[f32], grads: &[f32], fisher_diag: &[f32], ) -> Vec<f32>

Compute natural gradient step for general parameters with diagonal Fisher Fisher diag should be pre-computed from data

pub fn step_attention_logits( &self, logits: &[f32], grad_logits: &[f32], ) -> Vec<f32>

Compute natural gradient for attention logits Uses the Fisher metric on the output probability distribution

Trait Implementations

impl Clone for NaturalGradient

fn clone(&self) -> NaturalGradient

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NaturalGradient

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.