Struct AdamW 

pub struct AdamW {
    pub learning_rate: f32,
    pub betas: (f32, f32),
    pub eps: f32,
    pub weight_decay: f32,
    pub amsgrad: bool,
    pub m: Vec<Tensor>,
    pub v: Vec<Tensor>,
    pub vm: Vec<Tensor>,
    pub t: usize,
}

Adaptive momentum estimation optimizer

Fields

learning_rate: f32

learning rate (default: 1e-3)

betas: (f32, f32)

coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999))

eps: f32

term added to the denominator to improve numerical stability (default: 1e-8)

weight_decay: f32

weight decay (L2 penalty) (default: 0)

amsgrad: bool

whether to use the AMSGrad variant of this algorithm from the paper On the Convergence of Adam and Beyond (default: false)

m: Vec<Tensor>

m

v: Vec<Tensor>

v

vm: Vec<Tensor>

vm

t: usize

t

Implementations

impl AdamW

pub fn update<'a>( &mut self, parameters: impl IntoIterator<Item = &'a mut Tensor>, gradients: impl IntoIterator<Item = Option<Tensor>>, )

Updates parameters with gradients. Number of parameters must be the same as number of gradients. Gradients can be None, those are simply skipped.

Trait Implementations

impl Default for AdamW

fn default() -> Self

Returns the “default value” for a type.

impl<'a> IntoIterator for &'a AdamW

type Item = &'a Tensor

The type of the elements being iterated over.

type IntoIter = IntoIter<&'a Tensor>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl<'a> IntoIterator for &'a mut AdamW

type Item = &'a mut Tensor

The type of the elements being iterated over.

type IntoIter = IntoIter<&'a mut Tensor>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl Module for AdamW

fn iter<'a>(&'a self) -> impl Iterator<Item = &'a Tensor>

Iterate over all tensors immutably

fn iter_mut<'a>(&'a mut self) -> impl Iterator<Item = &'a mut Tensor>

Iterate over all tensors mutably

fn iter_tensors<'a>(&'a self) -> impl Iterator<Item = (String, &'a Tensor)>

Iterate over tensors without consuming the module

fn iter_tensors_mut<'a>( &'a mut self, ) -> impl Iterator<Item = (String, &'a mut Tensor)>

From tensors

fn realize(&self) -> Result<(), ZyxError>

Realize all tensors in the module

fn set_params(&mut self, params: &mut HashMap<String, Tensor>)

Set parameters, removes them from params, skips parameters that are not found in params.

fn save(&self, path: impl AsRef<Path>) -> Result<(), ZyxError>

Save tensors or modules to a file determined by file extension. Currently only safetensors is supported format.