Struct tensorflow_proto::tensorflow::tpu::AdamParameters

pub struct AdamParameters {
    pub beta1: f32,
    pub beta2: f32,
    pub epsilon: f32,
    pub initial_m: f32,
    pub initial_v: f32,
    pub use_non_lazy_adam: bool,
    pub use_sum_inside_sqrt: bool,
}

The Adam optimizer does not implement hyper-parameter update; use the dynamic learning rate feature instead, setting the learning rate to: user learning_rate * sqrt(1 - beta2^t) / (1 - beta1^t) Here, t is the current timestep.

https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer https://github.com/tensorflow/tensorflow/blob/ab51450c817674c8ff08a7ae4f8ac50cdc4bed8b/tensorflow/python/training/adam.py#L54

Note that the code by default implements the lazy version of Adam (https://www.tensorflow.org/api_docs/python/tf/contrib/opt/LazyAdamOptimizer) unless the use_non_lazy_adam parameter is set, in which case it implements the normal version of Adam that updates all parameters in the embedding table, even for entries that are not used in the current minibatch (https://www.tensorflow.org/api_docs/python/tf/contrib/opt/AdamOptimizer). If use_non_lazy_adam is enabled, gradient accumulation is also required to be enabled in order to get correct results; a warning will be printed otherwise (which may change to an error in the future). If use_sum_inside_sqrt is set, the Adam variable update formula will be changed from m / (sqrt(v) + epsilon) to m / sqrt(v + epsilon**2); this option improves the performance of TPU training and is not expected to harm model quality.

Fields

beta1: f32

beta2: f32

epsilon: f32

initial_m: f32

initial_v: f32

use_non_lazy_adam: bool

use_sum_inside_sqrt: bool

Trait Implementations

impl Clone for AdamParameters

pub fn clone(&self) -> AdamParameters

Returns a copy of the value.

pub fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AdamParameters

pub fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for AdamParameters

pub fn default() -> Self

Returns the "default value" for a type.

impl Message for AdamParameters

pub fn encode_raw<B>(&self, buf: &mut B) where
    B: BufMut, 

pub fn merge_field<B>(
    &mut self,
    tag: u32,
    wire_type: WireType,
    buf: &mut B,
    ctx: DecodeContext
) -> Result<(), DecodeError> where
    B: Buf, 

pub fn encoded_len(&self) -> usize

Returns the encoded length of the message without a length delimiter.

pub fn clear(&mut self)

Clears the message, resetting all fields to their default.

pub fn encode<B>(&self, buf: &mut B) -> Result<(), EncodeError> where
    B: BufMut, 

Encodes the message to a buffer.

pub fn encode_length_delimited<B>(&self, buf: &mut B) -> Result<(), EncodeError> where
    B: BufMut, 

Encodes the message with a length-delimiter to a buffer.

pub fn decode<B>(buf: B) -> Result<Self, DecodeError> where
    Self: Default,
    B: Buf, 

Decodes an instance of the message from a buffer.

pub fn decode_length_delimited<B>(buf: B) -> Result<Self, DecodeError> where
    Self: Default,
    B: Buf, 

Decodes a length-delimited instance of the message from the buffer.

pub fn merge<B>(&mut self, buf: B) -> Result<(), DecodeError> where
    B: Buf, 

Decodes an instance of the message from a buffer, and merges it into self.

pub fn merge_length_delimited<B>(&mut self, buf: B) -> Result<(), DecodeError> where
    B: Buf, 

Decodes a length-delimited instance of the message from buffer, and merges it into self.

impl PartialEq<AdamParameters> for AdamParameters

pub fn eq(&self, other: &AdamParameters) -> bool

This method tests for self and other values to be equal, and is used by ==.

pub fn ne(&self, other: &AdamParameters) -> bool

This method tests for !=.

impl StructuralPartialEq for AdamParameters