Struct Lion

Source

pub struct Lion { /* private fields */ }

Expand description

Lion (EvoLved Sign Momentum) optimizer.

Lion keeps a single momentum tensor and updates parameters with the sign of an interpolation between that momentum and the current gradient. Weight decay is decoupled from the gradient, as in AdamW.

Lion commonly needs a learning rate smaller than AdamW’s and a larger decoupled weight decay. These hyperparameters should be tuned together.

See Symbolic Discovery of Optimization Algorithms.

Trait Implementations§

Source §

impl Clone for Lion

Source §

fn clone(&self) -> Self

Returns a duplicate of the value. Read more

1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more

Source §

impl Optimizer for Lion

Source §

type State<const D: usize> = LionState<D>

The state of the optimizer for a single parameter of rank D. Read more

Source §

fn step<const D: usize>( &self, lr: LearningRate, tensor: Tensor<D>, grad: Tensor<D>, state: Option<Self::State<D>>, ) -> (Tensor<D>, Option<Self::State<D>>)

The optimizer step is performed for one tensor at a time with its gradient and state. Read more

Source §

fn to_device<const D: usize>( state: Self::State<D>, device: &Device, ) -> Self::State<D>

Change the device of the state. Read more

Auto Trait Implementations§

§

impl UnwindSafe for Lion

Blanket Implementations§

Source §

impl<T> Any for T
where T: 'static + ?Sized,

Source §

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more

Source §

impl<T> Borrow<T> for T
where T: ?Sized,

Source §

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more

Source §

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source §

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more

Source §

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source §

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source §

impl<T> CloneExpand for T
where T: Clone,

Source §

fn __expand_clone_method(&self, _: &Scope) -> T

Source §

impl<T> CloneToUninit for T
where T: Clone,

Source §

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)

Performs copy-assignment from self to dest. Read more

Source §

impl<T> Downcast for T
where T: Any,

Source §

fn into_any(self: Box<T>) -> Box<dyn Any>

Converts Box<dyn Trait> (where Trait: Downcast) to Box<dyn Any>, which can then be downcast into Box<dyn ConcreteType> where ConcreteType implements Trait.

Source §

fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>

Converts Rc<Trait> (where Trait: Downcast) to Rc<Any>, which can then be further downcast into Rc<ConcreteType> where ConcreteType implements Trait.

Source §

fn as_any(&self) -> &(dyn Any + 'static)

Converts &Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot generate &Any’s vtable from &Trait’s.

Source §

fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)

Converts &mut Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot generate &mut Any’s vtable from &mut Trait’s.

Source §

impl<T> Downcast<T> for T

Source §

fn downcast(&self) -> &T

Source §

impl<T> DowncastSend for T
where T: Any + Send,

Source §

fn into_any_send(self: Box<T>) -> Box<dyn Any + Send>

Converts Box<Trait> (where Trait: DowncastSend) to Box<dyn Any + Send>, which can then be downcast into Box<ConcreteType> where ConcreteType implements Trait.

Source §

impl<T> DowncastSync for T
where T: Any + Send + Sync,

Source §

fn into_any_sync(self: Box<T>) -> Box<dyn Any + Send + Sync>

Converts Box<Trait> (where Trait: DowncastSync) to Box<dyn Any + Send + Sync>, which can then be downcast into Box<ConcreteType> where ConcreteType implements Trait.

Source §

fn into_any_arc(self: Arc<T>) -> Arc<dyn Any + Send + Sync> ⓘ

Converts Arc<Trait> (where Trait: DowncastSync) to Arc<Any>, which can then be downcast into Arc<ConcreteType> where ConcreteType implements Trait.

Source §

impl<T> DynClone for T
where T: Clone,

Source §

fn __clone_box(&self, _: Private) -> *mut ()

Source §

impl<O> DynOptimizer for O
where O: Optimizer,

Source §

fn step_dyn( &self, rank: usize, lr: f64, tensor: BridgeTensor, grad: BridgeTensor, state: Option<DynState>, ) -> (BridgeTensor, Option<DynState>)

Perform an optimizer step for a single parameter of the given rank.

Source §

fn to_device_dyn(&self, state: DynState, device: &Device) -> DynState

Move a state to the given device.

Source §

fn state_flatten(&self, prefix: &str, state: &DynState, out: &mut StateSink)

Decompose a state into named tensors and scalars under prefix.

Source §

fn state_unflatten( &self, rank: usize, prefix: &str, src: &mut StateSource, device: &Device, ) -> Option<DynState>

Rebuild a state of the given rank from named tensors and scalars under prefix. Read more

Source §

impl<T> ErasedDestructor for T
where T: 'static,

Source §

impl<T> From<T> for T

Source §

fn from(t: T) -> T

Returns the argument unchanged.

Source §

impl<T> Instrument for T

Source §

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more

Source §

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more

Source §

impl<T, U> Into for T
where U: From<T>,

Source §

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source §

impl<T> IntoComptime for T

Source §

fn comptime(self) -> Self

Source §

impl<T> IntoEither for T

Source §

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more

Source §

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more

Source §