pub struct SpikeGuard { /* private fields */ }Expand description
Flags a training step whose loss jumps well above the recent average, so the caller can drop it before the optimizer sees it.
The average is an exponential moving mean of the accepted losses. A loss
that is not finite, or more than ratio times that mean once warmup
losses have been accepted, is a spike: it is not folded into the mean, and
SpikeGuard::consecutive counts how many came in a row. Skip a spike by
clearing the gradients instead of stepping; after several in a row, roll
back to the last checkpoint and its optimizer state.
use rusting_brain::SpikeGuard;
let mut guard = SpikeGuard::new(2.0, 0.99, 10);
for loss in [3.0, 2.9, 2.8, 2.9, 2.7, 2.8, 2.6, 2.7, 2.6, 2.5] {
assert!(!guard.is_spike(loss));
}
assert!(guard.is_spike(9.0));
assert!(guard.is_spike(f32::NAN));
assert_eq!(guard.consecutive(), 2);
assert!(!guard.is_spike(2.6));
assert_eq!(guard.consecutive(), 0);
// In a training loop:
// let loss = model.accumulate_step(&batch)?;
// if guard.is_spike(loss.lm_loss) {
// model.zero_grad();
// if guard.consecutive() >= 3 {
// model = TransformerLm::load_bin("last.bin")?;
// model.load_optimizer_state("last.opt")?;
// }
// } else {
// model.step_clipped(1.0, 1.0)?;
// }Implementations§
Source§impl SpikeGuard
impl SpikeGuard
Sourcepub fn new(ratio: f32, beta: f32, warmup: usize) -> Self
pub fn new(ratio: f32, beta: f32, warmup: usize) -> Self
A loss above ratio times the moving mean is a spike. beta is the
mean’s decay per accepted loss (0.99 averages about the last hundred),
and no loss is a spike on size alone until warmup have been accepted.
Sourcepub fn consecutive(&self) -> usize
pub fn consecutive(&self) -> usize
Spikes seen in a row since the last accepted loss.
Trait Implementations§
Source§impl Clone for SpikeGuard
impl Clone for SpikeGuard
impl Copy for SpikeGuard
Source§impl Debug for SpikeGuard
impl Debug for SpikeGuard
Source§impl PartialEq for SpikeGuard
impl PartialEq for SpikeGuard
impl StructuralPartialEq for SpikeGuard
Auto Trait Implementations§
impl Forget for SpikeGuard
impl Freeze for SpikeGuard
impl Move for SpikeGuard
impl RefUnwindSafe for SpikeGuard
impl Send for SpikeGuard
impl Sync for SpikeGuard
impl Unpin for SpikeGuard
impl UnsafeUnpin for SpikeGuard
impl UnwindSafe for SpikeGuard
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more