Struct LightningDecay 

pub struct LightningDecay {
    pub slopes: Vec<f32>,
    pub scale: f32,
}

The two numbers a layer’s decay is built from.

Split out because both are functions of the LAYER and the head count rather than of the file: minimax-01.cpp:288 derives the per-layer scale from the layer index, and the slopes are the geometric ladder ALiBi uses. A file that carried them would be carrying dead metadata.

Fields

slopes: Vec<f32>

One slope per head, in head order.

scale: f32

This layer’s scale on every slope.

Implementations

impl LightningDecay

pub fn for_layer(il: usize, n_layer: usize, n_head: usize) -> Self

pub fn per_step(&self, head: usize) -> f32

exp(-c s_h): what one token multiplies head h’s state by.

Trait Implementations

impl Clone for LightningDecay

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LightningDecay

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.