Struct AlibiPositionEncoding 

pub struct AlibiPositionEncoding {
    pub config: PositionEncodingConfig,
}

ALiBi (Attention with Linear Biases)

Used in models like BLOOM. Instead of adding position embeddings to inputs, ALiBi adds a bias to attention scores that linearly penalizes distance. This allows extrapolation to longer sequences than seen during training.

Reference: “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation” https://arxiv.org/abs/2108.12409

Fields

config: PositionEncodingConfig

Configuration

Implementations

impl AlibiPositionEncoding

pub fn new(config: PositionEncodingConfig) -> Result<Self>

Create a new ALiBi position encoding

pub fn build_bias_graph(&self, graph: &mut EinsumGraph) -> Result<Vec<usize>>

Build einsum graph for ALiBi bias

ALiBi adds linear biases to attention scores based on query-key distance: bias(i, j) = -m * |i - j| where m is a head-specific slope

Input tensors:

• 0: attention_scores [batch, n_heads, seq_len, seq_len]
• 1: alibi_slopes [n_heads] (precomputed slopes, one per head)
• 2: distance_matrix [seq_len, seq_len] (|i - j| for all positions)

Output tensors:

• output: [batch, n_heads, seq_len, seq_len] (scores with ALiBi bias)

pub fn n_heads(&self) -> usize

Get the number of attention heads

pub fn compute_slopes(&self) -> Vec<f64>

Compute ALiBi slopes for each attention head

Slopes are computed as: m_i = 2^(-8i/n) for i in 1..n_heads This gives different rates of distance penalty per head

Trait Implementations

impl Clone for AlibiPositionEncoding

fn clone(&self) -> AlibiPositionEncoding

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AlibiPositionEncoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.