Struct SparsePositionEncoding 

pub struct SparsePositionEncoding {
    pub max_seq_len: usize,
    pub dim: usize,
    pub n_buckets: usize,
    pub bucket_embeddings: Vec<Vec<f32>>,
}

Sparse relative position bias for attention (Raffel et al. T5-style, sparse variant).

Rather than dense relative position embeddings, only a subset of relative distances receive learned biases (sparsifying the position information).

Fields

max_seq_len: usize

Maximum sequence length.

dim: usize

Embedding dimension.

n_buckets: usize

Number of distinct relative distance buckets.

bucket_embeddings: Vec<Vec<f32>>

Bucket embedding table (n_buckets × dim).

Implementations

impl SparsePositionEncoding

pub fn new( max_seq_len: usize, dim: usize, n_buckets: usize, rng: &mut impl Rng, ) -> Self

Initialize with random bucket embeddings.

pub fn relative_position_bucket(&self, rel_pos: i32) -> usize

Map relative distance to a bucket index (logarithmic bucketing for large gaps).

pub fn position_bias_indices(&self, seq_len: usize) -> Vec<usize>

Compute position bias matrix: shape (seq_len × seq_len × dim). Returns a flat vector of (seq_len × seq_len) bucket indices.

pub fn get_position_biases(&self, seq_len: usize) -> Vec<Vec<f32>>

Look up position embeddings for a (seq_len × seq_len) attention matrix.