Struct RotaryPositionEncoding 

pub struct RotaryPositionEncoding {
    pub config: PositionEncodingConfig,
}

Rotary Position Embedding (RoPE)

Used in models like LLaMA, GPT-NeoX, PaLM. RoPE encodes position by rotating query and key vectors in the complex plane, providing natural relative position information without adding extra parameters.

Reference: “RoFormer: Enhanced Transformer with Rotary Position Embedding” https://arxiv.org/abs/2104.09864

Fields

config: PositionEncodingConfig

Configuration

Implementations

impl RotaryPositionEncoding

pub fn new(config: PositionEncodingConfig) -> Result<Self>

Create a new rotary position encoding

pub fn build_encoding_graph( &self, graph: &mut EinsumGraph, ) -> Result<Vec<usize>>

Build einsum graph for RoPE

RoPE applies rotation to query and key vectors in attention:

• Splits d_model into pairs of dimensions
• Rotates each pair by position-dependent angle
• Preserves relative position information

Input tensors:

• 0: x (input) [batch, seq_len, d_model]
• 1: cos_cached [max_seq_len, d_model/2] (precomputed cosines)
• 2: sin_cached [max_seq_len, d_model/2] (precomputed sines)

Output tensors:

• output: [batch, seq_len, d_model] (rotated embeddings)

pub fn base(&self) -> f64

Get the base frequency for RoPE

pub fn scaling_factor(&self) -> f64

Get the scaling factor for long sequences

Trait Implementations

impl Clone for RotaryPositionEncoding

fn clone(&self) -> RotaryPositionEncoding

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RotaryPositionEncoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.