Struct RotaryPositionalEncoding 

pub struct RotaryPositionalEncoding<F: Float + Debug + NumAssign> { /* private fields */ }

Rotary Positional Encoding (RoPE)

Implements rotary position embeddings as described in “RoFormer: Enhanced Transformer with Rotary Position Embedding”. RoPE encodes position information by rotating the query and key vectors, which has several advantages:

• Relative position information is naturally encoded
• Better length extrapolation than absolute position encodings
• Used in modern LLMs like LLaMA, GPT-NeoX, etc.

Implementations

impl<F: Float + Debug + NumAssign> RotaryPositionalEncoding<F>

pub fn new(d_model: usize, max_len: usize, base: f64) -> Self

Create a new RoPE encoding

Arguments

• d_model - Model dimension (must be even)
• max_len - Maximum sequence length
• base - Base for frequency computation (default: 10000.0)

pub fn default_base(d_model: usize, max_len: usize) -> Self

Create with default base (10000.0)

pub fn rotate(&self, x: &Array3<F>, offset: usize) -> Result<Array3<F>>

Apply rotary embedding to query or key tensor

Arguments

• x - Input tensor of shape [batch, seq_len, d_model]
• offset - Position offset (for KV cache during inference)

Returns

Rotated tensor with same shape

pub fn sin_cache(&self) -> &Array2<F>

Get sin cache

pub fn cos_cache(&self) -> &Array2<F>

Get cos cache

Trait Implementations

impl<F: Clone + Float + Debug + NumAssign> Clone for RotaryPositionalEncoding<F>

fn clone(&self) -> RotaryPositionalEncoding<F>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<F: Debug + Float + Debug + NumAssign> Debug for RotaryPositionalEncoding<F>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<F: Float + Debug + NumAssign> PositionalEncoding<F> for RotaryPositionalEncoding<F>

fn encode(&self, seq_len: usize) -> Array2<F>

Encode positions for a sequence of given length

fn apply(&self, input: &Array3<F>) -> Result<Array3<F>>

Apply positional encoding to an input tensor

fn clone_box(&self) -> Box<dyn PositionalEncoding<F> + Send + Sync>

where F: Send + Sync + 'static,

Clone the positional encoding into a boxed trait object

fn d_model(&self) -> usize

Get the model dimension

fn max_len(&self) -> usize

Get the maximum sequence length supported

fn forward(&self, input: &Array<F, IxDyn>) -> Result<Array<F, IxDyn>>

Forward pass with dynamic array

fn update(&mut self, _learning_rate: F) -> Result<()>

Update trainable parameters (for learned encodings)