Struct PositionalEncoding 

pub struct PositionalEncoding { /* private fields */ }

Sinusoidal positional encoding module for transformers.

This module adds fixed (non-learnable) positional encodings to input embeddings. It uses the same formulation as in the original “Attention is All You Need” paper, based on sine and cosine functions of different frequencies.

It supports both f32 and f64 types and applies dropout after adding the encodings.

Implementations

impl PositionalEncoding

pub fn new( d_model: u64, max_len: usize, dropout_prob: f32, dtype: DType, ) -> Result<Self, ZyxError>

Creates a new PositionalEncoding module.

Arguments

• d_model - The embedding dimension (must match the input’s last dimension).
• max_len - Maximum sequence length this module will support.
• dropout_prob - Dropout probability applied after adding the positional encoding.
• dtype - Data type of the encoding (must be DType::F32 or DType::F64).

Errors

Returns a ZyxError::ShapeError if a non-floating-point dtype is used.

Example

let pe = PositionalEncoding::new(512, 1024, 0.1, DType::F32)?;

pub fn forward(&self, x: impl Into<Tensor>) -> Result<Tensor, ZyxError>

Applies positional encoding to the input tensor.

Arguments

• x - A tensor of shape [batch_size, seq_len, d_model].

Returns

A new tensor with the same shape as the input, with positional encodings added and dropout applied.

Errors

Returns a ZyxError::ShapeError if:

• Input tensor is not 3-dimensional.
• The input dimension d_model does not match the positional encoding.
• The sequence length exceeds the configured max_len.

Trait Implementations

impl Debug for PositionalEncoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> IntoIterator for &'a PositionalEncoding

type Item = &'a Tensor

The type of the elements being iterated over.

type IntoIter = IntoIter<&'a Tensor>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl<'a> IntoIterator for &'a mut PositionalEncoding

type Item = &'a mut Tensor

The type of the elements being iterated over.

type IntoIter = IntoIter<&'a mut Tensor>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl Module for PositionalEncoding

fn iter<'a>(&'a self) -> impl Iterator<Item = &'a Tensor>

Iterate over all tensors immutably

fn iter_mut<'a>(&'a mut self) -> impl Iterator<Item = &'a mut Tensor>

Iterate over all tensors mutably

fn iter_tensors<'a>(&'a self) -> impl Iterator<Item = (String, &'a Tensor)>

Iterate over tensors without consuming the module

fn iter_tensors_mut<'a>( &'a mut self, ) -> impl Iterator<Item = (String, &'a mut Tensor)>

From tensors

fn realize(&self) -> Result<(), ZyxError>

Realize all tensors in the module

fn set_params(&mut self, params: &mut HashMap<String, Tensor>)

Set parameters, removes them from params, skips parameters that are not found in params.

fn save(&self, path: impl AsRef<Path>) -> Result<(), ZyxError>

Save tensors or modules to a file determined by file extension. Currently only safetensors is supported format.