Struct burn_core::nn::PositionalEncoding

pub struct PositionalEncoding<B: Backend> { /* private fields */ }

Positional encoding layer for transformer models.

This layer adds positional information to the input embeddings, allowing the transformer model to take into account the order of the sequence. The positional encoding is added to the input embeddings by computing a set of sinusoidal functions with different frequencies and phases.

Sinusoids are used for positional embedding introduced in Attention is all you need.

The reference implementation can be found here: LANGUAGE MODELING WITH NN.TRANSFORMER AND TORCHTEXT

Implementations

impl<B: Backend> PositionalEncoding<B>

pub fn forward(&self, input: Tensor<B, 3>) -> Tensor<B, 3>

Applies the forward pass on the input tensor by adding the sinusoids to the input.

Shapes

• input: [batch_size, seq_length, d_model]
• output: [batch_size, seq_length, d_model]

Panics

• Panics if the input sequence length is greater than the maximum sequence size.
• Panics if the input d_model is not equal to the d_model of the sinusoids.

Trait Implementations

impl<B> AutodiffModule<B> for PositionalEncoding<B>where B: AutodiffBackend + Backend, <B as AutodiffBackend>::InnerBackend: Backend,

type InnerModule = PositionalEncoding<<B as AutodiffBackend>::InnerBackend>

Inner module without auto-differentiation.

fn valid(&self) -> Self::InnerModule

Get the same module, but on the inner backend without auto-differentiation.

impl<B: Backend> Clone for PositionalEncoding<B>

fn clone(&self) -> Self

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<B: Debug + Backend> Debug for PositionalEncoding<B>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<B: Backend> Display for PositionalEncoding<B>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<B: Backend> Module<B> for PositionalEncoding<B>

type Record = PositionalEncodingRecord<B>

Type to save and load the module.

fn load_record(self, record: Self::Record) -> Self

Load the module state from a record.

fn into_record(self) -> Self::Record

Convert the module into a record containing the state.

fn num_params(&self) -> usize

Get the number of parameters the module has, including all of its sub-modules.

fn visit<V: ModuleVisitor<B>>(&self, visitor: &mut V)

Visit each tensor parameter in the module with a visitor.

fn map<M: ModuleMapper<B>>(self, mapper: &mut M) -> Self

Map each tensor parameter in the module with a mapper.

fn collect_devices(&self, devices: Devices<B>) -> Devices<B>

Return all the devices found in the underneath module tree added to the given vector without duplicates.

fn to_device(self, device: &B::Device) -> Self

Move the module and all of its sub-modules to the given device.

fn fork(self, device: &B::Device) -> Self

Fork the module and all of its sub-modules to the given device.

fn devices(&self) -> Devices<B>

Return all the devices found in the underneath module tree without duplicates.

fn no_grad(self) -> Self

Each tensor in the module tree will not require grad.

fn save_file<FR: FileRecorder, PB: Into<PathBuf>>( self, file_path: PB, recorder: &FR ) -> Result<(), RecorderError>

Save the module to a file using the provided file recorder.

fn load_file<FR: FileRecorder, PB: Into<PathBuf>>( self, file_path: PB, recorder: &FR ) -> Result<Self, RecorderError>

Load the module from a file using the provided file recorder.