Struct PositionalEncoding

pub struct PositionalEncoding<B: Backend> {
    pub sinusoids: Tensor<B, 3>,
    pub max_sequence_size: usize,
    pub max_timescale: usize,
}

Positional encoding layer for transformer models.

This layer adds positional information to the input embeddings, allowing the transformer model to take into account the order of the sequence. The positional encoding is added to the input embeddings by computing a set of sinusoidal functions with different frequencies and phases.

Sinusoids are used for positional embedding introduced in Attention is all you need.

The reference implementation can be found here: LANGUAGE MODELING WITH NN.TRANSFORMER AND TORCHTEXT

Should be created using PositionalEncodingConfig

Fields

sinusoids: Tensor<B, 3>

The sinusoids used to add positional information to the input embeddings.

max_sequence_size: usize

The maximum sequence size to use.

max_timescale: usize

Max time scale to use.

Implementations

impl<B: Backend> PositionalEncoding<B>

pub fn forward(&self, input: Tensor<B, 3>) -> Tensor<B, 3>

Applies the forward pass on the input tensor by adding the sinusoids to the input.

Shapes

• input: [batch_size, seq_length, d_model]
• output: [batch_size, seq_length, d_model]

Panics

• Panics if the input sequence length is greater than the maximum sequence size.
• Panics if the input d_model is not equal to the d_model of the sinusoids.

Trait Implementations

impl<B> AutodiffModule<B> for PositionalEncoding<B>

where B: AutodiffBackend + Backend, <B as AutodiffBackend>::InnerBackend: Backend,

type InnerModule = PositionalEncoding<<B as AutodiffBackend>::InnerBackend>

Inner module without auto-differentiation.

fn valid(&self) -> Self::InnerModule

Get the same module, but on the inner backend without auto-differentiation.

impl<B: Backend> Clone for PositionalEncoding<B>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<B: Debug + Backend> Debug for PositionalEncoding<B>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<B: Backend> Display for PositionalEncoding<B>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<B: Backend> Module<B> for PositionalEncoding<B>

type Record = PositionalEncodingRecord<B>

Type to save and load the module.

fn load_record(self, record: Self::Record) -> Self

Load the module state from a record.

fn into_record(self) -> Self::Record

Convert the module into a record containing the state.

fn num_params(&self) -> usize

Get the number of parameters the module has, including all of its sub-modules.

fn visit<Visitor: ModuleVisitor<B>>(&self, visitor: &mut Visitor)

Visit each tensor parameter in the module with a visitor.

fn map<Mapper: ModuleMapper<B>>(self, mapper: &mut Mapper) -> Self

Map each tensor parameter in the module with a mapper.

fn collect_devices(&self, devices: Devices<B>) -> Devices<B>

Return all the devices found in the underneath module tree added to the given vector without duplicates.

fn to_device(self, device: &B::Device) -> Self

Move the module and all of its sub-modules to the given device.

fn fork(self, device: &B::Device) -> Self

Fork the module and all of its sub-modules to the given device.

fn devices(&self) -> Devices<B>

Return all the devices found in the underneath module tree without duplicates.

fn no_grad(self) -> Self

Each tensor in the module tree will not require grad.

fn save_file<FR, PB>( self, file_path: PB, recorder: &FR, ) -> Result<(), RecorderError>

where FR: FileRecorder<B>, PB: Into<PathBuf>,

Available on crate feature std only.

Save the module to a file using the provided file recorder.

fn load_file<FR, PB>( self, file_path: PB, recorder: &FR, device: &B::Device, ) -> Result<Self, RecorderError>

where FR: FileRecorder<B>, PB: Into<PathBuf>,

Available on crate feature std only.

Load the module from a file using the provided file recorder.

fn quantize_weights(self, quantizer: &mut Quantizer) -> Self

Quantize the weights of the module.

impl<B: Backend> ModuleDisplay for PositionalEncoding<B>

fn custom_settings(&self) -> Option<DisplaySettings>

Custom display settings for the module.

fn custom_content(&self, content: Content) -> Option<Content>

Custom attributes for the module.

fn format(&self, passed_settings: DisplaySettings) -> String

Formats the module with provided display settings.

impl<B: Backend> ModuleDisplayDefault for PositionalEncoding<B>

fn content(&self, content: Content) -> Option<Content>

Attributes of the module used for display purposes.

fn num_params(&self) -> usize

Gets the number of the parameters of the module.