pub struct TransformerEncoderLayer<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default = MoiraiBackend, const H: usize = 8, M: AttentionMask = NullMask> {
pub norm1: LayerNorm<T, B>,
pub self_attn: MultiHeadAttention<T, B, H, M>,
pub dropout1: Dropout,
pub norm2: LayerNorm<T, B>,
pub ffn: FeedForward<T, B>,
pub dropout2: Dropout,
}Expand description
Single Transformer encoder layer.
Pre-LayerNorm sub-layer composition:
x₁ = x + Dropout(SelfAttn(LayerNorm(x)))
x₂ = x₁ + Dropout(FFN(LayerNorm(x₁)))Fields§
§norm1: LayerNorm<T, B>Pre-LayerNorm before self-attention.
self_attn: MultiHeadAttention<T, B, H, M>Self-attention sub-layer.
dropout1: DropoutDropout applied after self-attention.
norm2: LayerNorm<T, B>Pre-LayerNorm before FFN.
ffn: FeedForward<T, B>Feed-forward sub-layer.
dropout2: DropoutDropout applied after FFN.
Implementations§
Source§impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, M: AttentionMask> TransformerEncoderLayer<T, B, H, M>
impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, M: AttentionMask> TransformerEncoderLayer<T, B, H, M>
Sourcepub fn new(
d_model: usize,
d_ff: usize,
dropout_p: f64,
) -> Result<Self, InitializationError<B::Error>>where
T: RealScalar,
B: RandomInitOps<T>,
pub fn new(
d_model: usize,
d_ff: usize,
dropout_p: f64,
) -> Result<Self, InitializationError<B::Error>>where
T: RealScalar,
B: RandomInitOps<T>,
Construct an encoder layer.
§Errors
Returns a typed error when attention dimensions are invalid or the selected backend cannot initialize projection weights.
Sourcepub fn forward_with_mask(
&self,
input: &Var<T, B>,
key_padding_mask: Option<&Var<T, B>>,
) -> Result<Var<T, B>, ModuleError<B::Error>>
pub fn forward_with_mask( &self, input: &Var<T, B>, key_padding_mask: Option<&Var<T, B>>, ) -> Result<Var<T, B>, ModuleError<B::Error>>
Pre-LayerNorm forward with optional key padding mask.
Input/output shape: [batch, seq, d_model].
key_padding_mask shape: [batch, seq] (or backend-supported broadcast form).
Trait Implementations§
Source§impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, M: AttentionMask> Clone for TransformerEncoderLayer<T, B, H, M>
Manual Clone impl.
impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, M: AttentionMask> Clone for TransformerEncoderLayer<T, B, H, M>
Manual Clone impl.
Source§impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, M: AttentionMask> Module<T, B> for TransformerEncoderLayer<T, B, H, M>
impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, M: AttentionMask> Module<T, B> for TransformerEncoderLayer<T, B, H, M>
Source§fn forward(&self, input: &Var<T, B>) -> Result<Var<T, B>, ModuleError<B::Error>>
fn forward(&self, input: &Var<T, B>) -> Result<Var<T, B>, ModuleError<B::Error>>
Pre-LayerNorm forward. Input/output: [batch, seq, d_model].
Source§fn parameters(&self) -> Vec<Var<T, B>>
fn parameters(&self) -> Vec<Var<T, B>>
Collect all trainable parameters (including from sub-modules).
Source§fn named_parameters(&self) -> Vec<Parameter<T, B>>
fn named_parameters(&self) -> Vec<Parameter<T, B>>
Collect trainable parameters with stable hierarchical names. Read more
Source§fn load_parameters(&mut self, _params: &[Var<T, B>])
fn load_parameters(&mut self, _params: &[Var<T, B>])
Write optimizer-updated parameter values back into this module’s own
fields, consuming
params in the same order parameters() enumerates
them (i.e. params.len() == parameters().len()). Read moreSource§fn load_named_parameters(
&mut self,
parameters: &[Parameter<T, B>],
) -> Result<(), ParameterLoadError>
fn load_named_parameters( &mut self, parameters: &[Parameter<T, B>], ) -> Result<(), ParameterLoadError>
Load optimizer-updated named parameters after validating the complete
stable path inventory. Read more
Auto Trait Implementations§
impl<T, B = MoiraiBackend, const H: usize = 8, M = NullMask> !Freeze for TransformerEncoderLayer<T, B, H, M>
impl<T, B = MoiraiBackend, const H: usize = 8, M = NullMask> !RefUnwindSafe for TransformerEncoderLayer<T, B, H, M>
impl<T, B = MoiraiBackend, const H: usize = 8, M = NullMask> !Sync for TransformerEncoderLayer<T, B, H, M>
impl<T, B = MoiraiBackend, const H: usize = 8, M = NullMask> !UnwindSafe for TransformerEncoderLayer<T, B, H, M>
impl<T, B, const H: usize, M> Forget for TransformerEncoderLayer<T, B, H, M>
impl<T, B, const H: usize, M> Move for TransformerEncoderLayer<T, B, H, M>
impl<T, B, const H: usize, M> Send for TransformerEncoderLayer<T, B, H, M>
impl<T, B, const H: usize, M> Unpin for TransformerEncoderLayer<T, B, H, M>
impl<T, B, const H: usize, M> UnsafeUnpin for TransformerEncoderLayer<T, B, H, M>where
LayerNorm<T, B>: UnsafeUnpin,
MultiHeadAttention<T, B, H, M>: UnsafeUnpin,
FeedForward<T, B>: UnsafeUnpin,
Blanket Implementations§
Source§impl<T> ArchivePointee for T
impl<T> ArchivePointee for T
Source§type ArchivedMetadata = ()
type ArchivedMetadata = ()
The archived version of the pointer metadata for this type.
Source§fn pointer_metadata(
_: &<T as ArchivePointee>::ArchivedMetadata,
) -> <T as Pointee>::Metadata
fn pointer_metadata( _: &<T as ArchivePointee>::ArchivedMetadata, ) -> <T as Pointee>::Metadata
Converts some archived metadata to the pointer metadata for itself.
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> LayoutRaw for T
impl<T> LayoutRaw for T
Source§fn layout_raw(_: <T as Pointee>::Metadata) -> Result<Layout, LayoutError>
fn layout_raw(_: <T as Pointee>::Metadata) -> Result<Layout, LayoutError>
Returns the layout of the type.
Source§impl<T, N1, N2> Niching<NichedOption<T, N1>> for N2
impl<T, N1, N2> Niching<NichedOption<T, N1>> for N2
Source§unsafe fn is_niched(niched: *const NichedOption<T, N1>) -> bool
unsafe fn is_niched(niched: *const NichedOption<T, N1>) -> bool
Returns whether the given value has been niched. Read more
Source§fn resolve_niched(out: Place<NichedOption<T, N1>>)
fn resolve_niched(out: Place<NichedOption<T, N1>>)
Writes data to
out indicating that a T is niched.