pub struct TransformerDecoderLayer<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default = MoiraiBackend, const H: usize = 8, SelfM: AttentionMask = CausalMask, CrossM: AttentionMask = NullMask> {
pub norm1: LayerNorm<T, B>,
pub self_attn: MultiHeadAttention<T, B, H, SelfM>,
pub dropout1: Dropout,
pub norm2: LayerNorm<T, B>,
pub cross_attn: MultiHeadAttention<T, B, H, CrossM>,
pub dropout2: Dropout,
pub norm3: LayerNorm<T, B>,
pub ffn: FeedForward<T, B>,
pub dropout3: Dropout,
}Expand description
Single Transformer decoder layer.
Pre-LayerNorm sub-layer composition:
x₁ = x + Dropout(SelfAttn(LayerNorm(x)))
x₂ = x₁ + Dropout(CrossAttn(LayerNorm(x₁), memory, memory))
x₃ = x₂ + Dropout(FFN(LayerNorm(x₂)))Fields§
§norm1: LayerNorm<T, B>Pre-LayerNorm before self-attention.
self_attn: MultiHeadAttention<T, B, H, SelfM>Masked self-attention sub-layer.
dropout1: DropoutDropout applied after self-attention.
norm2: LayerNorm<T, B>Pre-LayerNorm before cross-attention.
cross_attn: MultiHeadAttention<T, B, H, CrossM>Cross-attention sub-layer attending to encoder memory.
dropout2: DropoutDropout applied after cross-attention.
norm3: LayerNorm<T, B>Pre-LayerNorm before FFN.
ffn: FeedForward<T, B>Feed-forward sub-layer.
dropout3: DropoutDropout applied after FFN.
Implementations§
Source§impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, SelfM: AttentionMask, CrossM: AttentionMask> TransformerDecoderLayer<T, B, H, SelfM, CrossM>
impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, SelfM: AttentionMask, CrossM: AttentionMask> TransformerDecoderLayer<T, B, H, SelfM, CrossM>
Sourcepub fn new(
d_model: usize,
d_ff: usize,
dropout_p: f64,
) -> Result<Self, InitializationError<B::Error>>where
T: RealScalar,
B: RandomInitOps<T>,
pub fn new(
d_model: usize,
d_ff: usize,
dropout_p: f64,
) -> Result<Self, InitializationError<B::Error>>where
T: RealScalar,
B: RandomInitOps<T>,
Construct a decoder layer.
§Errors
Returns a typed error when attention dimensions are invalid or the selected backend cannot initialize projection weights.
Sourcepub fn forward_decoder(
&self,
tgt: &Var<T, B>,
memory: &Var<T, B>,
) -> Result<Var<T, B>, ModuleError<B::Error>>
pub fn forward_decoder( &self, tgt: &Var<T, B>, memory: &Var<T, B>, ) -> Result<Var<T, B>, ModuleError<B::Error>>
Decode forward pass with cross-attention.
tgt: target inputs[batch, seq_tgt, d_model]memory: encoder outputs[batch, seq_src, d_model]
Source§impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, SelfM: AttentionMask, CrossM: AttentionMask> TransformerDecoderLayer<T, B, H, SelfM, CrossM>
impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, SelfM: AttentionMask, CrossM: AttentionMask> TransformerDecoderLayer<T, B, H, SelfM, CrossM>
Sourcepub fn parameters(&self) -> Vec<Var<T, B>>
pub fn parameters(&self) -> Vec<Var<T, B>>
Collect all trainable decoder-layer parameters.
Sourcepub fn named_parameters(&self) -> Vec<Parameter<T, B>>
pub fn named_parameters(&self) -> Vec<Parameter<T, B>>
Collect trainable parameters with stable hierarchical names.
Trait Implementations§
Source§impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, SelfM: AttentionMask, CrossM: AttentionMask> Clone for TransformerDecoderLayer<T, B, H, SelfM, CrossM>where
LayerNorm<T, B>: Clone,
MultiHeadAttention<T, B, H, SelfM>: Clone,
MultiHeadAttention<T, B, H, CrossM>: Clone,
FeedForward<T, B>: Clone,
Manual Clone impl.
impl<T: FloatElement + AttentionScalar + RealScalar, B: BackendOps<T> + AttentionOps<T> + Default, const H: usize, SelfM: AttentionMask, CrossM: AttentionMask> Clone for TransformerDecoderLayer<T, B, H, SelfM, CrossM>where
LayerNorm<T, B>: Clone,
MultiHeadAttention<T, B, H, SelfM>: Clone,
MultiHeadAttention<T, B, H, CrossM>: Clone,
FeedForward<T, B>: Clone,
Manual Clone impl.
Auto Trait Implementations§
impl<T, B = MoiraiBackend, const H: usize = 8, SelfM = CausalMask, CrossM = NullMask> !Freeze for TransformerDecoderLayer<T, B, H, SelfM, CrossM>
impl<T, B = MoiraiBackend, const H: usize = 8, SelfM = CausalMask, CrossM = NullMask> !RefUnwindSafe for TransformerDecoderLayer<T, B, H, SelfM, CrossM>
impl<T, B = MoiraiBackend, const H: usize = 8, SelfM = CausalMask, CrossM = NullMask> !Sync for TransformerDecoderLayer<T, B, H, SelfM, CrossM>
impl<T, B = MoiraiBackend, const H: usize = 8, SelfM = CausalMask, CrossM = NullMask> !UnwindSafe for TransformerDecoderLayer<T, B, H, SelfM, CrossM>
impl<T, B, const H: usize, SelfM, CrossM> Forget for TransformerDecoderLayer<T, B, H, SelfM, CrossM>
impl<T, B, const H: usize, SelfM, CrossM> Move for TransformerDecoderLayer<T, B, H, SelfM, CrossM>
impl<T, B, const H: usize, SelfM, CrossM> Send for TransformerDecoderLayer<T, B, H, SelfM, CrossM>where
LayerNorm<T, B>: Send,
MultiHeadAttention<T, B, H, SelfM>: Send,
MultiHeadAttention<T, B, H, CrossM>: Send,
FeedForward<T, B>: Send,
impl<T, B, const H: usize, SelfM, CrossM> Unpin for TransformerDecoderLayer<T, B, H, SelfM, CrossM>where
LayerNorm<T, B>: Unpin,
MultiHeadAttention<T, B, H, SelfM>: Unpin,
MultiHeadAttention<T, B, H, CrossM>: Unpin,
FeedForward<T, B>: Unpin,
impl<T, B, const H: usize, SelfM, CrossM> UnsafeUnpin for TransformerDecoderLayer<T, B, H, SelfM, CrossM>where
LayerNorm<T, B>: UnsafeUnpin,
MultiHeadAttention<T, B, H, SelfM>: UnsafeUnpin,
MultiHeadAttention<T, B, H, CrossM>: UnsafeUnpin,
FeedForward<T, B>: UnsafeUnpin,
Blanket Implementations§
Source§impl<T> ArchivePointee for T
impl<T> ArchivePointee for T
Source§type ArchivedMetadata = ()
type ArchivedMetadata = ()
The archived version of the pointer metadata for this type.
Source§fn pointer_metadata(
_: &<T as ArchivePointee>::ArchivedMetadata,
) -> <T as Pointee>::Metadata
fn pointer_metadata( _: &<T as ArchivePointee>::ArchivedMetadata, ) -> <T as Pointee>::Metadata
Converts some archived metadata to the pointer metadata for itself.
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> LayoutRaw for T
impl<T> LayoutRaw for T
Source§fn layout_raw(_: <T as Pointee>::Metadata) -> Result<Layout, LayoutError>
fn layout_raw(_: <T as Pointee>::Metadata) -> Result<Layout, LayoutError>
Returns the layout of the type.
Source§impl<T, N1, N2> Niching<NichedOption<T, N1>> for N2
impl<T, N1, N2> Niching<NichedOption<T, N1>> for N2
Source§unsafe fn is_niched(niched: *const NichedOption<T, N1>) -> bool
unsafe fn is_niched(niched: *const NichedOption<T, N1>) -> bool
Returns whether the given value has been niched. Read more
Source§fn resolve_niched(out: Place<NichedOption<T, N1>>)
fn resolve_niched(out: Place<NichedOption<T, N1>>)
Writes data to
out indicating that a T is niched.