Struct dfdx::nn::Transformer
source · [−]pub struct Transformer<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> {
pub encoder: TransformerEncoder<MODEL_DIM, NUM_HEADS, FF_DIM, NUM_ENCODER_LAYERS>,
pub decoder: TransformerDecoder<MODEL_DIM, NUM_HEADS, FF_DIM, NUM_DECODER_LAYERS>,
}
Expand description
Requires Nightly Transformer architecture as described in Attention is all you need.
This is comprised of a TransformerEncoder and a TransformerDecoder.
Generics:
MODEL_DIM
: Size of the input features to the encoder/decoder.NUM_HEADS
: Number of heads for MultiHeadAttention.NUM_ENCODER_LAYERS
: Number of TransformerEncoderBlock to useNUM_DECODER_LAYERS
: Number of TransformerDecoderBlock to useFF_DIM
: Feedforward hidden dimension for both encoder/decoder
Pytorch equivalent:
torch.nn.Transformer(
d_model=MODEL_DIM,
nhead=NUM_HEADS,
num_encoder_layers=NUM_ENCODER_LAYERS,
num_decoder_layers=NUM_DECODER_LAYERS,
dim_feedforward=FF_DIM,
batch_first=True,
)
Fields
encoder: TransformerEncoder<MODEL_DIM, NUM_HEADS, FF_DIM, NUM_ENCODER_LAYERS>
decoder: TransformerDecoder<MODEL_DIM, NUM_HEADS, FF_DIM, NUM_DECODER_LAYERS>
Trait Implementations
sourceimpl<const M: usize, const H: usize, const E: usize, const D: usize, const F: usize> CanUpdateWithGradients for Transformer<M, H, E, D, F>
impl<const M: usize, const H: usize, const E: usize, const D: usize, const F: usize> CanUpdateWithGradients for Transformer<M, H, E, D, F>
sourcefn update<G: GradientProvider>(
&mut self,
grads: &mut G,
unused: &mut UnusedTensors
)
fn update<G: GradientProvider>(
&mut self,
grads: &mut G,
unused: &mut UnusedTensors
)
Updates self given the GradientProvider. When any parameters that
are NOT present in
G
, then this function should
add the tensor’s UniqueId to UnusedTensors. Read moresourceimpl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> Clone for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
impl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> Clone for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
sourcefn clone(
&self
) -> Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
fn clone(
&self
) -> Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
Returns a copy of the value. Read more
1.0.0 · sourcefn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source
. Read moresourceimpl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> Debug for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
impl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> Debug for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
sourceimpl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> Default for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
impl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> Default for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
sourcefn default(
) -> Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
fn default(
) -> Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
Returns the “default value” for a type. Read more
sourceimpl<const M: usize, const H: usize, const E: usize, const D: usize, const F: usize> LoadFromNpz for Transformer<M, H, E, D, F>
impl<const M: usize, const H: usize, const E: usize, const D: usize, const F: usize> LoadFromNpz for Transformer<M, H, E, D, F>
sourceimpl<const M: usize, const H: usize, const E: usize, const D: usize, const F: usize, Src, Tgt> Module<(Src, Tgt)> for Transformer<M, H, E, D, F>where
Src: Tensor<Dtype = f32>,
Tgt: Tensor<Dtype = f32> + PutTape<Src::Tape>,
TransformerEncoder<M, H, F, E>: Module<Src, Output = Src>,
TransformerDecoder<M, H, F, D>: Module<(<Tgt as PutTape<Src::Tape>>::Output, Src::NoTape), Output = <Tgt as PutTape<Src::Tape>>::Output>,
impl<const M: usize, const H: usize, const E: usize, const D: usize, const F: usize, Src, Tgt> Module<(Src, Tgt)> for Transformer<M, H, E, D, F>where
Src: Tensor<Dtype = f32>,
Tgt: Tensor<Dtype = f32> + PutTape<Src::Tape>,
TransformerEncoder<M, H, F, E>: Module<Src, Output = Src>,
TransformerDecoder<M, H, F, D>: Module<(<Tgt as PutTape<Src::Tape>>::Output, Src::NoTape), Output = <Tgt as PutTape<Src::Tape>>::Output>,
sourceimpl<const M: usize, const H: usize, const E: usize, const D: usize, const F: usize, T> ModuleMut<T> for Transformer<M, H, E, D, F>where
Self: Module<T>,
impl<const M: usize, const H: usize, const E: usize, const D: usize, const F: usize, T> ModuleMut<T> for Transformer<M, H, E, D, F>where
Self: Module<T>,
type Output = <Transformer<M, H, E, D, F> as Module<T>>::Output
type Output = <Transformer<M, H, E, D, F> as Module<T>>::Output
The type that this unit produces given
Input
.sourcefn forward_mut(&mut self, t: T) -> Self::Output
fn forward_mut(&mut self, t: T) -> Self::Output
sourceimpl<const M: usize, const H: usize, const E: usize, const D: usize, const F: usize> ResetParams for Transformer<M, H, E, D, F>
impl<const M: usize, const H: usize, const E: usize, const D: usize, const F: usize> ResetParams for Transformer<M, H, E, D, F>
Auto Trait Implementations
impl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> RefUnwindSafe for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
impl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> Send for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
impl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> Sync for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
impl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> Unpin for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
impl<const MODEL_DIM: usize, const NUM_HEADS: usize, const NUM_ENCODER_LAYERS: usize, const NUM_DECODER_LAYERS: usize, const FF_DIM: usize> UnwindSafe for Transformer<MODEL_DIM, NUM_HEADS, NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, FF_DIM>
Blanket Implementations
sourceimpl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more