Type Definition dfdx::nn::TransformerEncoder

source · [−]

pub type TransformerEncoder<const MODEL_DIM: usize, const NUM_HEADS: usize, const FF_DIM: usize, const NUM_LAYERS: usize> = Repeated<TransformerEncoderBlock<MODEL_DIM, NUM_HEADS, FF_DIM>, NUM_LAYERS>;

Expand description

Requires Nightly A transformer encoder.

Generics

MODEL_DIM: The size of query/key/value tensors. Given to MultiHeadAttention.
NUM_HEADS: The number of heads in MultiHeadAttention.
FF_DIM: The size of the hidden layer in the feedforward network in TransformerEncoderBlock.
NUM_LAYERS: The number of TransformerEncoderBlock to use. TODO: Doctests