Crate caffe2op_sin

source ·


  • | Calculates a sinusoid position encoding | tensor as described in | Takes a 2-D tensor (of size M x K) of positions | as input, the embedding size as an argument, | and outputs a position encoding tensor | of size (M x K x embedding_size). Here | M is typically the max sequence length | and K is typically the batch size. | | The input tensor must satisfy input[m, | 0] == input[m, k] for all k. | | Encoded as amplitude * SIN(pos/alpha^(i/embedding_size)) | if i is even, else amplitude * COS(pos/alpha^(i/embedding_size)). | Here, pos is the position, alpha and | amplitude are tuning parameters, i | is the current dimension for the embedding, | and embedding_size is the number of | total dimensions in the embedding. |