Module self_supervised 

Self-Supervised Learning Pretext Tasks.

Pretext tasks generate labels from data itself for representation learning.

Tasks

• Masked Token Prediction (MLM-style)
• Rotation Prediction
• Contrastive Instance Discrimination
• Jigsaw Puzzle

Structs

BYOL

BYOL (Bootstrap Your Own Latent) framework (Grill et al., 2020).

ContrastiveTask

Contrastive Instance Discrimination.

JigsawPuzzle

Jigsaw Puzzle task - shuffle patches and predict permutation.

MaskedPrediction

Masked Token Prediction task. Randomly masks tokens and predicts the original values.

MoCo

MoCo (Momentum Contrast) framework (He et al., 2020).

RotationPrediction

Rotation Prediction for images (0°, 90°, 180°, 270°).

SimCLR

SimCLR framework (Chen et al., 2020).

SimCSE

SimCSE for text embeddings (Gao et al., 2021).