training
Self-Supervised Learning
A learning paradigm where the model generates its own labels from unlabeled data, typically by predicting missing or corrupted parts of the input. Self-supervised learning enables pre-training on massive unlabeled datasets before fine-tuning on smaller labeled datasets.
In practice
BERT is trained with self-supervision by masking random words in sentences and learning to predict them.