Concept
Pretraining & Transfer Learning
Training a large model on a broad corpus once, then adapting it to many downstream tasks — the paradigm behind BERT and GPT.
Papers
-
Language Models are Few-Shot Learners
Showed that scaling a Transformer language model to 175 billion parameters yields few-shot learning: the model performs new tasks from a prompt with no gradient updates. The result that made scale itself the research agenda.
arxiv.org ↗ -
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Pretrained a deep bidirectional Transformer with masked-language-modeling, then fine-tuned it to a state-of-the-art on a wide range of NLP tasks. Cemented the pretrain-then-fine-tune paradigm for language.
arxiv.org ↗