Concept
Transformers
An architecture built entirely from attention and feed-forward layers, dispensing with recurrence — now the default for language and much else.
Papers
-
Attention Is All You Need
"Attention is all you need": an architecture built purely from self-attention and feed-forward layers, no recurrence or convolution. Parallelizable and scalable, it became the substrate for BERT, GPT, and essentially all modern large language models.
arxiv.org ↗ -
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Pretrained a deep bidirectional Transformer with masked-language-modeling, then fine-tuned it to a state-of-the-art on a wide range of NLP tasks. Cemented the pretrain-then-fine-tune paradigm for language.
arxiv.org ↗ -
Language Models are Few-Shot Learners
Showed that scaling a Transformer language model to 175 billion parameters yields few-shot learning: the model performs new tasks from a prompt with no gradient updates. The result that made scale itself the research agenda.
arxiv.org ↗