Deep Learning Papers

Paper

Language Models are Few-Shot Learners

Showed that scaling a Transformer language model to 175 billion parameters yields few-shot learning: the model performs new tasks from a prompt with no gradient updates. The result that made scale itself the research agenda.

Visit arxiv.org ↗

About

  • Pretraining & Transfer Learning

    Training a large model on a broad corpus once, then adapting it to many downstream tasks — the paradigm behind BERT and GPT.

  • Transformers

    An architecture built entirely from attention and feed-forward layers, dispensing with recurrence — now the default for language and much else.

Authored by

References

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Pretrained a deep bidirectional Transformer with masked-language-modeling, then fine-tuned it to a state-of-the-art on a wide range of NLP tasks. Cemented the pretrain-then-fine-tune paradigm for language.

    note: Takes the pretraining paradigm to few-shot in-context learning. · relation: Applies

    arxiv.org ↗

  • Attention Is All You Need

    "Attention is all you need": an architecture built purely from self-attention and feed-forward layers, no recurrence or convolution. Parallelizable and scalable, it became the substrate for BERT, GPT, and essentially all modern large language models.

    note: Scales the Transformer decoder to 175B parameters. · relation: Builds on

    arxiv.org ↗