Deep Learning Papers

Concept

Attention Mechanism

Letting a model weight over all positions of its input, so it can focus on the relevant parts rather than a fixed-size summary.

Papers

  • Attention Is All You Need

    "Attention is all you need": an architecture built purely from self-attention and feed-forward layers, no recurrence or convolution. Parallelizable and scalable, it became the substrate for BERT, GPT, and essentially all modern large language models.

    arxiv.org ↗

  • Neural Machine Translation by Jointly Learning to Align and Translate

    Introduced attention to neural machine translation: instead of squeezing the source into one vector, the decoder learns to align to and weight over all source positions — the idea the Transformer would later build its whole architecture on.

    arxiv.org ↗