"Attention is all you need": the Transformer arrives (2017)
A 2017 paper replaced the sequential machinery of earlier networks with pure attention. The Transformer became the architecture behind almost every large language model since.
In June 2017, a team at Google published a paper with an unusually confident title: “Attention Is All You Need.” It introduced the Transformer, and it turned out to be one of the most consequential architectures in the history of AI.
The problem it solved
Before the Transformer, the leading models for language read text the way people read a sentence out loud: one word after another, in order. This sequential processing made them slow to train and prone to “forgetting” information from early in a long passage.
The Transformer threw out that sequential machinery. Instead it used a mechanism called self-attention, which lets every word in a sentence look directly at every other word and decide which ones matter. Because these comparisons can happen all at once rather than one step at a time, Transformers train far more efficiently on modern hardware, and they handle long-range connections in text much better.
Why it changed everything
The original paper was about machine translation, a fairly narrow goal. But the architecture generalised almost embarrassingly well. Within a couple of years it underpinned the models that would define the next era of natural language processing:
- Encoder-style Transformers powered understanding tasks like search and classification.
- Decoder-style Transformers powered generation, the family that leads directly to modern large language models.
Crucially, Transformers scale. Make the neural network bigger, feed it more text, give it more compute, and it keeps getting better in fairly predictable ways. That property is what made the “just scale it up” strategy of the 2020s viable.
Why it matters
The Transformer is the hinge between the deep-learning breakthroughs of the early 2010s and the language-model boom that followed. Nearly every system people now call “an AI”, including the model behind GPT-3 and its successors, is a Transformer at heart.
It is rare for a single architecture to dominate a field for years. The Transformer has done exactly that, and the phrase from its title has become one of the most quoted lines in the discipline.