BERT: bidirectional pre-training for language understanding (2018)
On 11 October 2018 Jacob Devlin and colleagues at Google posted the BERT paper on arXiv. A Transformer encoder with masked language modeling reads text in both directions and set new NLP benchmarks. Pretrain-then-fine-tune workflows spread across industry.
On 11 October 2018, Jacob Devlin and colleagues at Google posted BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding on arXiv (1810.04805). The paper argued for a simple recipe: take the encoder half of the Transformer from “Attention is all you need”, pretrain it deeply on plain text, then apply fine-tuning on labelled tasks.
Reading both ways
Earlier language models often processed text strictly left-to-right, the pattern that later GPT-style decoders would scale to extremes in GPT-3. BERT instead used bidirectional context: every token could attend to tokens on both sides. That is awkward if the model is always predicting the next word, because future tokens would leak the answer. BERT’s fix was masked language modeling (BERT / MLM): hide some input tokens at random and train the network to fill them in from surrounding context.
A second auxiliary task, next-sentence prediction, encouraged representations that understood paragraph-level relationships. Together, these objectives produced general-purpose vectors that fine-tuning could adapt to classification, question answering, and named-entity recognition with relatively little labelled data per task.
Benchmarks and adoption
On the GLUE and SQuAD benchmarks of the time, BERT jumped to the top of leaderboards by wide margins. The result mattered beyond academic scores. Search, advertising, and enterprise natural language processing teams already ran on shallow features or task-specific models; BERT offered a shared backbone that improved many pipelines after a fine-tuning pass.
Google released pretrained checkpoints, and the Hugging Face ecosystem soon made “download BERT, fine-tune on your labels” a default workflow. Encoder-only Transformers became the standard for understanding tasks, while decoder-only models carried the generation story forward.
Why it matters
BERT did not replace the scaling narrative that GPT-3 would amplify two years later. It split the field cleanly: bidirectional encoders for classification and retrieval, autoregressive decoders for open-ended text. Both lines descend from the same 2017 Transformer, which is why modern stacks often mix ideas from each.
For practitioners, BERT’s legacy is the pre-train, then fine-tune template on top of a language model objective tuned for context on both sides. That template still underpins embeddings in search indexes, document classifiers, and many enterprise NLP systems—even as chat-oriented LLMs dominate headlines.