Seq2Seq at NeurIPS 2014: LSTM encoder-decoders for translation
On 10 September 2014 Google researchers posted "Sequence to Sequence Learning with Neural Networks" on arXiv. Presented at NeurIPS 2014 in December, the LSTM encoder-decoder became the reference for neural translation before Transformers with attention reshaped language processing.
On 10 September 2014, Ilya Sutskever, Oriol Vinyals, and Quoc Le at Google posted Sequence to Sequence Learning with Neural Networks on arXiv (1409.3215). The paper trained a deep sequence-to-sequence (Seq2Seq) model: one LSTM encoder read a source sentence into a fixed vector, and a second LSTM decoder generated the translation word by word. NeurIPS 2014 in December made it the reference architecture for neural network machine translation until Transformers with attention took over.
Encoder-decoder with recurrence
Earlier language models often predicted the next token from a fixed window. Seq2Seq instead mapped an entire input sequence to an output sequence of different length, which is exactly what translation requires. Stacking LSTMs on both sides let deep learning reuse the same backpropagation toolkit that AlexNet had popularised for vision, now applied to pairs of sentences in supervised learning.
The authors reported large gains on English-to-French translation compared with earlier neural baselines, using reversed source sentences as a simple trick to put more short-term context near the decoder. The idea spread quickly: speech, summarisation, and dialogue systems all adopted encoder-decoder templates built from recurrent nets.
Before attention
Seq2Seq compressed the whole source sentence into one vector, which became a bottleneck for long passages. Follow-on work added attention so decoders could look back at encoder states, a stepping stone toward the pure attention stack in the Transformer article. NeurIPS later honoured the 2014 paper alongside GANs in its Test of Time awards, a sign of how central encoder-decoder NLP had become.
Why it matters
Seq2Seq showed that end-to-end neural networks could beat strong statistical pipelines on translation without hand-built features. It linked the neural language model line to modern LLMs, and it trained a generation of researchers who would soon replace LSTMs with self-attention. The architecture is still the mental model behind many machine learning tutorials on sequence mapping, even when the implementation is a Transformer.