---
title: "Sequence to Sequence Learning with Neural Networks (2014)"
description: "The authors propose an encoder–decoder design. 34.8 BLEU. 31 references, cited by 3,346 papers here."
authors: ["Ilya Sutskever", "Oriol Vinyals", "Quoc V. Le"]
year: 2014
venue: "arXiv:1409.3215"
source: "https://arxiv.org/abs/1409.3215"
doi: "10.48550/arxiv.1409.3215"
arxiv: "1409.3215"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/) / Sequence to Sequence Learning with Neural Networks

# Sequence to Sequence Learning with Neural Networks

**Authors:** [Ilya Sutskever](https://md.research.elamur.ai/free/en/authors/ilya-sutskever/), [Oriol Vinyals](https://md.research.elamur.ai/free/en/authors/oriol-vinyals/) and [Quoc V. Le](https://md.research.elamur.ai/free/en/authors/quoc-viet-le/) · **Year:** 2014 · **Venue:** arXiv:1409.3215 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1409.3215) · [PDF](https://arxiv.org/pdf/1409.3215) · [DOI 10.48550/arxiv.1409.3215](https://doi.org/10.48550/arxiv.1409.3215) · [arXiv 1409.3215](https://arxiv.org/abs/1409.3215) · [OpenAlex](https://openalex.org/W2130942839)

**Citations:** 15,063 citations (OpenAlex, 2026-10-14) · [3346](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard deep networks expect inputs and outputs of fixed size, but tasks like translation pair sequences whose lengths are unknown and differ. The authors propose an encoder–decoder design. One deep LSTM reads the source sentence token by token and compresses it into a single fixed-size vector. A second deep LSTM, acting as a conditional language model, then emits the target sentence word by word until it produces an end-of-sentence symbol. They trained on a cleaned 12M-sentence subset of the WMT'14 English to French data with capped vocabularies. An ensemble of these models, decoded with beam search, reaches 34.8 BLEU. That beats a phrase-based SMT baseline at 33.3, even though unknown words count against the network. Using the network to rerank the baseline's 1000-best lists lifts the score to 36.5. A simple but unexpectedly strong trick is to feed source sentences in reverse word order. The work matters because it showed that a generic, lightly engineered neural model can translate directly and competitively, which launched the seq2seq paradigm.

## Key points
- The architecture uses separate encoder and decoder LSTMs with four layers each. The final model is an ensemble of 5 LSTMs, each with 384M parameters and an 8,000-dimensional state, decoded by a left-to-right beam search.
- Training data is a WMT'14 English to French subset of 12M sentences (348M French words, 304M English words). Vocabularies are 160,000 source words and 80,000 target words, with all other words mapped to "UNK".
- Direct translation scores 34.81 BLEU versus 33.30 for the SMT baseline. Rescoring the baseline's 1000-best lists gives 36.5, which is 3.2 points above the baseline and near the best published result of 37.0.
- Reversing the source words (but not the target words) markedly improved results. With this trick, quality does not drop for sentences under 35 words and drops only slightly on the longest ones.
- PCA projections of the sentence vectors group phrases by meaning. The vectors react to word order but stay fairly stable under active/passive alternation.

## Interesting ideas
- How the input is ordered can make learning easy or hard: reversing the source puts related input and output items close in time, giving the optimizer short-range links to latch onto.
- A new model can earn its keep before it decodes well on its own by reranking the n-best output of an established system, which reliably adds quality.
- Performance is analysed along two separate axes, sentence length and average word-frequency rank, which shows where a model breaks rather than just how well it does overall.
- The representations are probed with paired inputs: the same content in active versus passive voice should end up close, while the same words in a different order should end up apart.
- An explicit end-of-sequence symbol lets one model define probabilities over outputs of every length, with no separate length predictor.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (back): When linearizing a meaning graph for generation, order the nodes so that those realized first in the sentence sit nearest the decoder's start, shortening the dependencies the generator must learn.
- **I2** (tree): Let gparse rescore k-best trees from the perceptron parser, combining the old parser's search with the transformer's judgement.
- **I3** (learn): Add length buckets and rare-word buckets to evaluation reports, so regressions in silver-data training show up in the slices where they occur.
- **I4** (frames): Build test sets of voice alternations that must yield identical who-did-what-to-whom frames, plus word-order swaps that must change the roles, as a negative control against bag-of-words behaviour.
- **I5** (answer): Free-form answer generators can use a learned stop token instead of hand-set length limits, keeping answer length a modelled and inspectable decision.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Ilya Sutskever, Oriol Vinyals and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. arXiv:1409.3215.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sutskever2014sequence,
  title = {{Sequence to Sequence Learning with Neural Networks}},
  author = {Ilya Sutskever, Oriol Vinyals and Quoc V. Le},
  year = {2014},
  doi = {10.48550/arxiv.1409.3215},
  eprint = {1409.3215},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1409.3215}
}
```

- [← previous Neural Machine Translation by Jointly Learning to Align and Translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
- [next → Convolutional Neural Networks for Sentence Classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/)

## References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Auli, M. Galley, C. Quirk, and G. Zweig. Joint language and translation modeling with recurrent neural networks. In EMNLP, 2013. [find](https://scholar.google.com/scholar?q=M%2E+Auli%2C+M%2E+Galley%2C+C%2E+Quirk%2C+and+G%2E+Zweig%2E+Joint+language+and+translation+modeling+with+recurrent+neural+networks%2E+In+EMNLP%2C+2013%2E)
2. D. Bahdanau, K. Cho, and Y. Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473, 2014. [↗](https://arxiv.org/abs/1409.0473)
3. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. [A neural probabilistic language model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/). In Journal of Machine Learning Research, pages 1137–1155, 2003.
4. Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166, 1994. [↗](https://doi.org/10.1109/72.279181)
5. K. Cho, B. Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Arxiv preprint arXiv:1406.[1078, 2014](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/). [↗](https://arxiv.org/abs/1406.1078)
6. D. Ciresan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In CVPR, 2012. [find](https://scholar.google.com/scholar?q=D%2E+Ciresan%2C+U%2E+Meier%2C+and+J%2E+Schmidhuber%2E+Multi%2Dcolumn+deep+neural+networks+for+image+classification%2E+In+CVPR%2C+2012%2E)
7. G. E. Dahl, D. Yu, L. Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing - Special Issue on Deep Learning for Speech and Language Processing, 2012. [find](https://scholar.google.com/scholar?q=G%2E+E%2E+Dahl%2C+D%2E+Yu%2C+L%2E+Deng%2C+and+A%2E+Acero%2E+Context%2Ddependent+pre%2Dtrained+deep+neural+networks+for+large+vocabulary+speech+recognition%2E+IEEE+Transactions+on+Audio%2C+Speech%2C+and+Language+Processing+%2D+Spec)
8. J. Devlin, R. Zbib, Z. Huang, T. Lamar, R. Schwartz, and J. Makhoul. [Fast and robust neural network joint models for statistical machine translation](https://md.research.elamur.ai/free/en/papers/devlin-2014-fast-robust-neural-network-joint/). In ACL, 2014.
9. Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh’s phrase-based machine translation systems for wmt-14. In WMT, 2014. [find](https://scholar.google.com/scholar?q=Nadir+Durrani%2C+Barry+Haddow%2C+Philipp+Koehn%2C+and+Kenneth+Heafield%2E+Edinburgh%E2%80%99s+phrase%2Dbased+machine+translation+systems+for+wmt%2D14%2E+In+WMT%2C+2014%2E)
10. A. Graves. Generating sequences with recurrent neural networks. In Arxiv preprint arXiv:1308.0850, 2013. [↗](https://arxiv.org/abs/1308.0850)
11. A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, 2006. [find](https://scholar.google.com/scholar?q=A%2E+Graves%2C+S%2E+Ferna%CC%81ndez%2C+F%2E+Gomez%2C+and+J%2E+Schmidhuber%2E+Connectionist+temporal+classification%3A+labelling+unsegmented+sequence+data+with+recurrent+neural+networks%2E+In+ICML%2C+2006%2E)
12. K. M. Hermann and P. Blunsom. Multilingual distributed representations without word alignment. In ICLR, 2014. [find](https://scholar.google.com/scholar?q=K%2E+M%2E+Hermann+and+P%2E+Blunsom%2E+Multilingual+distributed+representations+without+word+alignment%2E+In+ICLR%2C+2014%2E)
13. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 2012. [find](https://scholar.google.com/scholar?q=G%2E+Hinton%2C+L%2E+Deng%2C+D%2E+Yu%2C+G%2E+Dahl%2C+A%2E+Mohamed%2C+N%2E+Jaitly%2C+A%2E+Senior%2C+V%2E+Vanhoucke%2C+P%2E+Nguyen%2C+T%2E+Sainath%2C+and+B%2E+Kingsbury%2E+Deep+neural+networks+for+acoustic+modeling+in+speech+recognition%2E+IEEE+Sign)
14. S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Master’s thesis, Institut fur Informatik, Technische Universitat, Munchen, 1991. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter%2E+Untersuchungen+zu+dynamischen+neuronalen+netzen%2E+Master%E2%80%99s+thesis%2C+Institut+fur+Informatik%2C+Technische+Universitat%2C+Munchen%2C+1991%2E)
15. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter%2C+Y%2E+Bengio%2C+P%2E+Frasconi%2C+and+J%2E+Schmidhuber%2E+Gradient+flow+in+recurrent+nets%3A+the+difficulty+of+learning+long%2Dterm+dependencies%2C+2001%2E)
16. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 1997. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+1997%2E)
17. S. Hochreiter and J. Schmidhuber. LSTM can solve hard long time lag problems. 1997. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+LSTM+can+solve+hard+long+time+lag+problems%2E+1997%2E)
18. N. Kalchbrenner and P. Blunsom. [Recurrent continuous translation models](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/). In EMNLP, 2013.
19. A. Krizhevsky, I. Sutskever, and G. E. Hinton. [ImageNet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In NIPS, 2012.
20. Q.V. Le, M.A. Ranzato, R. Monga, M. Devin, K. Chen, G.S. Corrado, J. Dean, and A.Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. [find](https://scholar.google.com/scholar?q=Q%2EV%2E+Le%2C+M%2EA%2E+Ranzato%2C+R%2E+Monga%2C+M%2E+Devin%2C+K%2E+Chen%2C+G%2ES%2E+Corrado%2C+J%2E+Dean%2C+and+A%2EY%2E+Ng%2E+Building+high%2Dlevel+features+using+large+scale+unsupervised+learning%2E+In+ICML%2C+2012%2E)
21. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+L%2E+Bottou%2C+Y%2E+Bengio%2C+and+P%2E+Haffner%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+1998%2E)
22. T. Mikolov. Statistical Language Models based on Neural Networks. PhD thesis, Brno University of Technology, 2012. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2E+Statistical+Language+Models+based+on+Neural+Networks%2E+PhD+thesis%2C+Brno+University+of+Technology%2C+2012%2E)
23. T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur. Recurrent neural network based language model. In INTERSPEECH, pages 1045–1048, 2010. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2C+M%2E+Karafia%CC%81t%2C+L%2E+Burget%2C+J%2E+Cernocky%CC%80%2C+and+S%2E+Khudanpur%2E+Recurrent+neural+network+based+language+model%2E+In+INTERSPEECH%2C+pages+1045%E2%80%931048%2C+2010%2E)
24. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. [BLEU: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL, 2002.
25. R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063, 2012. [↗](https://arxiv.org/abs/1211.5063)
26. J. Pouget-Abadie, D. Bahdanau, B. van Merrienboer, K. Cho, and Y. Bengio. Overcoming the curse of sentence length for neural machine translation using automatic segmentation. arXiv preprint arXiv:1409.1257, 2014. [↗](https://arxiv.org/abs/1409.1257)
27. A. Razborov. On small depth threshold circuits. In Proc. 3rd Scandinavian Workshop on Algorithm Theory, 1992. [find](https://scholar.google.com/scholar?q=A%2E+Razborov%2E+On+small+depth+threshold+circuits%2E+In+Proc%2E+3rd+Scandinavian+Workshop+on+Algorithm+Theory%2C+1992%2E)
28. D. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Nature, 323(6088):533–536, 1986. [find](https://scholar.google.com/scholar?q=D%2E+Rumelhart%2C+G%2E+E%2E+Hinton%2C+and+R%2E+J%2E+Williams%2E+Learning+representations+by+back%2Dpropagating+errors%2E+Nature%2C+323%286088%29%3A533%E2%80%93536%2C+1986%2E)
29. H. Schwenk. University le mans. [http://www-lium.univ-lemans.fr/˜schwenk/cslm_joint_paper/](http://www-lium.univ-lemans.fr/˜schwenk/cslm_joint_paper/), 2014. [Online; accessed 03-September-2014]. [↗](http://www-lium.univ-lemans.fr/˜schwenk/cslm_joint_paper/)
30. M. Sundermeyer, R. Schluter, and H. Ney. LSTM neural networks for language modeling. In INTER-SPEECH, 2010. [find](https://scholar.google.com/scholar?q=M%2E+Sundermeyer%2C+R%2E+Schluter%2C+and+H%2E+Ney%2E+LSTM+neural+networks+for+language+modeling%2E+In+INTER%2DSPEECH%2C+2010%2E)
31. P. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of IEEE, 1990. [find](https://scholar.google.com/scholar?q=P%2E+Werbos%2E+Backpropagation+through+time%3A+what+it+does+and+how+to+do+it%2E+Proceedings+of+IEEE%2C+1990%2E)

*Updated 1 November 2026*