---
title: "Unsupervised Pretraining for Sequence to — Ramachandran 2017"
description: "Seq2seq networks tend to overfit when labeled data is scarce."
authors: ["Prajit Ramachandran", "Peter Liu", "Quoc Le"]
year: 2017
venue: "Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D17-1039/"
doi: "10.18653/v1/d17-1039"
arxiv: "1611.02683"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ramachandran-2017-unsupervised-pretraining-sequence-sequence-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ramachandran-2017-unsupervised-pretraining-sequence-sequence-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/4/) / Unsupervised Pretraining for Sequence to Sequence Learning

# Unsupervised Pretraining for Sequence to Sequence Learning

**Authors:** Prajit Ramachandran, Peter Liu, [Quoc Le](https://md.research.elamur.ai/free/en/authors/quoc-le/) · **Year:** 2017 · **Venue:** Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D17-1039/) · [PDF](https://aclanthology.org/D17-1039.pdf) · [DOI 10.18653/v1/d17-1039](https://doi.org/10.18653/v1/d17-1039) · [arXiv 1611.02683](https://arxiv.org/abs/1611.02683) · [OpenAlex](https://openalex.org/W2555428947)

**Citations:** 240 citations (openalex-snapshot · 2026-10-14) · [72](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Seq2seq networks tend to overfit when labeled data is scarce. The authors propose a simple way to use unlabeled text to address this. They train two language models on monolingual data, one for the source language and one for the target language. These LMs supply the initial weights for the embeddings and first LSTM layer of the encoder and decoder, and for the decoder softmax. The full model is then fine-tuned on parallel or labeled data. During fine-tuning, the LM losses stay in the objective with equal weight, so the pretrained knowledge is not forgotten. Residual connections and attention over both the first and top layers give small additional gains.

On WMT English→German, using News Crawl monolingual data, the approach set a new state of the art. It outperformed phrase-based systems, supervised NMT and backtranslation. For CNN/Daily Mail abstractive summarization, ROUGE scores only matched a prior baseline, but human judges preferred the pretrained system and found less repetition. The work shows that generic LM pretraining can benefit full encoder-decoder models, not just classifiers.

## Key points
- **MT setup:** Parallel data came from WMT'14, filtered by language detection to about 4M sentence pairs, with subword units from 89500 merge operations (≈90k vocabulary). The LMs were 1-layer 4096-unit LSTMs projected to 1024 dimensions, trained for a week on 32 Tesla K40 GPUs over News Crawl corpora of more than a billion tokens each.
- **MT single model:** It reached 24.0 BLEU on newstest2014 and 27.0 on newstest2015. That is 1.3 BLEU above the best single backtranslation system (22.7 / 25.7) and also beats the earlier 4-model ensemble.
- **MT ensemble:** A 5-model ensemble scored 24.7 / 28.1, matching or exceeding a 12-model backtranslation ensemble (23.8 / 27.6 per the extracted table).
- **Ablations:** In translation, pretraining only the encoder cost 1.6 BLEU, while pretraining only the decoder cost 1.0. In summarization the pattern reversed, with encoder pretraining mattering more, which the authors attribute to better optimization over long inputs. Dropping the LM objective hurt both tasks.
- **Human evaluation (200 CNN/Daily Mail documents):** The pretrained system scored higher in 83 cases, tied in 88 and lost in 29 (sign test p \< 0.0001). It also produced repeated phrases significantly less often (McNemar p \< 0.0001).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

72 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 57 not yet.

- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
- [Unsupervised Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://md.research.elamur.ai/free/en/papers/azar-2023-general-theoretical-paradigm-understand-learning/)
- [Semi-Supervised Sequence Modeling with Cross-View Training](https://md.research.elamur.ai/free/en/papers/clark-2018-semi-supervised-sequence-modeling-cross/)
- [Survey of Low-Resource Machine Translation](https://md.research.elamur.ai/free/en/papers/haddow-2022-survey-low-resource-machine-translation/)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/)
- [Learned in Translation: Contextualized Word Vectors](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/)
- [When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?](https://md.research.elamur.ai/free/en/papers/qi-2018-when-why-are-pre-trained/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/)
- [Masked Language Model Scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://md.research.elamur.ai/free/en/papers/zeng-2022-socratic-models-composing-zero-shot/)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ramachandran2017unsupervised,
  title = {{Unsupervised Pretraining for Sequence to Sequence Learning}},
  author = {Ramachandran, Prajit and Liu, Peter and Le, Quoc},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1039},
  eprint = {1611.02683},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1039}
}
```

- [← previous Unsupervised Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/)
- [next → URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors](https://md.research.elamur.ai/free/en/papers/littell-2017-uriel-lang2vec-representing-languages-as/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/4/)

## References

40 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Unsupervised pretraining has been intensively studied in the past years, most notably is the work by Dahl et al. (2012) who found that pretraining with deep belief networks improved feedforward \#\# References pretrained GloVe embedding vectors (Pennington et al., 2014) had more impact. Related to our… [find](https://scholar.google.com/scholar?q=Unsupervised+pretraining+has+been+intensively+studied+in+the+past+years%2C+most+notably+is+the+work+by+Dahl+et+al%2E+%282012%29+who+found+that+pretraining+with+deep+belief+networks+improved+feedforward+%5C%23%5C%23+R)
2. Robert B. Allen. 1987. Several studies on natural language and back-propagation. IEEE First International Conference on Neural Networks. [find](https://scholar.google.com/scholar?q=Robert+B%2E+Allen%2E+1987%2E+Several+studies+on+natural+language+and+back%2Dpropagation%2E+IEEE+First+International+Conference+on+Neural+Networks%2E)
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
4. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In Proc… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Barry+Haddow%2C+Matthias+Huck%2C+Chris+Hokamp%2C+Philipp+Koehn%2C+Varvara+Logacheva%2C+Christof+Monz%2C+Matteo+Negri%2C+Matt+Post%2C+Carolina+Scarton%2C+Lucia+Speci) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2015-findings-2015-workshop-statistical-machine/)
5. William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. 2015. Listen, attend and spell. arXiv preprint arXiv:1508.01211. — [arXiv:1508.01211](https://arxiv.org/abs/1508.01211) [find](https://scholar.google.com/scholar?q=William+Chan%2C+Navdeep+Jaitly%2C+Quoc+V%2E+Le%2C+and+Oriol+Vinyals%2E+2015%2E+Listen%2C+attend+and+spell%2E+arXiv+preprint+arXiv%3A1508%2E01211%2E+%E2%80%94+%5BarXiv%3A1508%2E01211%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1508%2E01211%29)
6. Jianshu Chen, Po-Sen Huang, Xiaodong He, Jianfeng Gao, and Li Deng. 2016. Unsupervised learning of predictors from unpaired input-output samples. abs/1606.04646. — [arXiv:1606.04646](https://arxiv.org/abs/1606.04646) [find](https://scholar.google.com/scholar?q=Jianshu+Chen%2C+Po%2DSen+Huang%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+and+Li+Deng%2E+2016%2E+Unsupervised+learning+of+predictors+from+unpaired+input%2Doutput+samples%2E+abs%2F1606%2E04646%2E+%E2%80%94+%5BarXiv%3A1606%2E04646%5D%28https%3A%2F%2Farxiv%2Eorg)
7. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. arXiv preprint arXiv:1606.04596. [find](https://scholar.google.com/scholar?q=Yong+Cheng%2C+Wei+Xu%2C+Zhongjun+He%2C+Wei+He%2C+Hua+Wu%2C+Maosong+Sun%2C+and+Yang+Liu%2E+2016%2E+Semi%2Dsupervised+learning+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1606%2E04596%2E)
8. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+Van+Merrie%CC%88nboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+RNN+encoder%2Ddecoder+for+statisti)
9. G. E. Dahl, D. Yu, L. Deng, and A. Acero. 2012. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):30–42. [find](https://scholar.google.com/scholar?q=G%2E+E%2E+Dahl%2C+D%2E+Yu%2C+L%2E+Deng%2C+and+A%2E+Acero%2E+2012%2E+Context%2Ddependent+pre%2Dtrained+deep+neural+networks+for+large%2Dvocabulary+speech+recognition%2E+IEEE+Transactions+on+Audio%2C+Speech%2C+and+Language+Processing%2C)
10. Andrew M. Dai and Quoc V. Le. 2015. supervised sequence learning. In NIPS. \#\# Conclusion Semi-Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman-Vural, and Kyunghyun Cho. 2016. Zero-resource translation with multilingual neural machine translation. arXiv preprint arXiv:1606.04164. — [arXiv:1606.04164](https://arxiv.org/abs/1606.04164) [find](https://scholar.google.com/scholar?q=Andrew+M%2E+Dai+and+Quoc+V%2E+Le%2E+2015%2E+supervised+sequence+learning%2E+In+NIPS%2E+%5C%23%5C%23+Conclusion+Semi%2DOrhan+Firat%2C+Baskaran+Sankaran%2C+Yaser+Al%2DOnaizan%2C+Fatos+T%2E+Yarman%2DVural%2C+and+Kyunghyun+Cho%2E+2016%2E+Zero%2Dr)
11. We presented a novel unsupervised pretraining method to improve sequence to sequence learning. The method can aid in both generalization and optimization. Our scheme involves pretraining two language models in the source and target domain, and initializing the embeddings, first LSTM layers, and soft… [find](https://scholar.google.com/scholar?q=We+presented+a+novel+unsupervised+pretraining+method+to+improve+sequence+to+sequence+learning%2E+The+method+can+aid+in+both+generalization+and+optimization%2E+Our+scheme+involves+pretraining+two+language+)
12. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211. [find](https://scholar.google.com/scholar?q=Ian+J+Goodfellow%2C+Mehdi+Mirza%2C+Da+Xiao%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2013%2E+An+empirical+investigation+of+catastrophic+forgetting+in+gradient%2Dbased+neural+networks%2E+arXiv+preprint+arXiv%3A1312%2E621)
13. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535. [find](https://scholar.google.com/scholar?q=Caglar+Gulcehre%2C+Orhan+Firat%2C+Kelvin+Xu%2C+Kyunghyun+Cho%2C+Loic+Barrault%2C+Huei%2DChi+Lin%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2015%2E+On+using+monolingual+corpora+in+neural+machine+translation) [our card](https://md.research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/)
14. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
15. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015a%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+arXiv+preprint+arXiv%3A1511%2E06709%2E)
16. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. Montreal neural machine translation systems for WMT’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Se%CC%81bastien+Jean%2C+Orhan+Firat%2C+Kyunghyun+Cho%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+2015%2E+Montreal+neural+machine+translation+systems+for+WMT%E2%80%9915%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Mach)
17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015b%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+arXiv+preprint+arXiv%3A1508%2E07909%2E)
18. Felix Stahlberg, Eva Hasler, and Bill Byrne. 2016. The edit distance transducer in action: The university of cambridge english-german system at wmt16. In Proceedings of the First Conference on Machine Translation, pages 377–384, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Felix+Stahlberg%2C+Eva+Hasler%2C+and+Bill+Byrne%2E+2016%2E+The+edit+distance+transducer+in+action%3A+The+university+of+cambridge+english%2Dgerman+system+at+wmt16%2E+In+Proceedings+of+the+First+Conference+on+Machine)
19. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+2016%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2E)
20. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner+and+Phil+Blunsom%2E+2013%2E+Recurrent+continuous+translation+models%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/)
21. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NIPS%2E)
22. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR%2E)
23. Subhashini Venugopalan, Lisa Anne Hendricks, Raymond Mooney, and Kate Saenko. 2016. Improving LSTM-based video description with linguistic knowledge mined from text. arXiv preprint arXiv:1604.01729. [find](https://scholar.google.com/scholar?q=Subhashini+Venugopalan%2C+Lisa+Anne+Hendricks%2C+Raymond+Mooney%2C+and+Kate+Saenko%2E+2016%2E+Improving+LSTM%2Dbased+video+description+with+linguistic+knowledge+mined+from+text%2E+arXiv+preprint+arXiv%3A1604%2E01729%2E)
24. Chin-Yew Lin. 2004. ROUGE: a package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004). [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+a+package+for+automatic+evaluation+of+summaries%2E+In+Proceedings+of+the+Workshop+on+Text+Summarization+Branches+Out+%28WAS+2004%29%2E)
25. Philip Williams, Rico Sennrich, Maria Nadejde, Matthias Huck, Barry Haddow, and Ondřej Bojar. 2016. Edinburgh’s statistical machine translation systems for wmt16. In Proceedings of the First Conference on Machine Translation, pages 399– 410, Berlin, Germany. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Philip+Williams%2C+Rico+Sennrich%2C+Maria+Nadejde%2C+Matthias+Huck%2C+Barry+Haddow%2C+and+Ondr%CC%8Cej+Bojar%2E+2016%2E+Edinburgh%E2%80%99s+statistical+machine+translation+systems+for+wmt16%2E+In+Proceedings+of+the+First+Conferen)
26. Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. In ICLR. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Lukasz+Kaiser%2E+2015%2E+Multi%2Dtask+sequence+to+sequence+learning%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S%2E+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+NIPS%2E)
28. Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig. 2016. Achieving human parity in conversational speech recognition. abs/1610.05256. — [arXiv:1610.05256](https://arxiv.org/abs/1610.05256) [find](https://scholar.google.com/scholar?q=Wayne+Xiong%2C+Jasha+Droppo%2C+Xuedong+Huang%2C+Frank+Seide%2C+Mike+Seltzer%2C+Andreas+Stolcke%2C+Dong+Yu%2C+and+Geoffrey+Zweig%2E+2016%2E+Achieving+human+parity+in+conversational+speech+recognition%2E+abs%2F1610%2E05256%2E+%E2%80%94+)
29. Ramesh Nallapati, Bing Xiang, and Bowen Zhou. 2016. Sequence-to-sequence RNNs for text summarization. arXiv preprint arXiv:1602.06023. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bing+Xiang%2C+and+Bowen+Zhou%2E+2016%2E+Sequence%2Dto%2Dsequence+RNNs+for+text+summarization%2E+arXiv+preprint+arXiv%3A1602%2E06023%2E)
30. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction. ACL. [find](https://scholar.google.com/scholar?q=Courtney+Napoles%2C+Matthew+Gormley%2C+and+Benjamin+Van+Durme%2E+2012%2E+Annotated+gigaword%2E+In+Proceedings+of+the+Joint+Workshop+on+Automatic+Knowledge+Base+Construction+and+Web%2Dscale+Knowledge+Extraction%2E+A) [our card](https://md.research.elamur.ai/free/en/papers/napoles-2012-annotated-gigaword/)
31. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329. [find](https://scholar.google.com/scholar?q=Wojciech+Zaremba%2C+Ilya+Sutskever%2C+and+Oriol+Vinyals%2E+2014%2E+Recurrent+neural+network+regularization%2E+arXiv+preprint+arXiv%3A1409%2E2329%2E) [our card](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/)
32. Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Jiajun+Zhang+and+Chengqing+Zong%2E+2016%2E+Exploiting+source%2Dside+monolingual+data+in+neural+machine+translation%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2016-exploiting-source-side-monolingual-data/)
33. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
34. Yu Zhang, William Chan, and Navdeep Jaitly. 2016. Very deep convolutional networks for end-to-end speech recognition. abs/1610.03022. — [arXiv:1610.03022](https://arxiv.org/abs/1610.03022) [find](https://scholar.google.com/scholar?q=Yu+Zhang%2C+William+Chan%2C+and+Navdeep+Jaitly%2E+2016%2E+Very+deep+convolutional+networks+for+end%2Dto%2Dend+speech+recognition%2E+abs%2F1610%2E03022%2E+%E2%80%94+%5BarXiv%3A1610%2E03022%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1610%2E03022%29)
35. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. ICML. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+2013%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+ICML%2E)
36. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Barret+Zoph%2C+Deniz+Yuret%2C+Jonathan+May%2C+and+Kevin+Knight%2E+2016%2E+Transfer+learning+for+low%2Dresource+neural+machine+translation%2E+In+EMNLP%2E)
37. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D%2E+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+EMNLP%2E)
38. Ramon P. Ñeco and Mikel L. Forcada. 1997. Asynchronous translations with recurrent neural nets. Neural Networks. [find](https://scholar.google.com/scholar?q=Ramon+P%2E+N%CC%83eco+and+Mikel+L%2E+Forcada%2E+1997%2E+Asynchronous+translations+with+recurrent+neural+nets%2E+Neural+Networks%2E)
39. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Rafal+Jozefowicz%2C+and+Ilya+Sutskever%2E+2017%2E+Learning+to+generate+reviews+and+discovering+sentiment%2E+arXiv+preprint+arXiv%3A1704%2E01444%2E) [our card](https://md.research.elamur.ai/free/en/papers/radford-2017-learning-generate-reviews-discovering-sentiment/)
40. Hasim Sak, Andrew W. Senior, and Françoise Beaufays. 2014. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH. [find](https://scholar.google.com/scholar?q=Hasim+Sak%2C+Andrew+W%2E+Senior%2C+and+Franc%CC%A7oise+Beaufays%2E+2014%2E+Long+short%2Dterm+memory+recurrent+neural+network+architectures+for+large+scale+acoustic+modeling%2E+In+INTERSPEECH%2E)