--- title: "MASS: Masked Sequence to Sequence Pre-training — Song 2019" description: "Pre-training methods like BERT work well for language understanding, but they train only an encoder." authors: ["Kaitao Song", "Xu Tan", "Tao Qin", "Jianfeng Lu", "Tie-Yan Liu"] year: 2019 venue: "arXiv:1905.02450" source: "https://arxiv.org/abs/1905.02450" doi: "10.48550/arxiv.1905.02450" arxiv: "1905.02450" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / MASS: Masked Sequence to Sequence Pre-training for Language Generation # MASS: Masked Sequence to Sequence Pre-training for Language Generation **Authors:** [Kaitao Song](https://md.research.elamur.ai/free/en/authors/kaitao-song/), [Xu Tan](https://md.research.elamur.ai/free/en/authors/xu-tan/), [Tao Qin](https://md.research.elamur.ai/free/en/authors/tao-qin/), [Jianfeng Lu](https://md.research.elamur.ai/free/en/authors/jianfeng-lu/), [Tie-Yan Liu](https://md.research.elamur.ai/free/en/authors/tie-yan-liu/) · **Year:** 2019 · **Venue:** arXiv:1905.02450 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1905.02450) · [PDF](https://arxiv.org/pdf/1905.02450) · [DOI 10.48550/arxiv.1905.02450](https://doi.org/10.48550/arxiv.1905.02450) · [arXiv 1905.02450](https://arxiv.org/abs/1905.02450) · [OpenAlex](https://openalex.org/W2944815030) **Citations:** 733 citations (OpenAlex, 2026-10-14) · [376](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/cited-by/) in our library, of the 140884 papers whose bibliographies we read ## Summary Pre-training methods like BERT work well for language understanding, but they train only an encoder. That makes them a poor fit for generation tasks, which use a full encoder-decoder. MASS addresses this with one pre-training objective that trains both halves together on unlabeled text. A contiguous span of a sentence is replaced with [M] tokens before it goes to the encoder. The decoder then has to reconstruct only that span. On the decoder side, tokens that the encoder could see are hidden, so the decoder has to draw on the encoder's representations. The authors pre-train one Transformer on WMT monolingual corpora covering several languages. They fine-tune it for three tasks across eight datasets: machine translation (unsupervised and low-resource), abstractive summarization and dialogue response generation. MASS beats models with no pre-training and earlier pre-training approaches in both zero-resource and low-resource settings. Its unsupervised English-French result is better than an early supervised attention-based NMT system. This suggests that pre-training the whole sequence-to-sequence model pays off for data-poor generation tasks. ## Key points - Unsupervised NMT on English-French reaches 37.5 BLEU, a state-of-the-art result. That is more than 4 BLEU above XLM (Lample & Conneau, 2019) on En-Fr and about 1 BLEU above it on Fr-En. - Evaluation data: WMT14 English-French, WMT16 English-German and WMT16 English-Romanian for translation, Gigaword for summarization and the Cornell Movie Dialog corpus for response generation. - The masked span length k is the key hyperparameter. Setting k = 1 matches BERT-style masked language modeling, and setting k = sentence length matches standard language modeling. Both pre-training perplexity and downstream scores are best when k is about 50% of the sentence, which became the default. - Ablation on unsupervised En-Fr: masking scattered tokens instead of a contiguous span ("Discrete") gives 36.9 BLEU. Feeding the decoder all tokens ("Feed") gives 35.3 BLEU. Full MASS gives 37.5. - For unsupervised NMT, fine-tuning uses only a back-translation loss on monolingual data, with no extra denoising auto-encoder loss. The code is available at github.com/microsoft/MASS. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{song2019mass, title = {{MASS: Masked Sequence to Sequence Pre-training for Language Generation}}, author = {Song, Kaitao and Tan, Xu and Qin, Tao and Lu, Jianfeng and Liu, Tie-Yan}, year = {2019}, doi = {10.48550/arxiv.1905.02450}, eprint = {1905.02450}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1905.02450} } ``` - [← previous Predicting the Type and Target of Offensive Posts in Social Media](https://md.research.elamur.ai/free/en/papers/zampieri-2019-predicting-type-target-offensive-posts/) - [next → SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://md.research.elamur.ai/free/en/papers/zampieri-2019-semeval-2019-task-6-identifying/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/) ## References 59 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ando, R. K. and Zhang, T. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(Nov):1817– 1853, 2005. [find](https://scholar.google.com/scholar?q=Ando%2C+R%2E+K%2E+and+Zhang%2C+T%2E+A+framework+for+learning+predictive+structures+from+multiple+tasks+and+unlabeled+data%2E+Journal+of+Machine+Learning+Research%2C+6%28Nov%29%3A1817%E2%80%93+1853%2C+2005%2E) 2. Artetxe, M., Labaka, G., Agirre, E., and Cho, K. Unsupervised neural machine translation. CoRR, 2017. [find](https://scholar.google.com/scholar?q=Artetxe%2C+M%2E%2C+Labaka%2C+G%2E%2C+Agirre%2C+E%2E%2C+and+Cho%2C+K%2E+Unsupervised+neural+machine+translation%2E+CoRR%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/) 3. Ayana, Shen, S., Liu, Z., and Sun, M. Neural headline generation with minimum risk training. ArXiv, 2016. [find](https://scholar.google.com/scholar?q=Ayana%2C+Shen%2C+S%2E%2C+Liu%2C+Z%2E%2C+and+Sun%2C+M%2E+Neural+headline+generation+with+minimum+risk+training%2E+ArXiv%2C+2016%2E) 4. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. ICLR 2015, 2015a. [find](https://scholar.google.com/scholar?q=Bahdanau%2C+D%2E%2C+Cho%2C+K%2E%2C+and+Bengio%2C+Y%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+ICLR+2015%2C+2015a%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) 5. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. 2015b. [find](https://scholar.google.com/scholar?q=Bahdanau%2C+D%2E%2C+Cho%2C+K%2E%2C+and+Bengio%2C+Y%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+2015b%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) 6. Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003. [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E%2C+Ducharme%2C+R%2E%2C+Vincent%2C+P%2E%2C+and+Jauvin%2C+C%2E+A+neural+probabilistic+language+model%2E+Journal+of+machine+learning+research%2C+3%28Feb%29%3A1137%E2%80%931155%2C+2003%2E) 7. Blitzer, J., McDonald, R., and Pereira, F. Domain adaptation with structural correspondence learning. In EMNLP, pp. 120–128. Association for Computational Linguistics, 2006. [find](https://scholar.google.com/scholar?q=Blitzer%2C+J%2E%2C+McDonald%2C+R%2E%2C+and+Pereira%2C+F%2E+Domain+adaptation+with+structural+correspondence+learning%2E+In+EMNLP%2C+pp%2E+120%E2%80%93128%2E+Association+for+Computational+Linguistics%2C+2006%2E) [our card](https://md.research.elamur.ai/free/en/papers/blitzer-2006-domain-adaptation-structural-correspondence-learning/) 8. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. In EMNLP, 2015. [find](https://scholar.google.com/scholar?q=Bowman%2C+S%2E+R%2E%2C+Angeli%2C+G%2E%2C+Potts%2C+C%2E%2C+and+Manning%2C+C%2E+D%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+EMNLP%2C+2015%2E) 9. Brown, P. F., Desouza, P. V., Mercer, R. L., Pietra, V. J. D., and Lai, J. C. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–479, 1992. [find](https://scholar.google.com/scholar?q=Brown%2C+P%2E+F%2E%2C+Desouza%2C+P%2E+V%2E%2C+Mercer%2C+R%2E+L%2E%2C+Pietra%2C+V%2E+J%2E+D%2E%2C+and+Lai%2C+J%2E+C%2E+Class%2Dbased+n%2Dgram+models+of+natural+language%2E+Computational+linguistics%2C+18%284%29%3A467%E2%80%93479%2C+1992%2E) 10. Cho, K., van Merrienboer, B., Gülçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014. [find](https://scholar.google.com/scholar?q=Cho%2C+K%2E%2C+van+Merrienboer%2C+B%2E%2C+Gu%CC%88lc%CC%A7ehre%2C+C%CC%A7%2E%2C+Bahdanau%2C+D%2E%2C+Bougares%2C+F%2E%2C+Schwenk%2C+H%2E%2C+and+Bengio%2C+Y%2E+Learning+phrase+representations+using+RNN+encoder%2Ddecoder+for+statistical+machine+translation%2E+In) 11. Collobert, R. and Weston, J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, pp. 160–167. ACM, 2008. [find](https://scholar.google.com/scholar?q=Collobert%2C+R%2E+and+Weston%2C+J%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+ICML%2C+pp%2E+160%E2%80%93167%2E+ACM%2C+2008%2E) 12. Dai, A. M. and Le, Q. V. Semi-supervised sequence learning. In NIPS, pp. 3079–3087, 2015. [find](https://scholar.google.com/scholar?q=Dai%2C+A%2E+M%2E+and+Le%2C+Q%2E+V%2E+Semi%2Dsupervised+sequence+learning%2E+In+NIPS%2C+pp%2E+3079%E2%80%933087%2C+2015%2E) 13. Danescu-Niculescu-Mizil, C. and Lee, L. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In ACL Workshop, 2011. [find](https://scholar.google.com/scholar?q=Danescu%2DNiculescu%2DMizil%2C+C%2E+and+Lee%2C+L%2E+Chameleons+in+imagined+conversations%3A+A+new+approach+to+understanding+coordination+of+linguistic+style+in+dialogs%2E+In+ACL+Workshop%2C+2011%2E) [our card](https://md.research.elamur.ai/free/en/papers/mizil-2011-chameleons-imagined-conversations-new-approach/) 14. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, 2018. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Lee%2C+K%2E%2C+and+Toutanova%2C+K%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+CoRR%2C+2018%2E) 15. Fedus, W., Goodfellow, I., and Dai, A. Maskgan: Better text generation via filling in the . In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Fedus%2C+W%2E%2C+Goodfellow%2C+I%2E%2C+and+Dai%2C+A%2E+Maskgan%3A+Better+text+generation+via+filling+in+the+%2E+In+ICLR%2C+2018%2E) 16. Firat, O., Sankaran, B., Al-Onaizan, Y., Vural, F. T. Y., and Cho, K. Zero-resource translation with multi-lingual neural machine translation. In EMNLP, pp. 268–277, 2016. [find](https://scholar.google.com/scholar?q=Firat%2C+O%2E%2C+Sankaran%2C+B%2E%2C+Al%2DOnaizan%2C+Y%2E%2C+Vural%2C+F%2E+T%2E+Y%2E%2C+and+Cho%2C+K%2E+Zero%2Dresource+translation+with+multi%2Dlingual+neural+machine+translation%2E+In+EMNLP%2C+pp%2E+268%E2%80%93277%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/firat-2016-zero-resource-translation-multi-lingual/) 17. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. Convolutional sequence to sequence learning. In ICML, volume 70, pp. 1243–1252, 2017. [find](https://scholar.google.com/scholar?q=Gehring%2C+J%2E%2C+Auli%2C+M%2E%2C+Grangier%2C+D%2E%2C+Yarats%2C+D%2E%2C+and+Dauphin%2C+Y%2E+N%2E+Convolutional+sequence+to+sequence+learning%2E+In+ICML%2C+volume+70%2C+pp%2E+1243%E2%80%931252%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/) 18. Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pp. 580–587, 2014. [find](https://scholar.google.com/scholar?q=Girshick%2C+R%2E%2C+Donahue%2C+J%2E%2C+Darrell%2C+T%2E%2C+and+Malik%2C+J%2E+Rich+feature+hierarchies+for+accurate+object+detection+and+semantic+segmentation%2E+In+CVPR%2C+pp%2E+580%E2%80%93587%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/girshick-2013-rich-feature-hierarchies-accurate-object/) 19. Graff, David, Kong, Junbo, Chen, Ke, Maeda, and Kazuaki. English gigaword. In Linguistic Data Consortium, 2003. [find](https://scholar.google.com/scholar?q=Graff%2C+David%2C+Kong%2C+Junbo%2C+Chen%2C+Ke%2C+Maeda%2C+and+Kazuaki%2E+English+gigaword%2E+In+Linguistic+Data+Consortium%2C+2003%2E) 20. Hassan, H., Aue, A., Chen, C., Chowdhary, V., Clark, J., Federmann, C., Huang, X., Junczys-Dowmunt, M., Lewis, W., Li, M., et al. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567, 2018. [find](https://scholar.google.com/scholar?q=Hassan%2C+H%2E%2C+Aue%2C+A%2E%2C+Chen%2C+C%2E%2C+Chowdhary%2C+V%2E%2C+Clark%2C+J%2E%2C+Federmann%2C+C%2E%2C+Huang%2C+X%2E%2C+Junczys%2DDowmunt%2C+M%2E%2C+Lewis%2C+W%2E%2C+Li%2C+M%2E%2C+et+al%2E+Achieving+human+parity+on+automatic+chinese+to+english+news+translatio) [our card](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/) 21. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, pp. 770–778, 2016. [find](https://scholar.google.com/scholar?q=He%2C+K%2E%2C+Zhang%2C+X%2E%2C+Ren%2C+S%2E%2C+and+Sun%2C+J%2E+Deep+residual+learning+for+image+recognition%2E+In+CVPR%2C+pp%2E+770%E2%80%93778%2C+2016%2E) 22. He, T., Tan, X., Xia, Y., He, D., Qin, T., Chen, Z., and Liu, T.-Y. Layer-wise coordination between encoder and decoder for neural machine translation. In Advances in Neural Information Processing Systems, pp. 7944–7954, 2018. [find](https://scholar.google.com/scholar?q=He%2C+T%2E%2C+Tan%2C+X%2E%2C+Xia%2C+Y%2E%2C+He%2C+D%2E%2C+Qin%2C+T%2E%2C+Chen%2C+Z%2E%2C+and+Liu%2C+T%2E%2DY%2E+Layer%2Dwise+coordination+between+encoder+and+decoder+for+neural+machine+translation%2E+In+Advances+in+Neural+Information+Processing+Sys) 23. Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. In ACL, volume 1, pp. 328– 339, 2018. [find](https://scholar.google.com/scholar?q=Howard%2C+J%2E+and+Ruder%2C+S%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+ACL%2C+volume+1%2C+pp%2E+328%E2%80%93+339%2C+2018%2E) 24. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR, 2015. [find](https://scholar.google.com/scholar?q=Kingma%2C+D%2E+P%2E+and+Ba%2C+J%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR%2C+2015%2E) 25. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. Skip-thought vectors. In NIPS, pp. 3294–3302, 2015. [find](https://scholar.google.com/scholar?q=Kiros%2C+R%2E%2C+Zhu%2C+Y%2E%2C+Salakhutdinov%2C+R%2E+R%2E%2C+Zemel%2C+R%2E%2C+Urtasun%2C+R%2E%2C+Torralba%2C+A%2E%2C+and+Fidler%2C+S%2E+Skip%2Dthought+vectors%2E+In+NIPS%2C+pp%2E+3294%E2%80%933302%2C+2015%2E) 26. Lample, G. and Conneau, A. Cross-lingual language model pretraining. CoRR, abs/1901.07291, 2019. [find](https://scholar.google.com/scholar?q=Lample%2C+G%2E+and+Conneau%2C+A%2E+Cross%2Dlingual+language+model+pretraining%2E+CoRR%2C+abs%2F1901%2E07291%2C+2019%2E) 27. Lample, G., Conneau, A., Denoyer, L., and Ranzato, M. Unsupervised machine translation using monolingual corpora only. CoRR, 2017. [find](https://scholar.google.com/scholar?q=Lample%2C+G%2E%2C+Conneau%2C+A%2E%2C+Denoyer%2C+L%2E%2C+and+Ranzato%2C+M%2E+Unsupervised+machine+translation+using+monolingual+corpora+only%2E+CoRR%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/) 28. Lample, G., Ott, M., Conneau, A., Denoyer, L., and Ranzato, M. Phrase-based & neural unsupervised machine translation. In EMNLP, pp. 5039–5049, 2018. [find](https://scholar.google.com/scholar?q=Lample%2C+G%2E%2C+Ott%2C+M%2E%2C+Conneau%2C+A%2E%2C+Denoyer%2C+L%2E%2C+and+Ranzato%2C+M%2E+Phrase%2Dbased+%26+neural+unsupervised+machine+translation%2E+In+EMNLP%2C+pp%2E+5039%E2%80%935049%2C+2018%2E) 29. Le, Q. and Mikolov, T. Distributed representations of sentences and documents. In ICML, pp. 1188–1196, 2014. [find](https://scholar.google.com/scholar?q=Le%2C+Q%2E+and+Mikolov%2C+T%2E+Distributed+representations+of+sentences+and+documents%2E+In+ICML%2C+pp%2E+1188%E2%80%931196%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/le-2014-distributed-representations-sentences-documents/) 30. Leng, Y., Tan, X., Qin, T., Li, X.-Y., and Liu, T.-Y. Unsupervised pivot translation for distant languages. In ACL, 2019. [find](https://scholar.google.com/scholar?q=Leng%2C+Y%2E%2C+Tan%2C+X%2E%2C+Qin%2C+T%2E%2C+Li%2C+X%2E%2DY%2E%2C+and+Liu%2C+T%2E%2DY%2E+Unsupervised+pivot+translation+for+distant+languages%2E+In+ACL%2C+2019%2E) 31. Logeswaran, L. and Lee, H. An efficient framework for learning sentence representations. CORR, 2018. [find](https://scholar.google.com/scholar?q=Logeswaran%2C+L%2E+and+Lee%2C+H%2E+An+efficient+framework+for+learning+sentence+representations%2E+CORR%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/logeswaran-2018-efficient-framework-learning-sentence-representations/) 32. McCann, B., Bradbury, J., Xiong, C., and Socher, R. Learned in translation: Contextualized word vectors. In NIPS, pp. 6294–6305, 2017. [find](https://scholar.google.com/scholar?q=McCann%2C+B%2E%2C+Bradbury%2C+J%2E%2C+Xiong%2C+C%2E%2C+and+Socher%2C+R%2E+Learned+in+translation%3A+Contextualized+word+vectors%2E+In+NIPS%2C+pp%2E+6294%E2%80%936305%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/) 33. Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., and Khudanpur, S. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Karafia%CC%81t%2C+M%2E%2C+Burget%2C+L%2E%2C+C%CC%8Cernocky%CC%80%2C+J%2E%2C+and+Khudanpur%2C+S%2E+Recurrent+neural+network+based+language+model%2E+In+Eleventh+Annual+Conference+of+the+International+Speech+Communication+Associa) 34. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. In NIPS, pp. 3111–3119, 2013. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Sutskever%2C+I%2E%2C+Chen%2C+K%2E%2C+Corrado%2C+G%2E+S%2E%2C+and+Dean%2C+J%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+NIPS%2C+pp%2E+3111%E2%80%933119%2C+2013%2E) 35. Ouyang, W., Li, H., Zeng, X., and Wang, X. Learning deep representation with large-scale attributes. In CVPR, pp. 1895–1903, 2015. [find](https://scholar.google.com/scholar?q=Ouyang%2C+W%2E%2C+Li%2C+H%2E%2C+Zeng%2C+X%2E%2C+and+Wang%2C+X%2E+Learning+deep+representation+with+large%2Dscale+attributes%2E+In+CVPR%2C+pp%2E+1895%E2%80%931903%2C+2015%2E) 36. Pennington, J., Socher, R., and Manning, C. Glove: Global vectors for word representation. In EMNLP, pp. 1532– 1543, 2014. [find](https://scholar.google.com/scholar?q=Pennington%2C+J%2E%2C+Socher%2C+R%2E%2C+and+Manning%2C+C%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+EMNLP%2C+pp%2E+1532%E2%80%93+1543%2C+2014%2E) 37. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In NAACL, volume 1, pp. 2227–2237, 2018. [find](https://scholar.google.com/scholar?q=Peters%2C+M%2E%2C+Neumann%2C+M%2E%2C+Iyyer%2C+M%2E%2C+Gardner%2C+M%2E%2C+Clark%2C+C%2E%2C+Lee%2C+K%2E%2C+and+Zettlemoyer%2C+L%2E+Deep+contextualized+word+representations%2E+In+NAACL%2C+volume+1%2C+pp%2E+2227%E2%80%932237%2C+2018%2E) 38. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Narasimhan%2C+K%2E%2C+Salimans%2C+T%2E%2C+and+Sutskever%2C+I%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E) 39. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. CoRR, 2016. [find](https://scholar.google.com/scholar?q=Rajpurkar%2C+P%2E%2C+Zhang%2C+J%2E%2C+Lopyrev%2C+K%2E%2C+and+Liang%2C+P%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+CoRR%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) 40. Ramachandran, P., Liu, P. J., and Le, Q. V. Unsupervised pretraining for sequence to sequence learning. CoRR, abs/1611.02683, 2016. [find](https://scholar.google.com/scholar?q=Ramachandran%2C+P%2E%2C+Liu%2C+P%2E+J%2E%2C+and+Le%2C+Q%2E+V%2E+Unsupervised+pretraining+for+sequence+to+sequence+learning%2E+CoRR%2C+abs%2F1611%2E02683%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/ramachandran-2017-unsupervised-pretraining-sequence-sequence-learning/) 41. Ren, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y. Almost unsupervised text to speech and automatic speech recognition. In ICML, 2019. [find](https://scholar.google.com/scholar?q=Ren%2C+Y%2E%2C+Tan%2C+X%2E%2C+Qin%2C+T%2E%2C+Zhao%2C+S%2E%2C+Zhao%2C+Z%2E%2C+and+Liu%2C+T%2E%2DY%2E+Almost+unsupervised+text+to+speech+and+automatic+speech+recognition%2E+In+ICML%2C+2019%2E) 42. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In ACL, volume 1, pp. 1715–1725, 2016. [find](https://scholar.google.com/scholar?q=Sennrich%2C+R%2E%2C+Haddow%2C+B%2E%2C+and+Birch%2C+A%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+ACL%2C+volume+1%2C+pp%2E+1715%E2%80%931725%2C+2016%2E) 43. Shang, L., Lu, Z., and Li, H. Neural responding machine for short-text conversation. In ACL, volume 1, pp. 1577– 1586, 2015. [find](https://scholar.google.com/scholar?q=Shang%2C+L%2E%2C+Lu%2C+Z%2E%2C+and+Li%2C+H%2E+Neural+responding+machine+for+short%2Dtext+conversation%2E+In+ACL%2C+volume+1%2C+pp%2E+1577%E2%80%93+1586%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/shang-2015-neural-responding-machine-short-text/) 44. Shen, Y., Tan, X., He, D., Qin, T., and Liu, T.-Y. Dense information flow for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1294–1303, June 201… [find](https://scholar.google.com/scholar?q=Shen%2C+Y%2E%2C+Tan%2C+X%2E%2C+He%2C+D%2E%2C+Qin%2C+T%2E%2C+and+Liu%2C+T%2E%2DY%2E+Dense+information+flow+for+neural+machine+translation%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Com) 45. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pp. 1631–1642, 2013. [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Perelygin%2C+A%2E%2C+Wu%2C+J%2E%2C+Chuang%2C+J%2E%2C+Manning%2C+C%2E+D%2E%2C+Ng%2C+A%2E%2C+and+Potts%2C+C%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+EMNLP%2C+pp%2E+1631%E2%80%931642%2C+2013%2E) 46. Song, K., Tan, X., He, D., Lu, J., Qin, T., and Liu, T.-Y. Double path networks for sequence to sequence learning. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 3064–3074, 2018. [find](https://scholar.google.com/scholar?q=Song%2C+K%2E%2C+Tan%2C+X%2E%2C+He%2C+D%2E%2C+Lu%2C+J%2E%2C+Qin%2C+T%2E%2C+and+Liu%2C+T%2E%2DY%2E+Double+path+networks+for+sequence+to+sequence+learning%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+pp%2E) 47. Suzuki, J. and Nagata, M. Cutting-off redundant repeating generations for neural abstractive summarization. In ACL, pp. 291–297, 2017. [find](https://scholar.google.com/scholar?q=Suzuki%2C+J%2E+and+Nagata%2C+M%2E+Cutting%2Doff+redundant+repeating+generations+for+neural+abstractive+summarization%2E+In+ACL%2C+pp%2E+291%E2%80%93297%2C+2017%2E) 48. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In CVPR, pp. 1–9, 2015. [find](https://scholar.google.com/scholar?q=Szegedy%2C+C%2E%2C+Liu%2C+W%2E%2C+Jia%2C+Y%2E%2C+Sermanet%2C+P%2E%2C+Reed%2C+S%2E%2C+Anguelov%2C+D%2E%2C+Erhan%2C+D%2E%2C+Vanhoucke%2C+V%2E%2C+and+Rabinovich%2C+A%2E+Going+deeper+with+convolutions%2E+In+CVPR%2C+pp%2E+1%E2%80%939%2C+2015%2E) 49. Tan, X., Ren, Y., He, D., Qin, T., and Liu, T.-Y. Multilingual neural machine translation with knowledge distillation. In ICLR, 2019. [find](https://scholar.google.com/scholar?q=Tan%2C+X%2E%2C+Ren%2C+Y%2E%2C+He%2C+D%2E%2C+Qin%2C+T%2E%2C+and+Liu%2C+T%2E%2DY%2E+Multilingual+neural+machine+translation+with+knowledge+distillation%2E+In+ICLR%2C+2019%2E) 50. Tjong Kim Sang, E. F. and De Meulder, F. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In NAACL, pp. 142–147. Association for Computational Linguistics, 2003. [find](https://scholar.google.com/scholar?q=Tjong+Kim+Sang%2C+E%2E+F%2E+and+De+Meulder%2C+F%2E+Introduction+to+the+conll%2D2003+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+In+NAACL%2C+pp%2E+142%E2%80%93147%2E+Association+for+Computational+Linguistics%2C+20) 51. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS, pp. 6000–6010, 2017. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E%2C+and+Polosukhin%2C+I%2E+Attention+is+all+you+need%2E+In+NIPS%2C+pp%2E+6000%E2%80%936010%2C+2017%2E) 52. Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. In ICML, pp. 1096–1103. ACM, 2008. [find](https://scholar.google.com/scholar?q=Vincent%2C+P%2E%2C+Larochelle%2C+H%2E%2C+Bengio%2C+Y%2E%2C+and+Manzagol%2C+P%2E%2DA%2E+Extracting+and+composing+robust+features+with+denoising+autoencoders%2E+In+ICML%2C+pp%2E+1096%E2%80%931103%2E+ACM%2C+2008%2E) 53. Vinyals, O. and Le, Q. V. A neural conversational model. CoRR, abs/1506.05869, 2015. [find](https://scholar.google.com/scholar?q=Vinyals%2C+O%2E+and+Le%2C+Q%2E+V%2E+A+neural+conversational+model%2E+CoRR%2C+abs%2F1506%2E05869%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/) 54. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.07461, 2018. [find](https://scholar.google.com/scholar?q=Wang%2C+A%2E%2C+Singh%2C+A%2E%2C+Michael%2C+J%2E%2C+Hill%2C+F%2E%2C+Levy%2C+O%2E%2C+and+Bowman%2C+S%2E+R%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+CoRR%2C+abs%2F1804%2E07461%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) 55. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnic… [find](https://scholar.google.com/scholar?q=Wu%2C+Y%2E%2C+Schuster%2C+M%2E%2C+Chen%2C+Z%2E%2C+Le%2C+Q%2E+V%2E%2C+Norouzi%2C+M%2E%2C+Macherey%2C+W%2E%2C+Krikun%2C+M%2E%2C+Cao%2C+Y%2E%2C+Gao%2C+Q%2E%2C+Macherey%2C+K%2E%2C+Klingner%2C+J%2E%2C+Shah%2C+A%2E%2C+Johnson%2C+M%2E%2C+Liu%2C+X%2E%2C+Kaiser%2C+L%2E%2C+Gouws%2C+S%2E%2C+Kato%2C+Y%2E%2C+Kudo%2C+T) 56. Yang, Z., Chen, W., Wang, F., and Xu, B. Unsupervised neural machine translation with weight sharing. In ACL, pp. 46–55, 2018. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Chen%2C+W%2E%2C+Wang%2C+F%2E%2C+and+Xu%2C+B%2E+Unsupervised+neural+machine+translation+with+weight+sharing%2E+In+ACL%2C+pp%2E+46%E2%80%9355%2C+2018%2E) 57. Yuan, X., Wang, T., Gulcehre, C., Sordoni, A., Bachman, P., Zhang, S., Subramanian, S., and Trischler, A. Machine comprehension by text-to-text neural question generation. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 15–25, 2017. [find](https://scholar.google.com/scholar?q=Yuan%2C+X%2E%2C+Wang%2C+T%2E%2C+Gulcehre%2C+C%2E%2C+Sordoni%2C+A%2E%2C+Bachman%2C+P%2E%2C+Zhang%2C+S%2E%2C+Subramanian%2C+S%2E%2C+and+Trischler%2C+A%2E+Machine+comprehension+by+text%2Dto%2Dtext+neural+question+generation%2E+In+Proceedings+of+the+2nd+Wo) 58. Zhang, J. and Zong, C. Exploiting source-side monolingual data in neural machine translation. In EMNLP, pp. 1535– 1545, 2016. [find](https://scholar.google.com/scholar?q=Zhang%2C+J%2E+and+Zong%2C+C%2E+Exploiting+source%2Dside+monolingual+data+in+neural+machine+translation%2E+In+EMNLP%2C+pp%2E+1535%E2%80%93+1545%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2016-exploiting-source-side-monolingual-data/) 59. Zoph, B., Yuret, D., May, J., and Knight, K. Transfer learning for low-resource neural machine translation. In EMNLP, pp. 1568–1575, 2016. [find](https://scholar.google.com/scholar?q=Zoph%2C+B%2E%2C+Yuret%2C+D%2E%2C+May%2C+J%2E%2C+and+Knight%2C+K%2E+Transfer+learning+for+low%2Dresource+neural+machine+translation%2E+In+EMNLP%2C+pp%2E+1568%E2%80%931575%2C+2016%2E)