---
title: "ProphetNet: Predicting Future N-gram for — Qi 2020"
description: "ProphetNet addresses a weakness of standard autoregressive Seq2Seq training."
authors: ["Weizhen Qi", "Yu Yan", "Yeyun Gong", "Dayiheng Liu", "Nan Duan", "Jiusheng Chen", "Ruofei Zhang", "Ming Zhou"]
year: 2020
venue: "Findings of the Association for Computational Linguistics: EMNLP 2020"
source: "https://aclanthology.org/2020.findings-emnlp.217/"
doi: "10.18653/v1/2020.findings-emnlp.217"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qi-2020-prophetnet-predicting-future-n-gram/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qi-2020-prophetnet-predicting-future-n-gram/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training

# ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training

**Authors:** [Weizhen Qi](https://md.research.elamur.ai/free/en/authors/weizhen-qi/), Yu Yan, [Yeyun Gong](https://md.research.elamur.ai/free/en/authors/yeyun-gong/), [Dayiheng Liu](https://md.research.elamur.ai/free/en/authors/dayiheng-liu/), [Nan Duan](https://md.research.elamur.ai/free/en/authors/nan-duan/), Jiusheng Chen, [Ruofei Zhang](https://md.research.elamur.ai/free/en/authors/ruofei-zhang/), [Ming Zhou](https://md.research.elamur.ai/free/en/authors/ming-zhou/) · **Year:** 2020 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2020 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.findings-emnlp.217/) · [PDF](https://aclanthology.org/2020.findings-emnlp.217.pdf) · [DOI 10.18653/v1/2020.findings-emnlp.217](https://doi.org/10.18653/v1/2020.findings-emnlp.217) · [OpenAlex](https://openalex.org/W3100439847)

**Citations:** 374 citations (openalex-snapshot · 2026-10-14) · [77](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
ProphetNet addresses a weakness of standard autoregressive Seq2Seq training. Teacher forcing only rewards guessing the immediately next token, so models tend to overfit short-range patterns such as bigrams and neglect longer-range coherence. The authors replace this one-step objective with future n-gram prediction: at each decoder position the model is trained to predict the next n tokens at once. To make this efficient, they extend XLNet's two-stream attention into an n-stream self-attention decoder. A main stream behaves like an ordinary Transformer decoder, and n extra predicting streams each forecast a token further ahead. Because the predicting streams share parameters with the main stream, they can be switched off at inference, which leaves a normal next-token model. Pre-training combines this objective with mask-based denoising auto-encoding on a 16GB corpus and a 160GB corpus. Once fine-tuned, ProphetNet achieves state-of-the-art performance, relative to models pre-trained on similar amounts of data, on two summarization benchmarks (CNN/DailyMail and Gigaword) and on question generation from SQuAD 1.1. This shows that adding a look-ahead training signal can improve generation quality without changing how decoding works.

## Key points
- **Training objective:** a weighted sum of the usual next-token loss and n−1 extra losses for tokens further ahead. The weights follow a power attenuation scheme controlled by a coefficient γ.
- **Pre-training data:**
  - a base-scale set (16GB, the BERT data, Wikipedia+BookCorpus);
  - a large-scale set (160GB, comparable to BART's).
- **Large-scale efficiency:** with the 160GB data, ProphetNet sets new state-of-the-art results on CNN/DailyMail and Gigaword. It uses roughly a third of BART's pre-training epochs and about a fifth of the corpus used by T5 and PEGASUS.
- **Without pre-training (CNN/DailyMail dev):** ProphetNet reaches ROUGE 40.66/18.05/37.79 (R-1/R-2/R-L), compared with 39.19/17.60/36.69 for a same-sized Transformer.
- **n-gram ablation (base models, CNN/DailyMail test):**
  - 2-gram and 3-gram variants score similarly (R-1 42.52 vs 42.61).
  - Both beat MASS base and the 1-gram variant.
  - The 2-gram version was chosen as the default because it trains about 15% faster.

## Interesting ideas
- **Auxiliary look-ahead losses as regularization:** asking hidden states to support predictions several steps ahead pushes them to encode planning information, not just local continuations. This trick could transfer to any autoregressive model, for example a Ukrainian LM or a tagger that decodes sequentially.
- **Training-only heads that vanish at inference:** the extra streams share weights with the main stream and are simply dropped after training. You get a richer training signal while the model keeps standard decoding cost and stays compatible with existing tooling.
- **Decaying weights for distant targets:** down-weighting predictions further in the future, with a tunable attenuation γ, makes it easy to balance the auxiliary signal against the main objective. This is a reusable recipe for multi-horizon losses in general.
- **Checking gains without pre-training:** the authors also test the objective with no pre-training at all. This separates the effect of the objective from the effect of the pre-training data, an evaluation habit worth copying when proposing new losses.
- **Choosing n by cost as well as score:** the 2-gram setting was selected over a marginally better 3-gram one for speed. This is a practical example of reporting the cost-benefit trade-off explicitly rather than chasing the top number.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

77 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 75 not yet.

- [Babies Learn to Look Ahead: Multi-Token Prediction in Small LMs](https://md.research.elamur.ai/free/en/papers/aynetdinov-2025-babies-learn-look-ahead-multi/)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/liu-2021-simcls-simple-framework-contrastive-learning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{qi2020prophetnet,
  title = {{ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training}},
  author = {Qi, Weizhen and Yan, Yu and Gong, Yeyun and Liu, Dayiheng and Duan, Nan and Chen, Jiusheng and Zhang, Ruofei and Zhou, Ming},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.217},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.217}
}
```

- [← previous Proceedings of the የአማርኛ ቋንቋ ትምህርት አሰጣጥ በጀርመን አገር—ዓውደጥናት/Workshop: Amharisch Unterrichten in Deutschland/Workshop on the Teaching of Amharic in Germany 1–2 February 2018, Hiob-Ludolf-Zentrum für Äthiopistik, Asien-Afrika-Institut,Hamburg](https://md.research.elamur.ai/free/en/papers/krzyzanowska-2020-proceedings-workshop-amharisch-unterrichten-deutschland/)
- [next → pyBART: Evidence-based Syntactic Transformations for IE](https://md.research.elamur.ai/free/en/papers/ud-2020-tiktinsky-pybart-evidence-based-syntactic-transformations/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

41 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018. Retrieve, rerank and rewrite: Soft template based neural summarization. In ACL. [find](https://scholar.google.com/scholar?q=Ziqiang+Cao%2C+Wenjie+Li%2C+Sujian+Li%2C+and+Furu+Wei%2E+2018%2E+Retrieve%2C+rerank+and+rewrite%3A+Soft+template+based+neural+summarization%2E+In+ACL%2E)
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2E)
3. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NeurIPS. [find](https://scholar.google.com/scholar?q=Li+Dong%2C+Nan+Yang%2C+Wenhui+Wang%2C+Furu+Wei%2C+Xiaodong+Liu%2C+Yu+Wang%2C+Jianfeng+Gao%2C+Ming+Zhou%2C+and+Hsiao%2DWuen+Hon%2E+2019%2E+Unified+language+model+pre%2Dtraining+for+natural+language+understanding+and+generatio) [our card](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)
4. Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from wikipedia. In ACL. [find](https://scholar.google.com/scholar?q=Xinya+Du+and+Claire+Cardie%2E+2018%2E+Harvesting+paragraph%2Dlevel+question%2Danswer+pairs+from+wikipedia%2E+In+ACL%2E)
5. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. arXiv preprint arXiv:1705.00106. [find](https://scholar.google.com/scholar?q=Xinya+Du%2C+Junru+Shao%2C+and+Claire+Cardie%2E+2017%2E+Learning+to+ask%3A+Neural+question+generation+for+reading+comprehension%2E+arXiv+preprint+arXiv%3A1705%2E00106%2E) [our card](https://md.research.elamur.ai/free/en/papers/du-2017-learning-ask-neural-question-generation/)
6. Sergey Edunov, Alexei Baevski, and Michael Auli. 2019. Pre-trained language model representations for language generation. arXiv preprint arXiv:1903.09722. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Alexei+Baevski%2C+and+Michael+Auli%2E+2019%2E+Pre%2Dtrained+language+model+representations+for+language+generation%2E+arXiv+preprint+arXiv%3A1903%2E09722%2E)
7. Angela Fan, David Grangier, and Michael Auli. 2017. Controllable abstractive summarization. arXiv preprint arXiv:1711.05217. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+David+Grangier%2C+and+Michael+Auli%2E+2017%2E+Controllable+abstractive+summarization%2E+arXiv+preprint+arXiv%3A1711%2E05217%2E) [our card](https://md.research.elamur.ai/free/en/papers/fan-2018-controllable-abstractive-summarization/)
8. Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. In EMNLP. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Yuntian+Deng%2C+and+Alexander+M+Rush%2E+2018%2E+Bottom%2Dup+abstractive+summarization%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/)
9. Caglar Gulcehre, Francis Dutil, Adam Trischler, and Yoshua Bengio. 2017. Plan, attend, generate: Planning for sequence-to-sequence models. In NIPS. [find](https://scholar.google.com/scholar?q=Caglar+Gulcehre%2C+Francis+Dutil%2C+Adam+Trischler%2C+and+Yoshua+Bengio%2E+2017%2E+Plan%2C+attend%2C+generate%3A+Planning+for+sequence%2Dto%2Dsequence+models%2E+In+NIPS%2E)
10. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Danqi+Chen%2C+Yinhan+Liu%2C+Daniel+S+Weld%2C+Luke+Zettlemoyer%2C+and+Omer+Levy%2E+2019%2E+Spanbert%3A+Improving+pre%2Dtraining+by+representing+and+predicting+spans%2E+arXiv+preprint+arXiv%3A1907%2E10529%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/)
11. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR%2E)
12. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Guillaume+Klein%2C+Yoon+Kim%2C+Yuntian+Deng%2C+Jean+Senellart%2C+and+Alexander+M+Rush%2E+2017%2E+Opennmt%3A+Open%2Dsource+toolkit+for+neural+machine+translation%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/klein-2017-opennmt-open-source-toolkit-neural/)
13. David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Chris Pal. 2016. Zoneout: Regularizing rnns by randomly preserving hidden activations. arXiv preprint arXiv:1606.01305. — [arXiv:1606.01305](https://arxiv.org/abs/1606.01305) [find](https://scholar.google.com/scholar?q=David+Krueger%2C+Tegan+Maharaj%2C+Ja%CC%81nos+Krama%CC%81r%2C+Mohammad+Pezeshki%2C+Nicolas+Ballas%2C+Nan+Rosemary+Ke%2C+Anirudh+Goyal%2C+Yoshua+Bengio%2C+Aaron+Courville%2C+and+Chris+Pal%2E+2016%2E+Zoneout%3A+Regularizing+rnns+by+rand)
14. Carolin Lawrence, Bhushan Kotnis, and Mathias Niepert. 2019. Attending to future tokens for bidirectional sequence generation. arXiv preprint arXiv:1908.05915. [find](https://scholar.google.com/scholar?q=Carolin+Lawrence%2C+Bhushan+Kotnis%2C+and+Mathias+Niepert%2E+2019%2E+Attending+to+future+tokens+for+bidirectional+sequence+generation%2E+arXiv+preprint+arXiv%3A1908%2E05915%2E)
15. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Ves+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2019%2E+Bart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+langu) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
16. Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Learning to decode for future success. arXiv preprint arXiv:1701.06549. — [arXiv:1701.06549](https://arxiv.org/abs/1701.06549) [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+2017%2E+Learning+to+decode+for+future+success%2E+arXiv+preprint+arXiv%3A1701%2E06549%2E+%E2%80%94+%5BarXiv%3A1701%2E06549%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1701%2E06549%29)
17. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+summarization+branches+out%2E)
18. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+2019%2E+Text+summarization+with+pretrained+encoders%2E+arXiv+preprint+arXiv%3A1908%2E08345%2E)
19. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. Regularizing and optimizing lstm language models. arXiv preprint arXiv:1708.02182. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Nitish+Shirish+Keskar%2C+and+Richard+Socher%2E+2017%2E+Regularizing+and+optimizing+lstm+language+models%2E+arXiv+preprint+arXiv%3A1708%2E02182%2E) [our card](https://md.research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/)
20. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Feifei+Zhai%2C+and+Bowen+Zhou%2E+2017%2E+Summarunner%3A+A+recurrent+neural+network+based+sequence+model+for+extractive+summarization+of+documents%2E+In+AAAI%2E) [our card](https://md.research.elamur.ai/free/en/papers/nallapati-2016-summarunner-recurrent-neural-network-based/)
21. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Caglar+Gulcehre%2C+Bing+Xiang%2C+et+al%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2E)
22. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. [find](https://scholar.google.com/scholar?q=Aaron+van+den+Oord%2C+Yazhe+Li%2C+and+Oriol+Vinyals%2E+2018%2E+Representation+learning+with+contrastive+predictive+coding%2E+arXiv+preprint+arXiv%3A1807%2E03748%2E) [our card](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/)
23. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In ICML. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+2013%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+In+ICML%2E)
24. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+NAACL%2E)
25. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL [https://s3-us-west-2.amazonaws](https://s3-us-west-2.amazonaws). com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. [↗](https://s3-us-west-2.amazonaws)
26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2E)
27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2019%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext)
28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
29. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685. [find](https://scholar.google.com/scholar?q=Alexander+M+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+arXiv+preprint+arXiv%3A1509%2E00685%2E)
30. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+ACL%2E)
31. Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, and Yoshua Bengio. 2018. Twin networks: Matching the future for sequence generation. In ICLR. [find](https://scholar.google.com/scholar?q=Dmitriy+Serdyuk%2C+Nan+Rosemary+Ke%2C+Alessandro+Sordoni%2C+Adam+Trischler%2C+Chris+Pal%2C+and+Yoshua+Bengio%2E+2018%2E+Twin+networks%3A+Matching+the+future+for+sequence+generation%2E+In+ICLR%2E)
32. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450. [find](https://scholar.google.com/scholar?q=Kaitao+Song%2C+Xu+Tan%2C+Tao+Qin%2C+Jianfeng+Lu%2C+and+Tie%2DYan+Liu%2E+2019%2E+Mass%3A+Masked+sequence+to+sequence+pre%2Dtraining+for+language+generation%2E+arXiv+preprint+arXiv%3A1905%2E02450%2E) [our card](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/)
33. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NIPS%2E)
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NIPS%2E)
35. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+arXiv+preprint+arXiv%3A1906%2E08) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
36. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. arXiv preprint arXiv:1905.12616. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Hannah+Rashkin%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+Franziska+Roesner%2C+and+Yejin+Choi%2E+2019%2E+Defending+against+neural+fake+news%2E+arXiv+preprint+arXiv%3A1905%2E12616%2E) [our card](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/)
37. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2019. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777. [find](https://scholar.google.com/scholar?q=Jingqing+Zhang%2C+Yao+Zhao%2C+Mohammad+Saleh%2C+and+Peter+J+Liu%2E+2019%2E+Pegasus%3A+Pre%2Dtraining+with+extracted+gap%2Dsentences+for+abstractive+summarization%2E+arXiv+preprint+arXiv%3A1912%2E08777%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)
38. Shiyue Zhang and Mohit Bansal. 2019. Addressing semantic drift in question generation for semi-supervised question answering. arXiv preprint arXiv:1909.06356. [find](https://scholar.google.com/scholar?q=Shiyue+Zhang+and+Mohit+Bansal%2E+2019%2E+Addressing+semantic+drift+in+question+generation+for+semi%2Dsupervised+question+answering%2E+arXiv+preprint+arXiv%3A1909%2E06356%2E)
39. Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In EMNLP. [find](https://scholar.google.com/scholar?q=Yao+Zhao%2C+Xiaochuan+Ni%2C+Yuanyuan+Ding%2C+and+Qifa+Ke%2E+2018%2E+Paragraph%2Dlevel+neural+question+generation+with+maxout+pointer+and+gated+self%2Dattention+networks%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2018-paragraph-level-neural-question-generation/)
40. Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. In National CCF Conference on Natural Language Processing and Chinese Computing, pages 662–671. [find](https://scholar.google.com/scholar?q=Qingyu+Zhou%2C+Nan+Yang%2C+Furu+Wei%2C+Chuanqi+Tan%2C+Hangbo+Bao%2C+and+Ming+Zhou%2E+2017%2E+Neural+question+generation+from+text%3A+A+preliminary+study%2E+In+National+CCF+Conference+on+Natural+Language+Processing+and+)
41. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–… [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Rich+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies+) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)