---
title: "ERNIE-GEN — paper by Xiao et al. (2020) — summary and ideas"
description: "ERNIE-GEN, from Baidu, is a sequence-to-sequence framework that covers both pre-training and… 30 references, cited by 42 papers here."
authors: ["Dongling Xiao", "Han Zhang", "Yukun Li", "Yu Sun", "Hao Tian", "Hua Wu", "Haifeng Wang"]
year: 2020
venue: "arXiv:2001.11314"
source: "https://arxiv.org/abs/2001.11314"
doi: "10.48550/arxiv.2001.11314"
arxiv: "2001.11314"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiao-2020-ernie-gen-enhanced-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiao-2020-ernie-gen-enhanced-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

# ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

**Authors:** Dongling Xiao, [Han Zhang](https://md.research.elamur.ai/free/en/authors/han-zhang/), [Yukun Li](https://md.research.elamur.ai/free/en/authors/yukun-li/), [Yu Sun](https://md.research.elamur.ai/free/en/authors/yu-sun/), [Hao Tian](https://md.research.elamur.ai/free/en/authors/hao-tian/), [Hua Wu](https://md.research.elamur.ai/free/en/authors/hua-wu/) and [Haifeng Wang](https://md.research.elamur.ai/free/en/authors/haifeng-wang/) · **Year:** 2020 · **Venue:** [arXiv:2001.11314](https://md.research.elamur.ai/free/en/venues/international-joint-conference-on-artificial-intelligence/) ([International Joint Conference on Artificial Intelligence 2020](https://md.research.elamur.ai/free/en/venues/international-joint-conference-on-artificial-intelligence/2020/)) · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2001.11314) · [PDF](https://arxiv.org/pdf/2001.11314) · [DOI 10.48550/arxiv.2001.11314](https://doi.org/10.48550/arxiv.2001.11314) · [arXiv 2001.11314](https://arxiv.org/abs/2001.11314) · [OpenAlex](https://openalex.org/W3035359363)

**Citations:** 121 citations (OpenAlex, 2026-10-14) · [42](https://md.research.elamur.ai/free/en/papers/xiao-2020-ernie-gen-enhanced-multi/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
ERNIE-GEN, from Baidu, is a sequence-to-sequence framework that covers both pre-training and fine-tuning for text generation tasks. It targets exposure bias: during training the decoder conditions on gold tokens, but at inference it conditions on its own outputs, so early errors propagate. The model uses a Multi-Flow Attention design built on a Transformer whose parameters are shared by encoder and decoder. It has three main components:
- **Infilling generation:** an inserted [ATTN] symbol, rather than the previous word, collects context at each step.
- **Noise-aware generation:** target tokens are randomly replaced with arbitrary vocabulary words, which teaches the model to tolerate its own mistakes.
- **Span-by-span flow:** the model predicts whole semantic spans, running in parallel with word-by-word prediction.

Pre-training data come from unlabeled text, where multiple fragments of varying length are sampled as targets. The authors report state-of-the-art results on summarization, question generation, dialogue and generative QA, with less pre-training data and fewer parameters than competing models. Code and models are publicly released.

## Key points
- **Benchmarks:** Gigaword and CNN/DailyMail (summarization), SQuAD (question generation), Persona-Chat (dialogue) and CoQA (generative QA). Code is available at github.com/PaddlePaddle/ERNIE.
- **Target sampling:** fragments cover about 25% of the input (γ = 0.25). Fragment lengths are drawn from U(1,4) with probability 0.4 or U(4,32) with probability 0.6.
- **Ablation (ERNIE-GEN BASE, Gigaword 10k, ROUGE-1/2/L):**
  - Full model: 33.75 / 15.23 / 31.35.
  - Without both noise-aware generation and span-by-span generation: 33.23 / 14.77 / 31.00.
- **Noise-aware fine-tuning vs. masking:** predicting every decoder token under noise outperformed UniLM-style masked-word fine-tuning (masking probability 0.7).
- **Attention analysis:** higher noise rates led the model to give lower attention weights to corrupted target tokens.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 40 works · 14,627 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 145 works

## Cite

Cite as: Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang. 2020. ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation. arXiv:2001.11314.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xiao2020ernie,
  title = {{ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation}},
  author = {Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang},
  year = {2020},
  doi = {10.48550/arxiv.2001.11314},
  eprint = {2001.11314},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2001.11314}
}
```

- [← previous Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/eikema-2020-is-map-decoding-all-you/)
- [next → If beam search is the answer, what was the question?](https://md.research.elamur.ai/free/en/papers/meister-2020-if-beam-search-is-answer/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

30 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. [Plato: Pre-trained dialogue generation model with discrete latent variable](https://md.research.elamur.ai/free/en/papers/bao-2020-plato-pre-trained-dialogue-generation/). In ACL, 2020.
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL, pages 4171–4186, 2019.
3. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiaowuen Hon. [Unified language model pre-training for natural language understanding and generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). In NIPS, pages 13042–13054, 2019.
4. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In NIPS, pages 1693–1701, 2015.
5. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. [Spanbert: Improving pre-training by representing and predicting spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/). arXiv preprint arXiv:1907.10529, 2019.
6. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radue Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radue+Soricut%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+ICLR%2C+2020%2E)
7. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461, 2019.
8. Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In EMNLP-IJCNLP, pages 3721–3731, 2019. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+Text+summarization+with+pretrained+encoders%2E+In+EMNLP%2DIJCNLP%2C+pages+3721%E2%80%933731%2C+2019%2E)
9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+p)
10. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In NAACL-HLT, pages 2227–2237, 2018.
11. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
12. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683, 2019.
13. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In EMNLP, pages 2383–2392, 2016.
14. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. [Sequence level training with recurrent neural networks](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/). In ICLR, 2016.
15. Siva Reddy, Danqi Chen, and Christopher D Manning. [Coqa: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). In ACL, pages 249–266, 2019.
16. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. [Leveraging pre-trained checkpoints for sequence generation tasks](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/). arXiv preprint arXiv:1907.12461, 2019.
17. Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, 2015. [find](https://scholar.google.com/scholar?q=Alexander+M+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+EMNLP%2C+2015%2E)
18. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. [Mass: Masked sequence to sequence pre-training for language generation](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/). In ICML, 2019.
19. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. [Ernie: Enhanced representation through knowledge integration](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/). arXiv preprint arXiv:1904.09223, 2019.
20. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. [Ernie 2.0: A continual pre-training framework for language understanding](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/). In AAAI, 2020.
21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS, pages 5998–6008, 2017.
22. Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du. A reinforced topic-aware convolutional sequence-to-sequence model for abstractive text summarization. In IJCAI, 2018. [find](https://scholar.google.com/scholar?q=Li+Wang%2C+Junlin+Yao%2C+Yunzhe+Tao%2C+Li+Zhong%2C+Wei+Liu%2C+and+Qiang+Du%2E+A+reinforced+topic%2Daware+convolutional+sequence%2Dto%2Dsequence+model+for+abstractive+text+summarization%2E+In+IJCAI%2C+2018%2E)
23. Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational autoencoder for text generation. In NAACL-HLT, 2019. [find](https://scholar.google.com/scholar?q=Wenlin+Wang%2C+Zhe+Gan%2C+Hongteng+Xu%2C+Ruiyi+Zhang%2C+Guoyin+Wang%2C+Dinghan+Shen%2C+Changyou+Chen%2C+and+Lawrence+Carin%2E+Topic%2Dguided+variational+autoencoder+for+text+generation%2E+In+NAACL%2DHLT%2C+2019%2E)
24. Qian Yang, Dinghan Shen, Yong Cheng, Wenlin Wang, Guoyin Wang, Lawrence Carin, et al. An end-to-end generative architecture for paraphrase generation. In EMNLP-IJCNLP, pages 3123–3133, 2019. [find](https://scholar.google.com/scholar?q=Qian+Yang%2C+Dinghan+Shen%2C+Yong+Cheng%2C+Wenlin+Wang%2C+Guoyin+Wang%2C+Lawrence+Carin%2C+et+al%2E+An+end%2Dto%2Dend+generative+architecture+for+paraphrase+generation%2E+In+EMNLP%2DIJCNLP%2C+pages+3123%E2%80%933133%2C+2019%2E)
25. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In NIPS, pages 5754–5764, 2019.
26. Shiyue Zhang and Mohit Bansal. [Addressing semantic drift in question generation for semi-supervised question answering](https://md.research.elamur.ai/free/en/papers/zhang-2019-addressing-semantic-drift-question-generation/). In EMNLP-IJCNLP, pages 2495–2509, 2019.
27. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. [Personalizing dialogue agents: I have a dog, do you have pets too?](https://md.research.elamur.ai/free/en/papers/zhang-2018-personalizing-dialogue-agents-i-have/) In ACL, pages 2204–2213, 2018.
28. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. [Pegasus: Pre-training with extracted gap-sentences for abstractive summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/). arXiv preprint arXiv:1912.08777, 2019.
29. Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. [Paragraph-level neural question generation with maxout pointer and gated self-attention networks](https://md.research.elamur.ai/free/en/papers/zhao-2018-paragraph-level-neural-question-generation/). In EMNLP, pages 3901–3910, 2018.
30. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In ICCV, pages 19–27, 2015.

*Updated 24 October 2026*