---
title: "Factual Error Correction for Abstractive — Cao 2020"
description: "Abstractive summarizers built on pre-trained transformers often produce summaries that contradict their source documents."
authors: ["Meng Cao", "Yue Dong", "Jiapeng Wu", "Jackie Chi Kit Cheung"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.506/"
doi: "10.18653/v1/2020.emnlp-main.506"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cao-2020-factual-error-correction-abstractive-summarization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cao-2020-factual-error-correction-abstractive-summarization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Factual Error Correction for Abstractive Summarization Models

# Factual Error Correction for Abstractive Summarization Models

**Authors:** [Meng Cao](https://md.research.elamur.ai/free/en/authors/meng-cao/), [Yue Dong](https://md.research.elamur.ai/free/en/authors/yue-dong/), Jiapeng Wu, [Jackie Chi Kit Cheung](https://md.research.elamur.ai/free/en/authors/jackie-chi-kit-cheung/) · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.506/) · [PDF](https://aclanthology.org/2020.emnlp-main.506.pdf) · [DOI 10.18653/v1/2020.emnlp-main.506](https://doi.org/10.18653/v1/2020.emnlp-main.506) · [OpenAlex](https://openalex.org/W3102645206)

**Citations:** 126 citations (openalex-snapshot · 2026-10-14) · [91](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Abstractive summarizers built on pre-trained transformers often produce summaries that contradict their source documents. Earlier estimates put this at roughly 30% of outputs. The authors propose a post-hoc corrector rather than a new summarizer or another detection metric. A fine-tuned BART model receives a draft summary concatenated with its source article and generates a revised summary. Training data comes from CNN/DailyMail: reference summaries are corrupted with heuristic swaps of entities, numbers, dates and pronouns, chosen after manual inspection of real system errors. The same model also works as a consistency checker, since any edit it makes counts as an "inconsistent" verdict.

On artificial test data, the corrector restores most corrupted summaries exactly. On the human-annotated K2019 test set, it detects inconsistencies clearly better than FactCC and NLI-based baselines. However, it repairs only a small share of genuine system errors. This shows that synthetic corruptions do not reflect the variety of real summarizer mistakes, which is an important caveat for weakly supervised factuality work.

## Key points
- **Training data:** 30% (85,583) of the 287,227 CNN/DailyMail training summaries were corrupted, matching the estimated real error rate. The corruptions break down into 35,113 entity, 20,204 pronoun, 16,858 date and 13,408 number swaps. Swapped-in entities come from the source document, using spaCy NER.
- **Artificial test set:** 62.13% of the 5,780 corrupted summaries were corrected to exactly match the reference. 73.73% of the 5,710 clean summaries were left unchanged. Consistency-checking accuracy was 84.38%.
- **K2019 consistency checking:** 83.10% accuracy and 0.6121 macro F1, versus 74.15% and 0.5106 for FactCC. BERT+MNLI and BERT+FEVER scored only about 52% accuracy.
- **Human evaluation on real outputs:**
  - Of 62 inconsistent summaries, the corrector edited 19 and fixed 11, a 17.74% success rate.
  - Of 441 consistent summaries, it broke the meaning of only 5 (1.13%).
- **Availability:** code and data are released at github.com/mcao610/Factual-Error-Correction. The model is built on fairseq BART.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

91 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 80 not yet.

- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/cao-2021-cliff-contrastive-learning-improving-faithfulness/)
- [Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/cao-2022-hallucinated-but-factual-inspecting-factuality/)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://md.research.elamur.ai/free/en/papers/chen-2023-benchmarking-large-language-models-retrieval/)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://md.research.elamur.ai/free/en/papers/gao-2023-rarr-researching-revising-what-language/)
- [Annotating and Modeling Fine-grained Factuality in Summarization](https://md.research.elamur.ai/free/en/papers/goyal-2021-annotating-modeling-fine-grained-factuality/)
- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://md.research.elamur.ai/free/en/papers/kamoi-2024-when-can-llms-actually-correct/)
- [大语言模型的幻觉问题研究综述](https://md.research.elamur.ai/free/en/papers/liu-2024-hallucinations-large-language/)
- [Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors](https://md.research.elamur.ai/free/en/papers/tang-2023-understanding-factual-errors-summarization-errors/)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cao2020factual,
  title = {{Factual Error Correction for Abstractive Summarization Models}},
  author = {Cao, Meng and Dong, Yue and Wu, Jiapeng and Cheung, Jackie Chi Kit},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.506},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.506}
}
```

- [← previous Fact or Fiction: Verifying Scientific Claims](https://md.research.elamur.ai/free/en/papers/wadden-2020-fact-or-fiction-verifying-scientific/)
- [next → Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq](https://md.research.elamur.ai/free/en/papers/wang-2020-fairseq-s2t-fast-speech-text/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Thirty-Second AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Ziqiang+Cao%2C+Furu+Wei%2C+Wenjie+Li%2C+and+Sujian+Li%2E+2018%2E+Faithful+to+the+original%3A+Fact+aware+neural+abstractive+summarization%2E+In+Thirty%2DSecond+AAAI+Conference+on+Artificial+Intelligence%2E)
2. Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi-fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Yue+Dong%2C+Shuohang+Wang%2C+Zhe+Gan%2C+Yu+Cheng%2C+Jackie+Chi+Kit+Cheung%2C+and+Jingjing+Liu%2E+2020%2E+Multi%2Dfact+correction+in+abstractive+text+summarization%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+M)
3. Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th (EMNLP-IJCNLP), pages 3730–3740, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexander+Fabbri%2C+Irene+Li%2C+Tianwei+She%2C+Suyi+Li%2C+and+Dragomir+Radev%2E+2019%2E+Multi%2Dnews%3A+A+large%2Dscale+multi%2Ddocument+summarization+dataset+and+abstractive+hierarchical+model%2E+In+Proceedings+of+the+57t) [our card](https://md.research.elamur.ai/free/en/papers/fabbri-2019-multi-news-large-scale-multi/)
4. Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1074%E2%80%931084%2C+Florence%2C+Italy%2E+Association+for+Computational+Linguistics%2E)
5. Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Tobias+Falke%2C+Leonardo+FR+Ribeiro%2C+Prasetya+Ajie+Utama%2C+Ido+Dagan%2C+and+Iryna+Gurevych%2E+2019%2E+Ranking+generated+summaries+by+correctness%3A+An+interesting+but+challenging+application+for+natural+language) [our card](https://md.research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/)
6. Tian Shi, Ping Wang, and Chandan K. Reddy. 2019. LeafNATS: An open-source toolkit and live demo system for neural abstractive text summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 66–71, Minnea… [find](https://scholar.google.com/scholar?q=Tian+Shi%2C+Ping+Wang%2C+and+Chandan+K%2E+Reddy%2E+2019%2E+LeafNATS%3A+An+open%2Dsource+toolkit+and+live+demo+system+for+neural+abstractive+text+summarization%2E+In+Proceedings+of+the+2019+Conference+of+the+North+Ame)
7. Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 166–175. [find](https://scholar.google.com/scholar?q=Ben+Goodrich%2C+Vinay+Rao%2C+Peter+J+Liu%2C+and+Mohammad+Saleh%2E+2019%2E+Assessing+the+factual+accuracy+of+generated+text%2E+In+Proceedings+of+the+25th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+)
8. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processi)
9. Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687–697, Melbourne, Australia. Associat… [find](https://scholar.google.com/scholar?q=Han+Guo%2C+Ramakanth+Pasunuru%2C+and+Mohit+Bansal%2E+2018%2E+Soft+layer%2Dspecific+multi%2Dtask+summarization+with+entailment+and+question+generation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+)
10. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. arXiv preprint arXiv:2004.04228. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+arXiv+preprint+arXiv%3A2004%2E04228%2E)
11. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2019%2E+Evaluating+the+factual+consistency+of+abstractive+text+summarization%2E+arXiv+preprint+arXiv%3A1910%2E12840%2E)
12. Krtin Kumar and Jackie Chi Kit Cheung. 2019. Understanding the Behaviour of Neural Abstractive Summarizers using Contrastive Examples. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… [find](https://scholar.google.com/scholar?q=Krtin+Kumar+and+Jackie+Chi+Kit+Cheung%2E+2019%2E+Understanding+the+Behaviour+of+Neural+Abstractive+Summarizers+using+Contrastive+Examples%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapt)
13. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2019a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777. [find](https://scholar.google.com/scholar?q=Jingqing+Zhang%2C+Yao+Zhao%2C+Mohammad+Saleh%2C+and+Peter+J+Liu%2E+2019a%2E+Pegasus%3A+Pre%2Dtraining+with+extracted+gap%2Dsentences+for+abstractive+summarization%2E+arXiv+preprint+arXiv%3A1912%2E08777%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)
14. Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D Manning, and Curtis P Langlotz. 2019b. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. arXiv preprint arXiv:1911.02541. [find](https://scholar.google.com/scholar?q=Yuhao+Zhang%2C+Derek+Merck%2C+Emily+Bao+Tsai%2C+Christopher+D+Manning%2C+and+Curtis+P+Langlotz%2E+2019b%2E+Optimizing+the+factual+correctness+of+a+summary%3A+A+study+of+summarizing+radiology+reports%2E+arXiv+preprint) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2020-optimizing-factual-correctness-summary-study/)
15. Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2020. Boosting factual correctness of abstractive summarization with knowledge graph. arXiv preprint arXiv:2003.08612. — [arXiv:2003.08612](https://arxiv.org/abs/2003.08612) [find](https://scholar.google.com/scholar?q=Chenguang+Zhu%2C+William+Hinthorn%2C+Ruochen+Xu%2C+Qingkai+Zeng%2C+Michael+Zeng%2C+Xuedong+Huang%2C+and+Meng+Jiang%2E+2020%2E+Boosting+factual+correctness+of+abstractive+summarization+with+knowledge+graph%2E+arXiv+prep)
16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Ves+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2019%2E+Bart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+langu) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
17. Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1430–1441. [find](https://scholar.google.com/scholar?q=Haoran+Li%2C+Junnan+Zhu%2C+Jiajun+Zhang%2C+and+Chengqing+Zong%2E+2018%2E+Ensure+the+correctness+of+the+summary%3A+Incorporate+entailment+knowledge+into+abstractive+sentence+summarization%2E+In+Proceedings+of+the+27)
18. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
19. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+2019%2E+Text+summarization+with+pretrained+encoders%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+International+Join)