---
title: "Improving Faithfulness in Abstractive Summarization with Contrast — paper by Chen et al. (2021)"
description: "The authors propose a post-processing correction step that works with any summarization model. 26 references, cited by 66 papers here."
authors: ["Sihao Chen", "Fan Zhang", "Kazoo Sone", "Dan Roth"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.475/"
doi: "10.18653/v1/2021.naacl-main.475"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2021-improving-faithfulness-abstractive-summarization-contrast/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2021-improving-faithfulness-abstractive-summarization-contrast/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection

# Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection

**Authors:** [Sihao Chen](https://md.research.elamur.ai/free/en/authors/sihao-chen/), Fan Zhang, Kazoo Sone and [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2021](https://md.research.elamur.ai/free/en/venues/naacl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.475/) · [PDF](https://aclanthology.org/2021.naacl-main.475.pdf) · [DOI 10.18653/v1/2021.naacl-main.475](https://doi.org/10.18653/v1/2021.naacl-main.475) · [OpenAlex](https://openalex.org/W3172210712)

**Citations:** 83 citations (OpenAlex, 2026-10-14) · [66](https://md.research.elamur.ai/free/en/papers/chen-2021-improving-faithfulness-abstractive-summarization-contrast/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Neural abstractive summarizers often produce extrinsic hallucinations, meaning content that does not appear in the source document. The problem is especially severe on XSum, where many reference summaries are themselves unfaithful. The authors propose a post-processing correction step that works with any summarization model.

In the generation step, named entities and quantities in a system summary are detected with the Stanza NER tagger (trained on OntoNotes). Each one is swapped for a source-document entity carrying the same NER label, producing alternative candidate summaries. In the selection step, a classifier built on BART with a linear layer scores these candidates together with the original summary, and the highest-scoring one is kept. The classifier is trained on faithful XSum references paired with synthetic negatives, using cross-entropy plus a margin ranking loss.

Applied to BART-large outputs, the method raises FEQA scores and human-judged faithfulness while costing little ROUGE or BERTScore. However, incorrect swaps introduce intrinsic hallucinations. The work matters because it offers a cheap, system-independent fix that keeps the fluency of large pretrained summarizers.

## Key points
- In 500 human-annotated XSum reference summaries, only 23.1% were faithful and 73.1% contained extrinsic hallucinations. Entities accounted for 35.9% and numbers/quantities for 18.2% of all summaries.
- The corrector modified 13.3% of BART-large test summaries (1,510 summaries).
- On the modified summaries, FEQA rose from 22.50 to 25.62 (p \< 0.001), while ROUGE-L fell from 38.63 to 36.62.
- In a human evaluation of 95 corrected summaries:
  - faithful summaries rose from 23.8% to 59.5%;
  - extrinsic hallucinations fell from 71.7% to 9.2%;
  - intrinsic hallucinations grew from 1.7% to 29.1%.
- Off-the-shelf MNLI models were not good enough for candidate selection. A cross-system analysis covered PtGen, TConvS2S, TranS2S and BertS2S, showing that recall depends on how many of each system's hallucinations involve entities.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 170 works · 60,739 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 159 works

## Cite

Cite as: Sihao Chen, Fan Zhang, Kazoo Sone and Dan Roth. 2021. Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2021improving,
  title = {{Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection}},
  author = {Sihao Chen, Fan Zhang, Kazoo Sone and Dan Roth},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.475},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.475}
}
```

- [← previous BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data](https://md.research.elamur.ai/free/en/papers/song-2021-bob-bert-over-bert-training/)
- [next → Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding](https://md.research.elamur.ai/free/en/papers/dziri-2021-neural-path-hunter-reducing-hallucination/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

26 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstractive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251–6258, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/cao-2020-factual-error-correction-abstractive-summarization/)
2. Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2020. Towards question-answering as an automatic metric for evaluating the content quality of a summary. arXiv preprint arXiv:2010.00490. [our card](https://md.research.elamur.ai/free/en/papers/deutsch-2021-question-answering-as-automatic-metric/)
3. Daniel Deutsch and Dan Roth. 2020. Understanding the extent to which summarization evaluation metrics measure the information quality of summaries. arXiv preprint arXiv:2010.12495. — [arXiv:2010.12495](https://arxiv.org/abs/2010.12495) [find](https://scholar.google.com/scholar?q=Daniel+Deutsch+and+Dan+Roth%2E+2020%2E+Understanding+the+extent+to+which+summarization+evaluation+metrics+measure+the+information+quality+of+summaries%2E+arXiv+preprint+arXiv%3A2010%2E12495%2E+%E2%80%94+%5BarXiv%3A2010%2E12495)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
5. Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055– 5070, Online. Association for Computational Lingu… [our card](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/)
6. We study contrast candidate generation and selection as a method to apply post-hoc fixes to extrinsically hallucinated summary on entities and quantities, under the setting where the summarization Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie dataset suffers from intrinsic and extrinsic halluc… [find](https://scholar.google.com/scholar?q=We+study+contrast+candidate+generation+and+selection+as+a+method+to+apply+post%2Dhoc+fixes+to+extrinsically+hallucinated+summary+on+entities+and+quantities%2C+under+the+setting+where+the+summarization+Tob)
7. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214–2220, Florence, Italy. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/)
8. Katja Filippova. 2020. Controlled hallucinations: Learning to generate faithfully from noisy data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 864–870, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Katja+Filippova%2E+2020%2E+Controlled+hallucinations%3A+Learning+to+generate+faithfully+from+noisy+data%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+864%E2%80%93870%2C+Online%2E+Asso)
9. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, N… [our card](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
10. Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3592–3603, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/goyal-2020-evaluating-factuality-generation-dependency-level/)
11. Tanya Goyal and Greg Durrett. 2021. Annotating and modeling fine-grained factuality in summarization. In Proceedings of the 2021 Proceedings Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [our card](https://md.research.elamur.ai/free/en/papers/goyal-2021-annotating-modeling-fine-grained-factuality/)
12. Daniel Kang and Tatsunori Hashimoto. 2020. Improved natural language generation via loss truncation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 718–731, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/kang-2020-improved-natural-language-generation-loss/)
13. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. Proceedings of the 34th Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Prannay+Khosla%2C+Piotr+Teterwak%2C+Chen+Wang%2C+Aaron+Sarna%2C+Yonglong+Tian%2C+Phillip+Isola%2C+Aaron+Maschinot%2C+Ce+Liu%2C+and+Dilip+Krishnan%2E+2020%2E+Supervised+contrastive+learning%2E+Proceedings+of+the+34th+Confer)
14. Wojciech Kryscinski, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua… [our card](https://md.research.elamur.ai/free/en/papers/kryscinski-2019-neural-text-summarization-critical-evaluation/)
15. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346. [find](https://scholar.google.com/scholar?q=Wojciech+Kryscinski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2020%2E+Evaluating+the+factual+consistency+of+abstractive+text+summarization%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Met)
16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
17. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/)
18. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3721–3731. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+2019%2E+Text+summarization+with+pretrained+encoders%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+International+Join)
19. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Joshua+Maynez%2C+Shashi+Narayan%2C+Bernd+Bohnet%2C+and+Ryan+McDonald%2E+2020%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comp)
20. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu̇lçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. Association f… [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+dos+Santos%2C+%C3%87ag%CC%86lar+Gu%C3%8C%E2%80%A1l%C3%A7ehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+RNNs+and+beyond%2E+In+Proceedings+of+The+20th+SIGNLL+)
21. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018a. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807. [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B+Cohen%2C+and+Mirella+Lapata%2E+2018a%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+In+Proceedings+of+the+2018+Co)
22. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pape… [our card](https://md.research.elamur.ai/free/en/papers/narayan-2018-ranking-sentences-extractive-summarization-reinforcement/)
23. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. [our card](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/)
24. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264–280. [our card](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/)
25. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389. [find](https://scholar.google.com/scholar?q=Alexander+M+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+)
26. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computation)

*Updated 24 October 2026*