---
title: "Entity-level Factual Consistency of Abstractive — Nan 2021"
description: "Abstractive summarizers often hallucinate named entities, inserting people, places or organizations that never occur in the source article."
authors: ["Feng Nan", "Ramesh Nallapati", "Zhiguo Wang", "Cicero Nogueira dos Santos", "Henghui Zhu", "Dejiao Zhang", "Kathleen McKeown", "Bing Xiang"]
year: 2021
venue: "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume"
source: "https://aclanthology.org/2021.eacl-main.235/"
doi: "10.18653/v1/2021.eacl-main.235"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nan-2021-entity-level-factual-consistency-abstractive/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nan-2021-entity-level-factual-consistency-abstractive/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / Entity-level Factual Consistency of Abstractive Text Summarization

# Entity-level Factual Consistency of Abstractive Text Summarization

**Authors:** Feng Nan, [Ramesh Nallapati](https://md.research.elamur.ai/free/en/authors/ramesh-nallapati/), [Zhiguo Wang](https://md.research.elamur.ai/free/en/authors/zhiguo-wang/), [Cicero Nogueira dos Santos](https://md.research.elamur.ai/free/en/authors/cicero-nogueira-dos-santos/), Henghui Zhu, Dejiao Zhang, [Kathleen McKeown](https://md.research.elamur.ai/free/en/authors/kathleen-mckeown/), [Bing Xiang](https://md.research.elamur.ai/free/en/authors/bing-xiang/) · **Year:** 2021 · **Venue:** Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.eacl-main.235/) · [PDF](https://aclanthology.org/2021.eacl-main.235.pdf) · [DOI 10.18653/v1/2021.eacl-main.235](https://doi.org/10.18653/v1/2021.eacl-main.235) · [OpenAlex](https://openalex.org/W3153193605)

**Citations:** 114 citations (openalex-snapshot · 2026-10-14) · [90](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Abstractive summarizers often hallucinate named entities, inserting people, places or organizations that never occur in the source article. ROUGE does not detect this kind of error. The authors address it with three entity-level metrics computed with spaCy NER and lenient, case-insensitive n-gram matching:
- **prec_s** measures how many summary entities can be found in the source.
- **prec_t** and **recall_t** (combined as F1_t) compare the generated entities with those in the reference summary.

Applying prec_s to the reference summaries of Newsroom, CNN/DailyMail and XSUM shows that the data itself is noisy, and XSUM is the worst case. The authors therefore filter the training data, removing reference sentences whose entities are absent from the document. They also test two training variants on top of BART-large (Fairseq):
- **Multi-task:** an encoder-side BIO tagger that marks summary-worthy entities.
- **JAENS:** the decoder first emits the salient entities and then writes the summary.

Filtering alone sharply reduces hallucination while keeping ROUGE largely intact. Both variants further raise entity precision and recall. This gives a cheap, reproducible way to measure and improve factuality at the entity level.

## Key points
- Reference-summary prec_s on the test sets is about 90.5% for Newsroom, 96.6% for CNNDM and only 79.3% for XSUM. The low XSUM figure arises because XSUM's summary is the article's lead sentence, which is removed from the source.
- Filtering shrinks the XSUM training set from 203,540 to 135,155 pairs (roughly a third removed). Newsroom and CNNDM lose far less.
- On XSUM, training on filtered data lifts model prec_s from 93.6% to 98.2%.
- The multi-task loss is the MLE loss plus α times the BIO loss, with α between 0.1 and 0.5. JAENS beats multi-task on CNNDM and Newsroom but can lower ROUGE.
- A manual check of 10 XSUM validation examples found that both the NER output and the entity matching were correct. Code is released at github.com/amazon-research/fact-check-summarization.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/)

## Cited by

Cited by

90 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 81 not yet.

- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/cao-2021-cliff-contrastive-learning-improving-faithfulness/)
- [Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/cao-2022-hallucinated-but-factual-inspecting-factuality/)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)
- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [Evaluating Large Language Models for Health-related Queries with Presuppositions](https://md.research.elamur.ai/free/en/papers/kaur-2024-evaluating-large-language-models-health/)
- [GUMsley: Evaluating Entity Salience in Summarization for 12 English Genres](https://md.research.elamur.ai/free/en/papers/lin-2024-gumsley-evaluating-entity-salience-summarization/)
- [大语言模型的幻觉问题研究综述](https://md.research.elamur.ai/free/en/papers/liu-2024-hallucinations-large-language/)
- [ChatGPT as a Factual Inconsistency Evaluator for Text Summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/)
- [Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors](https://md.research.elamur.ai/free/en/papers/tang-2023-understanding-factual-errors-summarization-errors/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nan2021entity,
  title = {{Entity-level Factual Consistency of Abstractive Text Summarization}},
  author = {Nan, Feng and Nallapati, Ramesh and Wang, Zhiguo and Santos, Cicero Nogueira dos and Zhu, Henghui and Zhang, Dejiao and McKeown, Kathleen and Xiang, Bing},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.235},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.235}
}
```

- [← previous Entity-Based Knowledge Conflicts in Question Answering](https://md.research.elamur.ai/free/en/papers/longpre-2021-entity-based-knowledge-conflicts-question/)
- [next → ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/sun-2021-ernie-3-0-large/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

18 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. [find](https://scholar.google.com/scholar?q=Ziqiang+Cao%2C+Furu+Wei%2C+Wenjie+Li%2C+and+Sujian+Li%2E+2018%2E+Faithful+to+the+original%3A+Fact+aware+neural+abstractive+summarization%2E)
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
3. Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45–54, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+David+Grangier%2C+and+Michael+Auli%2E+2018%2E+Controllable+abstractive+summarization%2E+In+Proceedings+of+the+2nd+Workshop+on+Neural+Machine+Translation+and+Generation%2C+pages+45%E2%80%9354%2C+Melbourne%2C+Aus) [our card](https://md.research.elamur.ai/free/en/papers/fan-2018-controllable-abstractive-summarization/)
4. Sebastian Gehrmann, Zachary Ziegler, and Alexander Rush. 2019. Generating abstractive summaries with finetuned language models. In Proceedings of the 12th International Conference on Natural Language Generation, pages 516–522, Tokyo, Japan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Zachary+Ziegler%2C+and+Alexander+Rush%2E+2019%2E+Generating+abstractive+summaries+with+finetuned+language+models%2E+In+Proceedings+of+the+12th+International+Conference+on+Natural+Language+)
5. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 708–719, New Orleans… [find](https://scholar.google.com/scholar?q=Max+Grusky%2C+Mor+Naaman%2C+and+Yoav+Artzi%2E+2018%2E+Newsroom%3A+A+dataset+of+1%2E3+million+summaries+with+diverse+extractive+strategies%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+th) [our card](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/)
6. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal+and+Ines+Montani%2E+2017%2E+spaCy+2%3A+Natural+language+understanding+with+Bloom+embeddings%2C+convolutional+neural+networks+and+incremental+parsing%2E+To+appear%2E)
7. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2019%2E+Evaluating+the+factual+consistency+of+abstractive+text+summarization%2E)
8. Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Languag… [find](https://scholar.google.com/scholar?q=Wojciech+Kryscinski%2C+Nitish+Shirish+Keskar%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2019%2E+Neural+text+summarization%3A+A+critical+evaluation%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+) [our card](https://md.research.elamur.ai/free/en/papers/kryscinski-2019-neural-text-summarization-critical-evaluation/)
9. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2019%2E+Bart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+l) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
10. Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2019. Don’t say that! making inconsistent dialogue unlikely with unlikelihood training. CoRR, abs/1911.03860. [find](https://scholar.google.com/scholar?q=Margaret+Li%2C+Stephen+Roller%2C+Ilia+Kulikov%2C+Sean+Welleck%2C+Y%2DLan+Boureau%2C+Kyunghyun+Cho%2C+and+Jason+Weston%2E+2019%2E+Don%E2%80%99t+say+that%21+making+inconsistent+dialogue+unlikely+with+unlikelihood+training%2E+CoRR%2C+a) [our card](https://md.research.elamur.ai/free/en/papers/li-2020-don-t-say-that-making/)
11. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu̇lçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. Associatio… [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+dos+Santos%2C+C%CC%A7ag%CC%86lar+GuI%CC%80%E2%80%A1lc%CC%A7ehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+RNNs+and+beyond%2E+In+Proceedings+of+The+20th+SIGN)
12. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As… [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+In+Proceedings+of+the+2018+Co)
13. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Sergey+Edunov%2C+Alexei+Baevski%2C+Angela+Fan%2C+Sam+Gross%2C+Nathan+Ng%2C+David+Grangier%2C+and+Michael+Auli%2E+2019%2E+fairseq%3A+A+fast%2C+extensible+toolkit+for+sequence+modeling%2E+In+Proceedings+of+NAACL%2DHL) [our card](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)
14. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+I%2E+Guyon%2C+U%2E+V%2E+Luxburg%2C+S%2E+Bengio%2C+H%2E)
16. Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731–3741, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sean+Welleck%2C+Jason+Weston%2C+Arthur+Szlam%2C+and+Kyunghyun+Cho%2E+2019%2E+Dialogue+natural+language+inference%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/welleck-2019-dialogue-natural-language-inference/)
17. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtowicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara+) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
18. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328–11339. PMLR. [find](https://scholar.google.com/scholar?q=Jingqing+Zhang%2C+Yao+Zhao%2C+Mohammad+Saleh%2C+and+Peter+Liu%2E+2020%2E+PEGASUS%3A+Pre%2Dtraining+with+extracted+gap%2Dsentences+for+abstractive+summarization%2E+In+Proceedings+of+the+37th+International+Conference+on+) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)