---
title: "Question Answering as an Automatic Evaluation Metric for News"
description: "The authors propose APES (Answering Performance for Evaluation of Summaries), an extrinsic… 31 references, cited by 65 papers here."
authors: ["Matan Eyal", "Tal Baumel", "Michael Elhadad"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1395/"
doi: "10.18653/v1/n19-1395"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/eyal-2019-question-answering-as-automatic-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/eyal-2019-question-answering-as-automatic-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Question Answering as an Automatic Evaluation Metric for News Article Summarization

# Question Answering as an Automatic Evaluation Metric for News Article Summarization

**Authors:** [Matan Eyal](https://md.research.elamur.ai/free/en/authors/matan-eyal/), Tal Baumel and [Michael Elhadad](https://md.research.elamur.ai/free/en/authors/michael-elhadad/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2019](https://md.research.elamur.ai/free/en/venues/naacl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1395/) · [PDF](https://aclanthology.org/N19-1395.pdf) · [DOI 10.18653/v1/n19-1395](https://doi.org/10.18653/v1/n19-1395) · [OpenAlex](https://openalex.org/W2945760033)

**Citations:** 91 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/eyal-2019-question-answering-as-automatic-evaluation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large news summarization corpora such as CNN/Daily Mail come with just one reference summary per article. With a single reference, ROUGE agrees less well with human judgments. The authors propose APES (Answering Performance for Evaluation of Summaries), an extrinsic metric that checks whether a summary carries the key content of its source. A reading-comprehension model (Chen et al., 2016, trained on CNN) receives the candidate summary together with fill-in-the-blank questions about salient entities. In CNN/Daily Mail, these questions come from the reference summaries. The APES score is the share of questions answered correctly.

The authors validate APES against the manual Pyramid and Responsiveness scores from the TAC 2011 AESOP task and find it a useful complement to ROUGE. APES also penalizes gold summaries whose words have been shuffled, which ROUGE-1 does not. They then build an abstractive PyTorch summarizer that boosts attention on predicted salient entities. It clearly beats the baselines on APES and gains about one ROUGE F-point. Both the metric library and the summarizer are released, giving news summarization a cheap, content-focused evaluation tool.

## Key points
- The QA component reaches 72.4 (CNN) and 75.8 (Daily Mail) accuracy. Feeding it the full source documents yields 61.1% APES, which serves as a practical upper bound.
- On the CNN/Daily Mail test set, the APES-optimized model scores 46.1 versus 39.8 for the baselines, including See et al. (2017). Supplying gold entity positions as an oracle lifts this only to 46.3, so the entity-saliency predictor is already strong.
- On validation, the model scores 46.6 APES, 41.2 R1, 18.4 R2 and 38.1 RL.
- In the error analysis, 1,630 of 11,490 test summaries got perfect APES and 1,691 got zero. Salient entities were mentioned more densely in the sources behind perfect summaries: 4.9 versus 3.6 mentions per entity.
- Two packages are released: an APES/ROUGE evaluation library (github.com/mataney/APES) and the summarizer with trained models (github.com/mataney/APES-optimizer). The method targets entity-centric news documents of roughly 1k words.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 240 works · 123,537 citations
- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 85 works · 66,118 citations
- [Ben-Gurion University of the Negev](https://md.research.elamur.ai/free/en/universities/ben-gurion-university-of-the-negev/) — University, Beersheba, Israel · 36 works · 3,066 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (3): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 242 works · [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 159 works

## Cite

Cite as: Matan Eyal, Tal Baumel and Michael Elhadad. 2019. Question Answering as an Automatic Evaluation Metric for News Article Summarization. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{eyal2019question,
  title = {{Question Answering as an Automatic Evaluation Metric for News Article Summarization}},
  author = {Matan Eyal, Tal Baumel and Michael Elhadad},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1395},
  url = {https://doi.org/10.18653/v1/n19-1395}
}
```

- [← previous Scoring Sentence Singletons and Pairs for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/lebanoff-2019-scoring-sentence-singletons-pairs-abstractive/)
- [next → Evaluating Style Transfer for Text](https://md.research.elamur.ai/free/en/papers/mir-2019-evaluating-style-transfer-text/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consortium, Philadelphia, 4:1. [find](https://scholar.google.com/scholar?q=David+Graff%2C+Junbo+Kong%2C+Ke+Chen%2C+and+Kazuaki+Maeda%2E+2003%2E+English+gigaword%2E+Linguistic+Data+Consortium%2C+Philadelphia%2C+4%3A1%2E)
2. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In Advances in Neural Information Processing Systems, pages 1693– 1701.
3. Tsutomu Hirao, Hidetaka Kamigaito, and Masaaki Nagata. 2018. Automatic pyramid evaluation exploiting edu-based extractive reference summaries. In EMNLP. [find](https://scholar.google.com/scholar?q=Tsutomu+Hirao%2C+Hidetaka+Kamigaito%2C+and+Masaaki+Nagata%2E+2018%2E+Automatic+pyramid+evaluation+exploiting+edu%2Dbased+extractive+reference+summaries%2E+In+EMNLP%2E)
4. Stacy President Hobson, Bonnie J Dorr, Christof Monz, and Richard Schwartz. 2007. Task-based evaluation of text summarization using relevance prediction. Information Processing & Management, 43(6):1482–1499. [find](https://scholar.google.com/scholar?q=Stacy+President+Hobson%2C+Bonnie+J+Dorr%2C+Christof+Monz%2C+and+Richard+Schwartz%2E+2007%2E+Task%2Dbased+evaluation+of+text+summarization+using+relevance+prediction%2E+Information+Processing+%26+Management%2C+43%286%29%3A148)
5. Matthew Honnibal and Mark Johnson. 2015. [An improved non-monotonic transition system for dependency parsing](https://md.research.elamur.ai/free/en/papers/honnibal-2015-improved-non-monotonic-transition-system/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1373–1378, Lisbon, Portugal. Association for Computational Linguistics.
6. Eduard Hovy, Chin-Yew Lin, Liang Zhou, and Junichi Fukumoto. 2006. Automated summarization evaluation with basic elements. In Proceedings of the Fifth Conference on Language Resources and Evaluation (LREC 2006), pages 604–611. Citeseer. [find](https://scholar.google.com/scholar?q=Eduard+Hovy%2C+Chin%2DYew+Lin%2C+Liang+Zhou%2C+and+Junichi+Fukumoto%2E+2006%2E+Automated+summarization+evaluation+with+basic+elements%2E+In+Proceedings+of+the+Fifth+Conference+on+Language+Resources+and+Evaluation+%28)
7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473.
8. Hongyan Jing, Regina Barzilay, Kathleen McKeown, and Michael Elhadad. 1998. Summarization evaluation methods: Experiments and analysis. In AAAI symposium on intelligent summarization, pages 51– 59. [find](https://scholar.google.com/scholar?q=Hongyan+Jing%2C+Regina+Barzilay%2C+Kathleen+McKeown%2C+and+Michael+Elhadad%2E+1998%2E+Summarization+evaluation+methods%3A+Experiments+and+analysis%2E+In+AAAI+symposium+on+intelligent+summarization%2C+pages+51%E2%80%93+59%2E)
9. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. [Deep communicating agents for abstractive summarization](https://md.research.elamur.ai/free/en/papers/celikyilmaz-2018-deep-communicating-agents-abstractive-summarization/). arXiv preprint arXiv:1803.10357.
10. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Open-NMT: Open-source toolkit for neural machine translation. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Guillaume+Klein%2C+Yoon+Kim%2C+Yuntian+Deng%2C+Jean+Senellart%2C+and+Alexander+M%2E+Rush%2E+2017%2E+Open%2DNMT%3A+Open%2Dsource+toolkit+for+neural+machine+translation%2E+In+Proc%2E+ACL%2E)
11. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. [A thorough examination of the cnn/daily mail reading comprehension task](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/). arXiv preprint arXiv:1606.02858.
12. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out: Proceedings of the ACL-04 workshop, volume 8. Barcelona, Spain.
13. Annie Louis and Ani Nenkova. 2013. [Automatically assessing machine summary content without a gold standard](https://md.research.elamur.ai/free/en/papers/louis-2013-automatically-assessing-machine-summary-content/). Computational Linguistics, 39(2):267– 300.
14. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3104%E2%80%933112%2E)
15. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. [Summarunner: A recurrent neural network based sequence model for extractive summarization of documents](https://md.research.elamur.ai/free/en/papers/nallapati-2016-summarunner-recurrent-neural-network-based/). hiP (yi= 1— hi, si, d), 1:1.
16. Yi Tay, Minh C Phan, Luu Anh Tuan, and Siu Cheung Hui. 2017. Skipflow: Incorporating neural coherence features for end-to-end automatic text scoring. arXiv preprint arXiv:1711.04981. — [arXiv:1711.04981](https://arxiv.org/abs/1711.04981) [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Minh+C+Phan%2C+Luu+Anh+Tuan%2C+and+Siu+Cheung+Hui%2E+2017%2E+Skipflow%3A+Incorporating+neural+coherence+features+for+end%2Dto%2Dend+automatic+text+scoring%2E+arXiv+preprint+arXiv%3A1711%2E04981%2E+%E2%80%94+%5BarXiv%3A1711%2E049)
17. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Caglar+Gulcehre%2C+Bing+Xiang%2C+et+al%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2E)
18. Ravikiran Vadlapudi and Rahul Katragadda. 2010. On automated evaluation of readability of summaries: Capturing grammaticality, focus, structure and coherence. In Proceedings of the NAACL HLT 2010 student research workshop, pages 7–12. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ravikiran+Vadlapudi+and+Rahul+Katragadda%2E+2010%2E+On+automated+evaluation+of+readability+of+summaries%3A+Capturing+grammaticality%2C+focus%2C+structure+and+coherence%2E+In+Proceedings+of+the+NAACL+HLT+2010+stud)
19. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. [Ranking sentences for extractive summarization with reinforcement learning](https://md.research.elamur.ai/free/en/papers/narayan-2018-ranking-sentences-extractive-summarization-reinforcement/). arXiv preprint arXiv:1802.08636.
20. Ani Nenkova, Rebecca Passonneau, and Kathleen McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Language Processing (TSLP), 4(2):4. [find](https://scholar.google.com/scholar?q=Ani+Nenkova%2C+Rebecca+Passonneau%2C+and+Kathleen+McKeown%2E+2007%2E+The+pyramid+method%3A+Incorporating+human+content+selection+variation+in+summarization+evaluation%2E+ACM+Transactions+on+Speech+and+Language+Pr)
21. Karolina Owczarzak. 2009. Dependency-based evaluation summaries. In ACL/IJCNLP. [find](https://scholar.google.com/scholar?q=Karolina+Owczarzak%2E+2009%2E+Dependency%2Dbased+evaluation+summaries%2E+In+ACL%2FIJCNLP%2E)
22. Depeval(summ): for automatic Karolina Owczarzak and Hoa Trang Dang. 2011. Overview of the tac 2011 summarization track: Guided task and aesop task. In Proceedings of the Text Analysis Conference (TAC 2011), Gaithersburg, Maryland, USA, November. [find](https://scholar.google.com/scholar?q=Depeval%28summ%29%3A+for+automatic+Karolina+Owczarzak+and+Hoa+Trang+Dang%2E+2011%2E+Overview+of+the+tac+2011+summarization+track%3A+Guided+task+and+aesop+task%2E+In+Proceedings+of+the+Text+Analysis+Conference+%28TAC+)
23. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS-W. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+pytorch%2E+In+NIPS%2DW%2E)
24. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. [A deep reinforced model for abstractive summarization](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/). arXiv preprint arXiv:1705.04304.
25. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685. [find](https://scholar.google.com/scholar?q=Alexander+M+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+arXiv+preprint+arXiv%3A1509%2E00685%2E)
26. Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752. [find](https://scholar.google.com/scholar?q=Evan+Sandhaus%2E+2008%2E+The+new+york+times+annotated+corpus%2E+Linguistic+Data+Consortium%2C+Philadelphia%2C+6%2812%29%3Ae26752%2E)
27. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+arXiv+preprint+arXiv%3A1704%2E04368%2E)
28. Josef Steinberger and Karel Ježek. 2012. Evaluation measures for text summarization. Computing and Informatics, 28(2):251–275. [find](https://scholar.google.com/scholar?q=Josef+Steinberger+and+Karel+Jez%CC%8Cek%2E+2012%2E+Evaluation+measures+for+text+summarization%2E+Computing+and+Informatics%2C+28%282%29%3A251%E2%80%93275%2E)
29. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692–2700. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Meire+Fortunato%2C+and+Navdeep+Jaitly%2E+2015%2E+Pointer+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+2692%E2%80%932700%2E)
30. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridg)
31. Qian Yang, Rebecca J. Passonneau, and Gerard de Melo. 2016. Peak: Pyramid evaluation via automated knowledge extraction. In AAAI. [find](https://scholar.google.com/scholar?q=Qian+Yang%2C+Rebecca+J%2E+Passonneau%2C+and+Gerard+de+Melo%2E+2016%2E+Peak%3A+Pyramid+evaluation+via+automated+knowledge+extraction%2E+In+AAAI%2E)

*Updated 24 October 2026*