---
title: "A Thorough Examination of the CNN/Daily Mail — Chen 2016"
description: "This paper asks how much real language understanding the large cloze-style reading comprehension datasets from Hermann et al."
authors: ["Danqi Chen", "Jason Bolton", "Christopher D. Manning"]
year: 2016
venue: "arXiv:1606.02858"
source: "https://arxiv.org/abs/1606.02858"
doi: "10.48550/arxiv.1606.02858"
arxiv: "1606.02858"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

# A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

**Authors:** [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/), [Jason Bolton](https://md.research.elamur.ai/free/en/authors/jason-bolton/), [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/) · **Year:** 2016 · **Venue:** arXiv:1606.02858 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1606.02858) · [PDF](https://arxiv.org/pdf/1606.02858) · [DOI 10.48550/arxiv.1606.02858](https://doi.org/10.48550/arxiv.1606.02858) · [arXiv 1606.02858](https://arxiv.org/abs/1606.02858) · [OpenAlex](https://openalex.org/W2962809918)

**Citations:** 491 citations (openalex-snapshot · 2026-10-14) · [198](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
This paper asks how much real language understanding the large cloze-style reading comprehension datasets from Hermann et al. (2015) actually require. Those datasets were built from CNN and Daily Mail news articles. Each question is a summary bullet point with one anonymized entity removed, and the system must recover that entity from the article. The authors study the task in two ways. First, they build two straightforward systems: a feature-based entity ranking classifier and a simplified attention-based neural model derived from the AttentiveReader. Second, they hand-label 100 CNN questions by the kind of reasoning each one needs. The neural model reaches 73.6% on CNN and 76.6% on Daily Mail, which is 7–10 points above earlier published results. The hand analysis shows that most answerable questions can be solved by finding one relevant sentence and matching it, often through paraphrase. About a quarter of the questions are broken by coreference errors or anonymization, which leaves little room for further gains. The authors conclude that the benchmark is easier and noisier than it first appeared, so it says little about deeper multi-sentence reasoning.

## Key points
- **Data:** CNN has about 380k training questions and Daily Mail about 879k. Passages average roughly 30 sentences and 760–810 tokens, with around 26 candidate entities per passage. Entities are replaced with `@entityN` markers after NER and coreference resolution.
- **Entity-centric classifier:** it ranks candidate entities with hand-built features, including frequency, first position, n-gram match around the placeholder, word distance, sentence co-occurrence and dependency-parse matches. On its own it already beats the earlier baselines.
- **Hand analysis of 100 CNN items:** 13 are exact match, 41 paraphrasing, 19 partial clue, 2 multi-sentence, 8 coreference errors and 17 ambiguous or hard. On this sample the classifier scores 66% and the neural net 74%.
- **Paraphrasing:** the neural model's advantage comes mainly from paraphrasing and partial-clue questions, where learned distributed representations handle lexical variation better. Both systems get all exact-match questions right and fail on most noisy ones.
- **Code:** released at github.com/danqi/rc-cnn-dailymail. The data comes from DeepMind's rc-data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/)

## Cited by

Cited by

198 papers of our library cite it (of the 140884 whose bibliographies we read); 34 of them have a page here, 164 not yet.

- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://md.research.elamur.ai/free/en/papers/alt-2020-tacred-revisited-thorough-evaluation-tacred/)
- [STARC: Structured Annotations for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/berzak-2020-starc-structured-annotations-reading-comprehension/)
- [Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization](https://md.research.elamur.ai/free/en/papers/chen-2020-multi-view-sequence-sequence-models/)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/du-2017-learning-ask-neural-question-generation/)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
- [Evaluating Models' Local Decision Boundaries via Contrast Sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/)
- [A Deterministic Algorithm for Bridging Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/hou-2018-deterministic-algorithm-bridging-anaphora-resolution/)
- [Enhanced Word Representations for Bridging Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/hou-2018-enhanced-word-representations-bridging-anaphora/)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
- [Text Understanding with the Attention Sum Reader Network](https://md.research.elamur.ai/free/en/papers/kadlec-2016-text-understanding-attention-sum-reader/)
- [How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/)
- [Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)
- [The NarrativeQA Reading Comprehension Challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)
- [Neural Text Summarization: A Critical Evaluation](https://md.research.elamur.ai/free/en/papers/kryscinski-2019-neural-text-summarization-critical-evaluation/)
- [Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds](https://md.research.elamur.ai/free/en/papers/labutov-2018-multi-relational-question-answering-narratives/)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
- [Reasoning Over Paragraph Effects in Situations](https://md.research.elamur.ai/free/en/papers/lin-2019-reasoning-over-paragraph-effects-situations/)
- [Event Extraction as Machine Reading Comprehension](https://md.research.elamur.ai/free/en/papers/liu-2020-event-extraction-as-machine-reading/)
- [LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/)
- [Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
- [Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning](https://md.research.elamur.ai/free/en/papers/razeghi-2022-impact-pretraining-term-frequencies-few/)
- [A guide to the dataset explosion in QA, NLI, and commonsense reasoning](https://md.research.elamur.ai/free/en/papers/rogers-2020-guide-dataset-explosion-qa-nli/)
- [Bidirectional Attention Flow for Machine Comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)
- [DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension](https://md.research.elamur.ai/free/en/papers/sun-2019-dream-challenge-data-set-models/)
- [Machine Comprehension Using Match-LSTM and Answer Pointer](https://md.research.elamur.ai/free/en/papers/wang-2016-machine-comprehension-using-match-lstm/)
- [Gated Self-Matching Networks for Reading Comprehension and Question Answering](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/)
- [Constructing Datasets for Multi-hop Reading Comprehension Across Documents](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/)
- [Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework](https://md.research.elamur.ai/free/en/papers/white-2017-inference-is-everything-recasting-semantic/)
- [Dynamic Coattention Networks For Question Answering](https://md.research.elamur.ai/free/en/papers/xiong-2016-dynamic-coattention-networks-question-answering/)
- [ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2016thorough,
  title = {{A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task}},
  author = {Chen, Danqi and Bolton, Jason and Manning, Christopher D.},
  year = {2016},
  doi = {10.48550/arxiv.1606.02858},
  eprint = {1606.02858},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1606.02858}
}
```

- [← previous A Stacking Gated Neural Architecture for Implicit Discourse Relation Classification](https://md.research.elamur.ai/free/en/papers/qin-2016-stacking-gated-neural-architecture-implicit/)
- [next → ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs](https://md.research.elamur.ai/free/en/papers/yin-2016-abcnn-attention-based-convolutional-neural/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In Empirical Methods in Natural Language Processing (EMNLP), pages 1499–1510. [find](https://scholar.google.com/scholar?q=Jonathan+Berant%2C+Vivek+Srikumar%2C+Pei%2DChun+Chen%2C+Abby+Vander+Linden%2C+Brittany+Harding%2C+Brad+Huang%2C+Peter+Clark%2C+and+Christopher+D%2E+Manning%2E+2014%2E+Modeling+biological+processes+for+reading+comprehension) [our card](https://md.research.elamur.ai/free/en/papers/berant-2014-modeling-biological-processes-reading-comprehension/)
2. Christopher J.C. Burges. 2013. Towards the machine comprehension of text: An essay. Technical report, Microsoft Research Technical Report MSR-TR-2013-125. [find](https://scholar.google.com/scholar?q=Christopher+J%2EC%2E+Burges%2E+2013%2E+Towards+the+machine+comprehension+of+text%3A+An+essay%2E+Technical+report%2C+Microsoft+Research+Technical+Report+MSR%2DTR%2D2013%2D125%2E)
3. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Empirical Methods in Natural Language Processing (EMNLP), pages 740–750. [find](https://scholar.google.com/scholar?q=Danqi+Chen+and+Christopher+Manning%2E+2014%2E+A+fast+and+accurate+dependency+parser+using+neural+networks%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+740%E2%80%93750%2E) [our card](https://md.research.elamur.ai/free/en/papers/missing-2014-chen-fast-accurate-dependency-parser-neural/)
4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734… [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+rnn+encoder%E2%80%93decoder+for+statistic)
5. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS), pages 1684–1692. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+Advances+in+Neural+Information) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
6. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Antoine+Bordes%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2016%2E+The+goldilocks+principle%3A+Reading+children%E2%80%99s+books+with+explicit+memory+representations%2E+In+International+Conference+on+Learning+Repre) [our card](https://md.research.elamur.ai/free/en/papers/hill-2015-goldilocks-principle-reading-children-s/)
7. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Rudolf+Kadlec%2C+Martin+Schmid%2C+Ondrej+Bajgar%2C+and+Jan+Kleindienst%2E+2016%2E+Text+understanding+with+the+attention+sum+reader+network%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/kadlec-2016-text-understanding-attention-sum-reader/)
8. Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, and Kentaro Inui. 2016. Dynamic entity representation with max-pooling improves machine reading. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Sosuke+Kobayashi%2C+Ran+Tian%2C+Naoaki+Okazaki%2C+and+Kentaro+Inui%2E+2016%2E+Dynamic+entity+representation+with+max%2Dpooling+improves+machine+reading%2E+In+North+American+Association+for+Computational+Linguistics)
9. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Ankit+Kumar%2C+Ozan+Irsoy%2C+Peter+Ondruska%2C+Mohit+Iyyer%2C+James+Bradbury%2C+Ishaan+Gulrajani%2C+Victor+Zhong%2C+Romain+Paulus%2C+and+Richard+Socher%2E+2016%2E+Ask+me+anything%3A+Dynamic+memory+networks+for+natural+lang) [our card](https://md.research.elamur.ai/free/en/papers/kumar-2015-ask-me-anything-dynamic-memory/)
10. Moontae Lee, Xiaodong He, Wen-tau Yih, Jianfeng Gao, Li Deng, and Paul Smolensky. 2016. Reasoning in vector space: An exploratory study of question answering. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Moontae+Lee%2C+Xiaodong+He%2C+Wen%2Dtau+Yih%2C+Jianfeng+Gao%2C+Li+Deng%2C+and+Paul+Smolensky%2E+2016%2E+Reasoning+in+vector+space%3A+An+exploratory+study+of+question+answering%2E+In+International+Conference+on+Learning+R)
11. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421. [find](https://scholar.google.com/scholar?q=Thang+Luong%2C+Hieu+Pham%2C+and+Christopher+D%2E+Manning%2E+2015%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+1412%E2%80%9314)
12. Peter Norvig. 1978. A Unified Theory of Inference for Text Understanding. Ph.D. thesis, University of California, Berkeley. [find](https://scholar.google.com/scholar?q=Peter+Norvig%2E+1978%2E+A+Unified+Theory+of+Inference+for+Text+Understanding%2E+Ph%2ED%2E+thesis%2C+University+of+California%2C+Berkeley%2E)
13. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+1532%E2%80%931543%2E)
14. Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 193–203. [find](https://scholar.google.com/scholar?q=Matthew+Richardson%2C+Christopher+J%2EC%2E+Burges%2C+and+Erin+Renshaw%2E+2013%2E+MCTest%3A+A+challenge+dataset+for+the+open%2Ddomain+machine+comprehension+of+text%2E+In+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/en/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/)
15. Mrinmaya Sachan, Kumar Dubey, Eric Xing, and Matthew Richardson. 2015. Learning answer-entailing structures for machine comprehension. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL/IJCNLP), pages 239–249. [find](https://scholar.google.com/scholar?q=Mrinmaya+Sachan%2C+Kumar+Dubey%2C+Eric+Xing%2C+and+Matthew+Richardson%2E+2015%2E+Learning+answer%2Dentailing+structures+for+machine+comprehension%2E+In+Association+for+Computational+Linguistics+and+International+Jo)
16. Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems (NIPS), pages 2431–2439. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+arthur+szlam%2C+Jason+Weston%2C+and+Rob+Fergus%2E+2015%2E+End%2Dto%2Dend+memory+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2C+pages+2431%E2%80%932439%2E)
17. Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL/IJCNLP), pages 700–706. [find](https://scholar.google.com/scholar?q=Hai+Wang%2C+Mohit+Bansal%2C+Kevin+Gimpel%2C+and+David+McAllester%2E+2015%2E+Machine+comprehension+with+syntax%2C+frames%2C+and+semantics%2E+In+Association+for+Computational+Linguistics+and+International+Joint+Confere)
18. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Jason+Weston%2C+Sumit+Chopra%2C+and+Antoine+Bordes%2E+2015%2E+Memory+networks%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
19. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2016. Towards AI-complete question answering: A set of prerequisite toy tasks. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Jason+Weston%2C+Antoine+Bordes%2C+Sumit+Chopra%2C+and+Tomas+Mikolov%2E+2016%2E+Towards+AI%2Dcomplete+question+answering%3A+A+set+of+prerequisite+toy+tasks%2E+In+International+Conference+on+Learning+Representations+%28I) [our card](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/)
20. Qiang Wu, Christopher J. Burges, Krysta M. Svore, and Jianfeng Gao. 2010. Adapting boosting for information retrieval measures. Information Retrieval, pages 254–270. [find](https://scholar.google.com/scholar?q=Qiang+Wu%2C+Christopher+J%2E+Burges%2C+Krysta+M%2E+Svore%2C+and+Jianfeng+Gao%2E+2010%2E+Adapting+boosting+for+information+retrieval+measures%2E+Information+Retrieval%2C+pages+254%E2%80%93270%2E)