--- title: "NewsQA: A Machine Comprehension Dataset — Trischler 2017" description: "Before NewsQA, large machine comprehension (MC) datasets were either too small for deep learning or generated synthetically." authors: ["Adam Trischler", "Tong Wang", "Xingdi Yuan", "Justin Harris", "Alessandro Sordoni", "Philip Bachman", "Kaheer Suleman"] year: 2017 venue: "Proceedings of the 2nd Workshop on Representation Learning for NLP" source: "https://aclanthology.org/W17-2623/" doi: "10.18653/v1/w17-2623" arxiv: "1611.09830" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / NewsQA: A Machine Comprehension Dataset # NewsQA: A Machine Comprehension Dataset **Authors:** [Adam Trischler](https://md.research.elamur.ai/free/en/authors/adam-trischler/), [Tong Wang](https://md.research.elamur.ai/free/en/authors/tong-wang/), [Xingdi Yuan](https://md.research.elamur.ai/free/en/authors/xingdi-yuan/), [Justin Harris](https://md.research.elamur.ai/free/en/authors/justin-harris/), [Alessandro Sordoni](https://md.research.elamur.ai/free/en/authors/alessandro-sordoni/), Philip Bachman, [Kaheer Suleman](https://md.research.elamur.ai/free/en/authors/kaheer-suleman/) · **Year:** 2017 · **Venue:** Proceedings of the 2nd Workshop on Representation Learning for NLP · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W17-2623/) · [PDF](https://aclanthology.org/W17-2623.pdf) · [DOI 10.18653/v1/w17-2623](https://doi.org/10.18653/v1/w17-2623) · [arXiv 1611.09830](https://arxiv.org/abs/1611.09830) · [OpenAlex](https://openalex.org/W2557764419) **Citations:** 692 citations (openalex-snapshot · 2026-10-14) · [396](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Before NewsQA, large machine comprehension (MC) datasets were either too small for deep learning or generated synthetically. Even SQuAD, which uses crowdsourced questions, was already close to human-level scores. The Microsoft Maluuba team built NewsQA, a span-extraction corpus of 119,633 crowdsourced questions about 12,744 CNN articles. The people who wrote the questions saw only each article's headline and highlights. This led to curiosity-driven questions that use different wording from the article, and some of them cannot be answered from it. A separate group of workers then marked answer spans in the full article, and a validation round settled disagreements. Compared with SQuAD, the documents are much longer, and more questions need inference or need facts combined from several sentences. Humans scored 69.4% F1. The best baseline, FastQA, was 13.3 points lower, while the same human–model gap on SQuAD was under 2 points. NewsQA is freely available and is a harder benchmark that complements SQuAD. It can be used to test cross-domain transfer and to train models to recognise questions without an answer. ## Key points - **Collection pipeline:** articles were sampled at random from 90,266 CNN articles and split 90/5/5 into train/dev/test. Each question was answered by 2.73 workers on average. About 43.2% of questions went to validation, and 86.0% end up with an answer that at least two workers agreed on. - **Unanswerable questions:** questions without an agreed answer are kept in the dataset and marked. Answerers could also choose a null answer. - **Document length:** NewsQA articles average 30.7 sentences, compared with 4.9 for SQuAD paragraphs. A simple isf sentence-selection heuristic finds the answer sentence 79.6% of the time on SQuAD but only 35.4% on NewsQA. The gap stays large even when SQuAD paragraphs are joined into longer documents. - **Human scores:** on 1,000 dev questions, humans scored 69.4 F1, 46.5 EM, 56.0 BLEU and 3.596 CIDEr. The authors say the low EM partly reflects equally correct answers with different span boundaries. - **Neural baselines:** FastQA beat the authors' mLSTM on both datasets. Model errors were highest on synthesis and inference questions, and the model handled named-entity answers best. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 396 papers of our library cite it (of the 140884 whose bibliographies we read); 46 of them have a page here, 350 not yet. - [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/) - [STARC: Structured Annotations for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/berzak-2020-starc-structured-annotations-reading-comprehension/) - [QuAC: Question Answering in Context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/) - [A Span-Extraction Dataset for Chinese Machine Reading Comprehension](https://md.research.elamur.ai/free/en/papers/cui-2018-span-extraction-dataset-chinese/) - [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/) - [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/) - [English Machine Reading Comprehension Datasets: A Survey](https://md.research.elamur.ai/free/en/papers/dzendzik-2021-english-machine-reading-comprehension-datasets/) - [ELI5: Long Form Question Answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/) - [HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark](https://md.research.elamur.ai/free/en/papers/findings-2023-cohen-heq-large-diverse-hebrew-reading/) - [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/) - [DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications](https://md.research.elamur.ai/free/en/papers/he-2017-dureader-chinese-machine-reading/) - [CTRLsum: Towards Generic Controllable Text Summarization](https://md.research.elamur.ai/free/en/papers/he-2022-ctrlsum-generic-controllable-text-summarization/) - [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/) - [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/) - [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/) - [How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/) - [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/) - [Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/) - [UNIFIEDQA: Crossing Format Boundaries with a Single QA System](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/) - [Inquisitive Question Generation for High Level Text Comprehension](https://md.research.elamur.ai/free/en/papers/ko-2020-inquisitive-question-generation-high-level/) - [The NarrativeQA Reading Comprehension Challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/) - [Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds](https://md.research.elamur.ai/free/en/papers/labutov-2018-multi-relational-question-answering-narratives/) - [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/) - [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/) - [UNQOVERing Stereotyping Biases via Underspecified Questions](https://md.research.elamur.ai/free/en/papers/li-2020-unqovering-stereotyping-biases-underspecified-questions/) - [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/lin-2023-ra-dit-retrieval-augmented-dual/) - [Entity-Based Knowledge Conflicts in Question Answering](https://md.research.elamur.ai/free/en/papers/longpre-2021-entity-based-knowledge-conflicts-question/) - [DocVQA: A Dataset for VQA on Document Images](https://md.research.elamur.ai/free/en/papers/mathew-2020-docvqa-dataset-vqa-document-images/) - [The Natural Language Decathlon: Multitask Learning as Question Answering](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/) - [emrQA: A Large Corpus for Question Answering on Electronic Medical Records](https://md.research.elamur.ai/free/en/papers/pampari-2018-emrqa-large-corpus-question-answering/) - [Know What You Don't Know: Unanswerable Questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/) - [A guide to the dataset explosion in QA, NLI, and commonsense reasoning](https://md.research.elamur.ai/free/en/papers/rogers-2020-guide-dataset-explosion-qa-nli/) - [DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension](https://md.research.elamur.ai/free/en/papers/saha-2018-duorc-complex-language-understanding-paraphrased/) - [MLSUM: The Multilingual Summarization Corpus](https://md.research.elamur.ai/free/en/papers/scialom-2020-mlsum-multilingual-summarization-corpus/) - [QuestEval: Summarization Asks for Fact-based Evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/) - [Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering](https://md.research.elamur.ai/free/en/papers/siriwardhana-2023-improving-domain-adaptation-retrieval-augmented/) - [ASQA: Factoid Questions Meet Long-Form Answers](https://md.research.elamur.ai/free/en/papers/stelmakh-2022-asqa-factoid-questions-meet-long/) - [MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension](https://md.research.elamur.ai/free/en/papers/talmor-2019-multiqa-empirical-investigation-generalization-transfer/) - [Large Language Models for Data Annotation and Synthesis: A Survey](https://md.research.elamur.ai/free/en/papers/tan-2024-large-language-models-data-annotation/) - [Exploring and Predicting Transferability across NLP Tasks](https://md.research.elamur.ai/free/en/papers/vu-2020-exploring-predicting-transferability-across-nlp/) - [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/) - [BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/xu-2019-bert-post-training-review-reading/) - [Curriculum Learning for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/xu-2020-curriculum-learning-natural-language-understanding/) - [LinkBERT: Pretraining Language Models with Document Links](https://md.research.elamur.ai/free/en/papers/yasunaga-2022-linkbert-pretraining-language-models-document/) - [ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/) - [A Robustly Optimized BERT Pre-training Approach with Post-training](https://md.research.elamur.ai/free/en/papers/zhuang-2021-robustly-optimized-bert-pre-training/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{trischler2017newsqa, title = {{NewsQA: A Machine Comprehension Dataset}}, author = {Trischler, Adam and Wang, Tong and Yuan, Xingdi and Harris, Justin and Sordoni, Alessandro and Bachman, Philip and Suleman, Kaheer}, booktitle = {Proceedings of the 2nd Workshop on Representation Learning for NLP}, year = {2017}, doi = {10.18653/v1/w17-2623}, eprint = {1611.09830}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/w17-2623} } ``` - [← previous New perspectives on cohesion and coherence: Implications for translation](https://md.research.elamur.ai/free/en/papers/kunz-2017-new-perspectives-cohesion-coherence/) - [next → No case for Case in locality: Case does not help interpretation when intervention blocks A-bar chains](https://md.research.elamur.ai/free/en/papers/lingbulk-2017-friedmann-no-case-case-locality-case/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:1610.00956 . — [arXiv:1610.00956](https://arxiv.org/abs/1610.00956) [find](https://scholar.google.com/scholar?q=Ondrej+Bajgar%2C+Rudolf+Kadlec%2C+and+Jan+Kleindienst%2E+2016%2E+Embracing+data+abundance%3A+Booktest+dataset+for+reading+comprehension%2E+arXiv+preprint+arXiv%3A1610%2E00956+%2E+%E2%80%94+%5BarXiv%3A1610%2E00956%5D%28https%3A%2F%2Farxiv%2Eorg%2F)