--- title: "Who did What: A Large-Scale Person-Centered Cloze Dataset" description: "Large cloze-style reading comprehension benchmarks such as CNN/Daily Mail build questions from… 3 references, cited by 59 papers here." authors: ["Takeshi Onishi", "Hai Wang", "Mohit Bansal", "Kevin Gimpel", "David McAllester"] year: 2016 venue: "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/D16-1241/" doi: "10.18653/v1/d16-1241" arxiv: "1608.05457" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/onishi-2016-who-did-what-large-scale/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/onishi-2016-who-did-what-large-scale/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Who did What: A Large-Scale Person-Centered Cloze Dataset # Who did What: A Large-Scale Person-Centered Cloze Dataset **Authors:** Takeshi Onishi, [Hai Wang](https://md.research.elamur.ai/free/en/authors/hai-wang/), [Mohit Bansal](https://md.research.elamur.ai/free/en/authors/mohit-bansal/), [Kevin Gimpel](https://md.research.elamur.ai/free/en/authors/kevin-gimpel/) and David McAllester · **Year:** 2016 · **Venue:** [Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2016](https://md.research.elamur.ai/free/en/venues/emnlp/2016/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/D16-1241/) · [PDF](https://aclanthology.org/D16-1241.pdf) · [DOI 10.18653/v1/d16-1241](https://doi.org/10.18653/v1/d16-1241) · [arXiv 1608.05457](https://arxiv.org/abs/1608.05457) · [OpenAlex](https://openalex.org/W2963681467) **Citations:** 114 citations (OpenAlex, 2026-10-14) · [59](https://md.research.elamur.ai/free/en/papers/onishi-2016-who-did-what-large-scale/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary Large cloze-style reading comprehension benchmarks such as CNN/Daily Mail build questions from article summaries and anonymize entities. This often lets models succeed through surface matching or frequency heuristics. Onishi et al. address this with Who-did-What (WDW), a set of over 200,000 multiple-choice problems drawn from the LDC English Gigaword newswire corpus. Each problem pairs two separate articles. A person mention is blanked out of the opening sentence of one article, and Apache Lucene retrieves a different article covering the same events, published within two weeks, to serve as the passage. Answer candidates are the real, non-anonymized person names found in that passage. The authors then solve an optimization that discards items easily answered by four simple heuristics, pushing those heuristics down to chance level. Humans still answer 84% correctly, yet established neural readers score notably lower than on CNN. The resulting human–machine gap makes WDW a harder test of semantic understanding. Because it needs no human-written summaries, the construction method can also be applied to other news collections. ## Key points - **Construction pipeline:** Stanford NER and the Stanford PCFG parser find person mentions, and each blank removes the entire noun phrase headed by the person, so descriptors cannot give the answer away. Passages that overlap the question too closely are filtered out, and only problems with 2–5 choices are kept. - **Baseline suppression:** The four suppressed baselines are first person in the passage, most frequent person, a Gigaword 5-gram language model, and unigram last-name frequency. Suppression removed 49.9% of the problems, and every baseline fell from 0.43–0.61 accuracy to about 0.32, which is random level. - **Splits:** - train: 127,786 problems - "relaxed" train (less suppression): 185,978 problems - validation: 10,000 problems - test: 10,000 problems Validation and test come from the most recent articles, which limits topical overlap with training. Problems average about 3.4–3.5 choices. - **Human vs. model performance:** Human accuracy is 84%, compared with roughly 75% on CNN and 82% on CBT named entities. - Attentive Reader drops from 63% on CNN to 55% on WDW. - Attention Sum Reader drops from 70% to 59%. - Readers that exploit answer frequency (AS, GA) lose up to 17 points. - **Availability:** The dataset is distributed via tticnlp.github.io/who_did_what. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Toyota Technological Institute at Chicago](https://md.research.elamur.ai/free/en/universities/toyota-technological-institute-at-chicago/) — University, Chicago, United States · 29 works · 10,273 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 260 works ## Cite Cite as: Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel and David McAllester. 2016. Who did What: A Large-Scale Person-Centered Cloze Dataset. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{onishi2016who, title = {{Who did What: A Large-Scale Person-Centered Cloze Dataset}}, author = {Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel and David McAllester}, booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing}, year = {2016}, doi = {10.18653/v1/d16-1241}, eprint = {1608.05457}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/d16-1241} } ``` - [← previous Results of the WMT16 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/) - [next → Character-based Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/jussa-2016-character-based-neural-machine-translation/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) ## References 3 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. [Bollacker et al.2008] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250. [Chen e… [find](https://scholar.google.com/scholar?q=%5BBollacker+et+al%2E2008%5D+Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge%2C+and+Jamie+Taylor%2E+2008%2E+Freebase%3A+a+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+Proceed) 2. Result marked I is from (Hermann et al., 2015), results marked II are from (Chen et al., 2016), result marked III is from (Kadlec et al., 2016), and result marked IV is from (Dhingra et al., 2016). the CNN/Daily Mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Associ… [find](https://scholar.google.com/scholar?q=Result+marked+I+is+from+%28Hermann+et+al%2E%2C+2015%29%2C+results+marked+II+are+from+%28Chen+et+al%2E%2C+2016%29%2C+result+marked+III+is+from+%28Kadlec+et+al%2E%2C+2016%29%2C+and+result+marked+IV+is+from+%28Dhingra+et+al%2E%2C+2016%29%2E+th) 3. Burges, and Erin Renshaw. 2013. [MCTest: A challenge dataset for the open-domain machine comprehension of text](https://md.research.elamur.ai/free/en/papers/richardson-2013-mctest-challenge-dataset-open-domain/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203. [Singhal2012] Amit Singhal. 2012. Introducing the knowledge graph: things, not stri… *Updated 24 October 2026*