---
title: "CommonsenseQA — Talmor 2019"
description: "The authors aim to determine whether systems can answer questions that require general knowledge about the world rather than finding the answer in a…"
authors: ["Alon Talmor", "Jonathan Herzig", "Nicholas Lourie", "Jonathan Berant"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1421/"
doi: "10.18653/v1/n19-1421"
arxiv: "1811.00937"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

# CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

**Authors:** [Alon Talmor](https://md.research.elamur.ai/free/en/authors/alon-talmor/), [Jonathan Herzig](https://md.research.elamur.ai/free/en/authors/jonathan-herzig/), Nicholas Lourie, [Jonathan Berant](https://md.research.elamur.ai/free/en/authors/jonathan-berant/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1421/) · [PDF](https://aclanthology.org/N19-1421.pdf) · [DOI 10.18653/v1/n19-1421](https://doi.org/10.18653/v1/n19-1421) · [arXiv 1811.00937](https://arxiv.org/abs/1811.00937)

**Citations:** [1146](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The authors aim to determine whether systems can answer questions that require general knowledge about the world rather than finding the answer in a given text. To this end, they created CommonsenseQA, a dataset of 12,247 five-way multiple-choice questions built on ConceptNet. One source concept and three target concepts, all linked to it by the same relation (e.g., AtLocation), are extracted from the graph. Crowdworkers write one question per target concept such that only that concept is correct, while the other two serve as distractors. They then add two more distractors: one selected from ConceptNet and one they write themselves. Because of this design, the questions require background knowledge that people rarely state explicitly in text. The authors evaluated many baseline models. The best was fine-tuned BERT-large, with an accuracy of 55.9%, whereas humans reach 88.9%. The dataset thus shows that current language understanding models lack common sense, and it provides a large-scale benchmark for future research.

## Key points
- Out of roughly 32M ConceptNet triplets, 236,208 were selected. To do this, the authors kept 22 relations (removing general ones such as RelatedTo and IsA) and discarded non-English concepts and concepts longer than four words.
- Each question has 5 answer choices: the correct answer, the two other target concepts from the same group, one choice from ConceptNet, and one written by a crowdworker.
- BERT-large achieves 55.9%, while humans achieve 88.9%. Google web snippets used as context for BiDAF++ barely help. On a random split, results are on average 5 points lower than on the split by source concept.
- An analysis of 100 examples showed the following BERT-large accuracy: 77.7% when surface cues are present, 42.8% on negation and antonyms, 38.4% on factoid knowledge, 35.4% when the answer is more specific than a distractor, and only 23.8% when the answer must satisfy several conditions simultaneously.
- Extrapolating the learning curve suggests that even with 100K examples, accuracy would be only about 75%. The data and code are publicly available: tau-nlp.org/commonsenseqa and GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · [Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{talmor2019commonsenseqa,
  title = {{CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge}},
  author = {Talmor, Alon and Herzig, Jonathan and Lourie, Nicholas and Berant, Jonathan},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1421},
  eprint = {1811.00937},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1421}
}
```

- [← previous The Risk of Racial Bias in Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/)
- [next → Predicting the Type and Target of Offensive Posts in Social Media](https://md.research.elamur.ai/free/en/papers/zampieri-2019-predicting-type-target-offensive-posts/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In ACL. [find](https://scholar.google.com/scholar?q=Zheng+Cai%2C+Lifu+Tu%2C+and+Kevin+Gimpel%2E+2017%2E+Pay+attention+to+the+ending%3A+Strong+neural+baselines+for+the+roc+story+cloze+task%2E+In+ACL%2E)
2. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. 2013. One billion word Douglas B. Lenat. 1995. Cyc: A large-scale investment in knowledge infrastructure. Commun. ACM, 38:32–38. [find](https://scholar.google.com/scholar?q=C%2E+Chelba%2C+T%2E+Mikolov%2C+M%2E+Schuster%2C+Q%2E+Ge%2C+T%2E+Brants%2C+P%2E+Koehn%2C+and+T%2E+Robinson%2E+2013%2E+One+billion+word+Douglas+B%2E+Lenat%2E+1995%2E+Cyc%3A+A+large%2Dscale+investment+in+knowledge+infrastructure%2E+Commun%2E+ACM%2C+)
3. Hector J. Levesque. 2011. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2E+2011%2E+The+winograd+schema+challenge%2E+In+AAAI+Spring+Symposium%3A+Logical+Formalizations+of+Commonsense+Reasoning%2E)
4. Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. 2016. Learning natural language inference using bidirectional lstm model and inner-attention. arXiv preprint arXiv:1605.09090. — [arXiv:1605.09090](https://arxiv.org/abs/1605.09090) [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Chengjie+Sun%2C+Lei+Lin%2C+and+Xiaolong+Wang%2E+2016%2E+Learning+natural+language+inference+using+bidirectional+lstm+model+and+inner%2Dattention%2E+arXiv+preprint+arXiv%3A1605%2E09090%2E+%E2%80%94+%5BarXiv%3A1605%2E09090%5D%28)
5. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=P%2E+Rajpurkar%2C+J%2E+Zhang%2C+K%2E+Lopyrev%2C+and+P%2E+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
6. Peter LoBue and Alexander Yates. 2011. Types of common-sense knowledge needed for recognizing textual entailment. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 329–334. Association for Computation… [find](https://scholar.google.com/scholar?q=Peter+LoBue+and+Alexander+Yates%2E+2011%2E+Types+of+common%2Dsense+knowledge+needed+for+recognizing+textual+entailment%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguis) [our card](https://md.research.elamur.ai/free/en/papers/lobue-2011-types-common-sense-knowledge-needed/)
7. M. Roemmele, C. Bejan, and A. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning. [find](https://scholar.google.com/scholar?q=M%2E+Roemmele%2C+C%2E+Bejan%2C+and+A%2E+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+AAAI+Spring+Symposium+on+Logical+Formalizations+of+Commonsense+Reasoning)
8. J. McCarthy. 1959. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes. [find](https://scholar.google.com/scholar?q=J%2E+McCarthy%2E+1959%2E+Programs+with+common+sense%2E+In+Proceedings+of+the+Teddington+Conference+on+the+Mechanization+of+Thought+Processes%2E)
9. John McCarthy and Patrick J. Hayes. 1969. Some philosophical problems from the standpoint of artificial intelligence. In B. Meltzer and D. Michie, editors, Machine Intelligence 4, pages 463–502. Edinburgh University Press. Reprinted in McC90. [find](https://scholar.google.com/scholar?q=John+McCarthy+and+Patrick+J%2E+Hayes%2E+1969%2E+Some+philosophical+problems+from+the+standpoint+of+artificial+intelligence%2E+In+B%2E+Meltzer+and+D%2E+Michie%2C+editors%2C+Machine+Intelligence+4%2C+pages+463%E2%80%93502%2E+Edinb)
10. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E)
11. N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=N%2E+Mostafazadeh%2C+N%2E+Chambers%2C+X%2E+He%2C+D%2E+Parikh%2C+D%2E+Batra%2C+L%2E+Vanderwende%2C+P%2E+Kohli%2C+and+J%2E+Allen%2E+2016%2E+A+corpus+and+cloze+evaluation+for+deeper+understanding+of+commonsense+stories%2E+In+North+American)
12. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Workshop on Cognitive Computing at NIPS. [find](https://scholar.google.com/scholar?q=T%2E+Nguyen%2C+M%2E+Rosenberg%2C+X%2E+Song%2C+J%2E+Gao%2C+S%2E+Tiwary%2C+R%2E+Majumder%2C+and+L%2E+Deng%2E+2016%2E+MS+MARCO%3A+A+human+generated+machine+reading+comprehension+dataset%2E+In+Workshop+on+Cognitive+Computing+at+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)
13. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. CoRR, abs/1803.05223. [find](https://scholar.google.com/scholar?q=Simon+Ostermann%2C+Ashutosh+Modi%2C+Michael+Roth%2C+Stefan+Thater%2C+and+Manfred+Pinkal%2E+2018%2E+Mcscript%3A+A+novel+dataset+for+assessing+machine+comprehension+using+script+knowledge%2E+CoRR%2C+abs%2F1803%2E05223%2E)
14. J. Pennington, R. Socher, and C. D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=J%2E+Pennington%2C+R%2E+Socher%2C+and+C%2E+D%2E+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/)
15. Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the roc story cloze task. In CoNLL. [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Maarten+Sap%2C+Ioannis+Konstas%2C+Leila+Zilles%2C+Yejin+Choi%2C+and+Noah+A%2E+Smith%2E+2017%2E+The+effect+of+different+writing+tasks+on+linguistic+style%3A+A+case+study+of+the+roc+story+cloze+task%2E+In+C)
16. M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv. [find](https://scholar.google.com/scholar?q=M%2E+Seo%2C+A%2E+Kembhavi%2C+A%2E+Farhadi%2C+and+H%2E+Hajishirzi%2E+2016%2E+Bidirectional+attention+flow+for+machine+comprehension%2E+arXiv%2E) [our card](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)
17. Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, pages 4444–4451. [find](https://scholar.google.com/scholar?q=Robert+Speer%2C+Joshua+Chin%2C+and+Catherine+Havasi%2E+2017%2E+Conceptnet+5%2E5%3A+An+open+multilingual+graph+of+general+knowledge%2E+In+AAAI%2C+pages+4444%E2%80%934451%2E)
18. O. Tange. 2011. Gnu parallel - the command-line power tool. ;login: The USENIX Magazine, 36(1):42–47. [find](https://scholar.google.com/scholar?q=O%2E+Tange%2E+2011%2E+Gnu+parallel+%2D+the+command%2Dline+power+tool%2E+%3Blogin%3A+The+USENIX+Magazine%2C+36%281%29%3A42%E2%80%9347%2E)
19. Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. [find](https://scholar.google.com/scholar?q=Trieu+H+Trinh+and+Quoc+V+Le%2E+2018%2E+A+simple+method+for+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1806%2E02847%2E)
20. T. Winograd. 1972. Understanding Natural Language. Academic Press. [find](https://scholar.google.com/scholar?q=T%2E+Winograd%2E+1972%2E+Understanding+Natural+Language%2E+Academic+Press%2E)
21. Lei Yu, Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. 2014. Deep learning for answer sentence selection. arXiv preprint arXiv:1412.1632. — [arXiv:1412.1632](https://arxiv.org/abs/1412.1632) [find](https://scholar.google.com/scholar?q=Lei+Yu%2C+Karl+Moritz+Hermann%2C+Phil+Blunsom%2C+and+Stephen+Pulman%2E+2014%2E+Deep+learning+for+answer+sentence+selection%2E+arXiv+preprint+arXiv%3A1412%2E1632%2E+%E2%80%94+%5BarXiv%3A1412%2E1632%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1412%2E1632%29)
22. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2018a. From recognition to cognition: Visual commonsense reasoning. arXiv preprint arXiv:1811.10830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2018a%2E+From+recognition+to+cognition%3A+Visual+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1811%2E10830%2E)
23. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018b. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018b%2E+Swag%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+arXiv+preprint+arXiv%3A1808%2E05326%2E)
24. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. TACL, 5:379–395. [find](https://scholar.google.com/scholar?q=Sheng+Zhang%2C+Rachel+Rudinger%2C+Kevin+Duh%2C+and+Benjamin+Van+Durme%2E+2017%2E+Ordinal+common%2Dsense+inference%2E+TACL%2C+5%3A379%E2%80%93395%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2017-ordinal-common-sense-inference/)
25. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proc%2E+of+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
26. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proc. of *SEM. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Jason+Naradowsky%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+and+Benjamin+Van+Durme%2E+2018%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+In+Proc%2E+of+%2ASEM%2E)
27. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018. Improving language understanding by generative pre-training. Technical Report, OpenAI. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+K%2E+Narasimhan%2C+T%2E+Salimans%2C+and+I%2E+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+Technical+Report%2C+OpenAI%2E)