--- title: "A Benchmark Arabic Dataset for Commonsense Explanation — AL-Tawalbeh 2020" description: "The authors release a benchmark for Arabic Commonsense Explanation (CSE). 18 references, cited by 3 papers here." authors: ["Saja AL-Tawalbeh", "Mohammad AL-Smadi"] year: 2020 venue: "arXiv:2012.10251" source: "https://arxiv.org/abs/2012.10251" doi: "10.48550/arxiv.2012.10251" arxiv: "2012.10251" licence: "CC0 1.0 (arXiv)" licence_url: "http://creativecommons.org/publicdomain/zero/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/al-tawalbeh-2020-benchmark-arabic-dataset-commonsense-explanation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/al-tawalbeh-2020-benchmark-arabic-dataset-commonsense-explanation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / A Benchmark Arabic Dataset for Commonsense Explanation # A Benchmark Arabic Dataset for Commonsense Explanation **Authors:** Saja AL-Tawalbeh and [Mohammad AL-Smadi](https://md.research.elamur.ai/free/en/authors/mohammad-al-smadi/) · **Year:** 2020 · **Venue:** arXiv:2012.10251 · **Licence:** [CC0 1.0 (arXiv)](http://creativecommons.org/publicdomain/zero/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2012.10251) · [PDF](https://arxiv.org/pdf/2012.10251) · [DOI 10.48550/arxiv.2012.10251](https://doi.org/10.48550/arxiv.2012.10251) · [arXiv 2012.10251](https://arxiv.org/abs/2012.10251) · [OpenAlex](https://openalex.org/W3117813264) **Citations:** 1 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 140,884 papers whose bibliographies we read ## Summary Arabic NLP has had few resources for testing whether machines can reason with everyday knowledge, and none for explaining why a statement is absurd. The authors release a benchmark for Arabic Commonsense Explanation (CSE). It is derived from the English data of the SemEval-2020 Commonsense Validation and Explanation (ComVE) task and keeps only the explanation subtask. Each item pairs a sentence that contradicts ordinary world knowledge with three candidate reasons, and a system must pick the reason that actually accounts for the absurdity. The resource contains 12k items split into train, validation and test sets. It is published on GitHub under CC-BY-SA 4.0 together with an evaluation script. As baselines, the authors train three transfer-learning language models (BERT, USE and ULMFiT) and report accuracy. All three barely beat chance: BERT reaches 34.20% against 32.5% for random guessing, while BERT on the original English data had reached 45.6%. The result shows the task is far from solved for Arabic and gives later work a fixed point of comparison. ## Key points - Each example has four fields: one false sentence plus three explanation options, with a label marking the correct option (A/B/C). - The splits are 10,000 train, 1,000 validation and 1,000 test items, built on the SemEval-2020 Task 4 (ComVE) English data. - Baseline accuracy on test: Random 32.5%, USE 33%, ULMFiT 32.8%, BERT 34.20%. - For comparison, BERT scored 45.6% when run on the English source data that this benchmark was adapted from (Wang et al., 2019). - The dataset and evaluation code are released at github.com/msmadi under CC-BY-SA 4.0, and the work is presented as the first public Arabic dataset of this kind. ## Interesting ideas - Wrong options mention the same entities as the false sentence and can even be true (e.g. yachts are usually white, beds beige), so a model must judge explanatory relevance rather than topical overlap. - The examples (grandfather put in a cot, yacht put in a bed) are absurd because a role filler breaks a size or category restriction of the verb's slots. - A random baseline is reported next to the trained models, which exposes that a 34.2% score is essentially chance-level. - Running the same model family on an English benchmark and its ported version exposes a cross-language gap (45.6% vs 34.2%). - Fixed public splits plus a shipped evaluation script make results directly comparable across later papers. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (world): Build tests where the world model must name the violated constraint (yacht bigger than bed) and reject true but irrelevant facts about the same entities. - **I2** (frames): Check VerbNet selectional restrictions on Theme and Destination roles in placement frames, so the frame layer itself can flag an implausible "who put what where". - **I3** (learn): Always pair evaluations with a random or shuffled-input control, so near-chance gains are not mistaken for real learning. - **I4** (back): Parallel item sets could test whether translation through the meta-world keeps the impossibility that makes a sentence false. - **I5** (answer): Freeze test sets for school-course and fable QA readers and publish one scoring script, so reader versions are compared on identical terms. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · [🐪 Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Institutions - [Jordan University of Science and Technology](https://md.research.elamur.ai/free/en/universities/jordan-university-of-science-and-technology/) — University, Irbid, Jordan · 9 works · 1,785 citations ## Cite Cite as: Saja AL-Tawalbeh and Mohammad AL-Smadi. 2020. A Benchmark Arabic Dataset for Commonsense Explanation. arXiv:2012.10251. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{altawalbeh2020benchmark, title = {{A Benchmark Arabic Dataset for Commonsense Explanation}}, author = {Saja AL-Tawalbeh and Mohammad AL-Smadi}, year = {2020}, doi = {10.48550/arxiv.2012.10251}, eprint = {2012.10251}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2012.10251} } ``` - [← previous Variation in Coreference Strategies across Genres and Production Media](https://md.research.elamur.ai/free/en/papers/aktas-2020-variation-coreference-strategies-across-genres/) - [next → A guide to the dataset explosion in QA, NLI, and commonsense reasoning](https://md.research.elamur.ai/free/en/papers/rogers-2020-guide-dataset-explosion-qa-nli/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) ## References 18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. [↗](https://arxiv.org/abs/1803.11175) [our card](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/) 2. Ernest Davis. 2016. How to write science questions that are easy for people and hard for computers. AI magazine, 37(1):13–22. [find](https://scholar.google.com/scholar?q=Ernest+Davis%2E+2016%2E+How+to+write+science+questions+that+are+easy+for+people+and+hard+for+computers%2E+AI+magazine%2C+37%281%29%3A13%E2%80%9322%2E) 3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/) 4. Catherine Havasi, Robert Speer, Kenneth Arnold, Henry Lieberman, Jason Alonso, and Jesse Moeller. 2010. Open mind common sense: Crowd-sourcing for common sense. In Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Catherine+Havasi%2C+Robert+Speer%2C+Kenneth+Arnold%2C+Henry+Lieberman%2C+Jason+Alonso%2C+and+Jesse+Moeller%2E+2010%2E+Open+mind+common+sense%3A+Crowd%2Dsourcing+for+common+sense%2E+In+Workshops+at+the+Twenty%2DFourth+AAAI+) 5. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. [↗](https://arxiv.org/abs/1801.06146) 6. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. arXiv preprint arXiv:1909.00277. [↗](https://arxiv.org/abs/1909.00277) [our card](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/) 7. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E) 8. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018a. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. arXiv preprint arXiv:1803.05223. [↗](https://arxiv.org/abs/1803.05223) 9. Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018b. Semeval-2018 task 11: Machine comprehension using commonsense knowledge. In Proceedings of the 12th International Workshop on semantic evaluation, pages 747–757. [find](https://scholar.google.com/scholar?q=Simon+Ostermann%2C+Michael+Roth%2C+Ashutosh+Modi%2C+Stefan+Thater%2C+and+Manfred+Pinkal%2E+2018b%2E+Semeval%2D2018+task+11%3A+Machine+comprehension+using+commonsense+knowledge%2E+In+Proceedings+of+the+12th+Internationa) 10. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+2011+AAAI+Spring+Symposium+Series%2E) 11. Sirwe Saeedi, Aliakbar Panahi, Seyran Saeedi, and Alvis C Fong. 2020. Cs-nlp team at semeval-2020 task 4: Evaluation of state-of-the-artnlp deep learning architectures on commonsense reasoning task. arXiv preprint arXiv:2006.01205. [↗](https://arxiv.org/abs/2006.01205) 12. Chenglei Si, Shuohang Wang, Min-Yen Kan, and Jing Jiang. 2019. What does bert learn from multiple-choice reading comprehension datasets? arXiv preprint arXiv:1910.12391. [↗](https://arxiv.org/abs/1910.12391) 13. Saja Tawalbeh and Mohammad AL-Smadi. 2020. Is this sentence valid? an arabic dataset for commonsense validation. arXiv preprint arXiv:2008.10873. [↗](https://arxiv.org/abs/2008.10873) [our card](https://md.research.elamur.ai/free/en/papers/tawalbeh-2020-this-sentence-valid-arabic-dataset/) 14. Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. Does it make sense? and why? a pilot study for sense making and explanation. arXiv preprint arXiv:1906.00363. [↗](https://arxiv.org/abs/1906.00363) 15. Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu, and Yue Zhang. 2020. Semeval-2020 task 4: Commonsense validation and explanation. arXiv preprint arXiv:2007.00236. [↗](https://arxiv.org/abs/2007.00236) 16. K. Xu, Jing Tin, and J. Youn. 2019. A bert based model for multiple-choice reading comprehension. [find](https://scholar.google.com/scholar?q=K%2E+Xu%2C+Jing+Tin%2C+and+J%2E+Youn%2E+2019%2E+A+bert+based+model+for+multiple%2Dchoice+reading+comprehension%2E) 17. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326. [↗](https://arxiv.org/abs/1808.05326) 18. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2017-ordinal-common-sense-inference/)