---
title: "Is this sentence valid? An Arabic Dataset for Commonsense Validation — Tawalbeh 2020"
description: "The authors build one by translating the English data from the Commonsense Validation and Explanation (ComVE) task… 23 references, cited by 3 papers here."
authors: ["Saja Tawalbeh", "Mohammad AL-Smadi"]
year: 2020
venue: "arXiv:2008.10873"
source: "https://arxiv.org/abs/2008.10873"
doi: "10.48550/arxiv.2008.10873"
arxiv: "2008.10873"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tawalbeh-2020-this-sentence-valid-arabic-dataset/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tawalbeh-2020-this-sentence-valid-arabic-dataset/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / Is this sentence valid? An Arabic Dataset for Commonsense Validation

# Is this sentence valid? An Arabic Dataset for Commonsense Validation

**Authors:** Saja Tawalbeh and [Mohammad AL-Smadi](https://md.research.elamur.ai/free/en/authors/mohammad-al-smadi/) · **Year:** 2020 · **Venue:** arXiv:2008.10873 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2008.10873) · [PDF](https://arxiv.org/pdf/2008.10873) · [DOI 10.48550/arxiv.2008.10873](https://doi.org/10.48550/arxiv.2008.10873) · [arXiv 2008.10873](https://arxiv.org/abs/2008.10873) · [OpenAlex](https://openalex.org/W3080594620)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/tawalbeh-2020-this-sentence-valid-arabic-dataset/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Arabic had no public benchmark for checking whether a machine can tell sensible statements from absurd ones. The authors build one by translating the English data from the Commonsense Validation and Explanation (ComVE) task (SemEval-2020 Task 4, after Wang et al., 2019) into Arabic. The English pairs were originally written by Amazon Mechanical Turk workers. Each item holds two nearly identical sentences, one plausible and one that violates everyday knowledge, plus a label naming the invalid one. The resulting corpus has 12,000 pairs and is released on GitHub under CC BY-SA 4.0.

As baselines, they train BERT, USE and ULMFiT to pick the nonsensical sentence and measure accuracy.
- BERT clearly leads at 62.7%.
- The other two models barely exceed or fall below coin-flip level.
- For comparison, BERT scores 70.1% on the English original.

The work matters mainly as a first resource for Arabic commonsense evaluation and as a reference point for future models. The paper says little about how the translation was carried out or checked.

## Key points
- The dataset is derived from the English ComVE / SemEval-2020 Task 4 sense-making data and is presented as the first Arabic commonsense validation set.
- Size and split: 12,000 rows in total, with 10,000 train, 1,000 validation and 1,000 test.
- Each row contains Sent0, Sent1 and a label (0 or 1) pointing to the sentence that does not make sense.
- Baseline accuracy: USE 48.3, ULMFiT 50.5, BERT 62.7. For reference, English BERT reaches 70.1 (Wang et al., 2019).
- The data is licensed CC BY-SA 4.0 and hosted in the msmadi GitHub repository.

## Interesting ideas
- Pairs differ by only a word or two (e.g. "football is square" vs "round"), so lexical overlap gives no clue and only knowledge about the world separates them, which makes this a clean probe of plausibility.
- Asking which of two sentences is invalid, rather than scoring one sentence alone, turns validation into a relative judgment that sidesteps the need for a calibrated absolute threshold.
- Some pairs swap arguments ("I read stars" vs "Stars read me"), so the anomaly lies in who does what, not in the vocabulary used.
- Two of the three baselines land near the 50% chance level, which shows that sentence-embedding similarity alone cannot solve the task and serves as a built-in sanity floor.
- Translating an established benchmark keeps labels and tasks parallel, so scores in the new language can be compared directly with the English original (62.7 vs 70.1).

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (world): such minimal pairs could test whether the compiled knowledge (measures, ontology, typical properties) flags the impossible state in the built world of the text and can explain why.
- **I2** (answer): a reader facing multiple-choice or plausibility questions could compare the consistency of the worlds built from each option rather than judging each option in isolation.
- **I3** (frames): VerbNet selectional restrictions on agent animacy could detect such role reversals and give an explainable reason for rejecting the sentence.
- **I4** (learn): a bag-of-words or embedding baseline on minimal pairs would make a useful negative control, since any Mova component scoring near it is not really using structure.
- **I5** (back): translating pairs through the meta-world and checking that the invalid sentence stays invalid would test whether meaning survives the text → graph → text path.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · [🐪 Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/)

## Institutions

- [Jordan University of Science and Technology](https://md.research.elamur.ai/free/en/universities/jordan-university-of-science-and-technology/) — University, Irbid, Jordan · 9 works · 1,785 citations

## Cite

Cite as: Saja Tawalbeh and Mohammad AL-Smadi. 2020. Is this sentence valid? An Arabic Dataset for Commonsense Validation. arXiv:2008.10873.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tawalbeh2020is,
  title = {{Is this sentence valid? An Arabic Dataset for Commonsense Validation}},
  author = {Saja Tawalbeh and Mohammad AL-Smadi},
  year = {2020},
  doi = {10.48550/arxiv.2008.10873},
  eprint = {2008.10873},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2008.10873}
}
```

- [← previous Event Extraction from Unstructured Amharic Text](https://md.research.elamur.ai/free/en/papers/tadesse-2020-event-extraction-unstructured-amharic-text/)
- [next → Automated Prediction of Medieval Arabic Diacritics](https://md.research.elamur.ai/free/en/papers/alnajjar-2020-automated-prediction-medieval-arabic-diacritics/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nicholas Asher and Laure Vieu. 1995. Toward a geometry of common sense: A semantics and a complete axiomatization of mereotopology. In IJCAI (1), pages 846–852. [find](https://scholar.google.com/scholar?q=Nicholas+Asher+and+Laure+Vieu%2E+1995%2E+Toward+a+geometry+of+common+sense%3A+A+semantics+and+a+complete+axiomatization+of+mereotopology%2E+In+IJCAI+%281%29%2C+pages+846%E2%80%93852%2E)
2. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. [↗](https://arxiv.org/abs/1803.11175) [our card](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder/)
3. Ernest Davis and Leora Morgenstern. 2004. Introduction: Progress in formal commonsense reasoning. Artificial Intelligence, 153(1-2):1–12. [find](https://scholar.google.com/scholar?q=Ernest+Davis+and+Leora+Morgenstern%2E+2004%2E+Introduction%3A+Progress+in+formal+commonsense+reasoning%2E+Artificial+Intelligence%2C+153%281%2D2%29%3A1%E2%80%9312%2E)
4. Ernest Davis. 2016. How to write science questions that are easy for people and hard for computers. AI magazine, 37(1):13–22. [find](https://scholar.google.com/scholar?q=Ernest+Davis%2E+2016%2E+How+to+write+science+questions+that+are+easy+for+people+and+hard+for+computers%2E+AI+magazine%2C+37%281%29%3A13%E2%80%9322%2E)
5. Ernest Davis. 2017. Logical formalizations of commonsense reasoning: a survey. Journal of Artificial Intelligence Research, 59:651–723. [find](https://scholar.google.com/scholar?q=Ernest+Davis%2E+2017%2E+Logical+formalizations+of+commonsense+reasoning%3A+a+survey%2E+Journal+of+Artificial+Intelligence+Research%2C+59%3A651%E2%80%93723%2E)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
7. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. [↗](https://arxiv.org/abs/1801.06146)
8. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. arXiv preprint arXiv:1909.00277. [↗](https://arxiv.org/abs/1909.00277) [our card](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/)
9. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. [↗](https://arxiv.org/abs/1809.02789)
10. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849. [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+Allen%2E+2016%2E+A+corpus+and+cloze+evaluation+for+deeper+understanding+of+commo)
11. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. arXiv preprint arXiv:1803.05223. [↗](https://arxiv.org/abs/1803.05223)
12. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+2011+AAAI+Spring+Symposium+Series%2E)
13. Sirwe Saeedi, Aliakbar Panahi, Seyran Saeedi, and Alvis C Fong. 2020. Cs-nlp team at semeval-2020 task 4: Evaluation of state-of-the-artnlp deep learning architectures on commonsense reasoning task. arXiv preprint arXiv:2006.01205. [↗](https://arxiv.org/abs/2006.01205)
14. Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 752–757. [find](https://scholar.google.com/scholar?q=Rishi+Sharma%2C+James+Allen%2C+Omid+Bakhshandeh%2C+and+Nasrin+Mostafazadeh%2E+2018%2E+Tackling+the+story+ending+biases+in+the+story+cloze+test%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+C)
15. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937. [↗](https://arxiv.org/abs/1811.00937) [our card](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/)
16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
17. Su Wang, Greg Durrett, and Katrin Erk. 2018. Modeling semantic plausibility by injecting world knowledge. arXiv preprint arXiv:1804.00619. [↗](https://arxiv.org/abs/1804.00619)
18. Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. Does it make sense? and why? a pilot study for sense making and explanation. arXiv preprint arXiv:1906.00363. [↗](https://arxiv.org/abs/1906.00363)
19. Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu, and Yue Zhang. 2020a. Semeval-2020 task 4: Commonsense validation and explanation. arXiv preprint arXiv:2007.00236. [↗](https://arxiv.org/abs/2007.00236)
20. Hongru Wang, Xiangru Tang, Sunny Lai, and Kwong Sak Leung. 2020b. Cuhk at semeval-2020 task 4: Commonsense explanation, reasoning and prediction with multi-task learning. arXiv preprint arXiv:2006.09161. [↗](https://arxiv.org/abs/2006.09161)
21. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320–3328. [find](https://scholar.google.com/scholar?q=Jason+Yosinski%2C+Jeff+Clune%2C+Yoshua+Bengio%2C+and+Hod+Lipson%2E+2014%2E+How+transferable+are+features+in+deep+neural+networks%3F+In+Advances+in+neural+information+processing+systems%2C+pages+3320%E2%80%933328%2E)
22. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326. [↗](https://arxiv.org/abs/1808.05326)
23. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2017-ordinal-common-sense-inference/)