---
title: "Common Sense Beyond English — paper by Lin et al. (2021)"
description: "On top of it they define Mickey Probe, a zero-shot test in which a model ranks several similar… 30 references, cited by 65 papers here."
authors: ["Bill Yuchen Lin", "Seyeon Lee", "Xiaoyang Qiao", "Xiang Ren"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.102/"
doi: "10.18653/v1/2021.acl-long.102"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lin-2021-common-sense-beyond-english-evaluating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-2021-common-sense-beyond-english-evaluating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning

# Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning

**Authors:** [Bill Yuchen Lin](https://md.research.elamur.ai/free/en/authors/bill-yuchen-lin/), [Seyeon Lee](https://md.research.elamur.ai/free/en/authors/seyeon-lee/), Xiaoyang Qiao and [Xiang Ren](https://md.research.elamur.ai/free/en/authors/xiang-ren/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.102/) · [PDF](https://aclanthology.org/2021.acl-long.102.pdf) · [DOI 10.18653/v1/2021.acl-long.102](https://doi.org/10.18653/v1/2021.acl-long.102) · [OpenAlex](https://openalex.org/W3174234060)

**Citations:** 41 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/lin-2021-common-sense-beyond-english-evaluating/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Benchmarks and probing methods for commonsense reasoning (CSR) have been built almost entirely for English. As a result, it is unclear how well multilingual language models (ML-LMs) handle everyday knowledge in other languages. The authors address this with new resources and a new training method. They collect MickeyCorpus, a parallel set of 561k sentences in 11 languages derived from OMCS. On top of it they define Mickey Probe, a zero-shot test in which a model ranks several similar statements by plausibility using pseudo-likelihood (PLL). Unlike the LAMA probe, this test handles multi-token concepts and allows fair comparison across languages. They also machine-translate CSQA and CODAH into 15 further languages, producing X-CSQA and X-CODAH for cross-lingual transfer evaluation. Tested models such as mBERT and XLM-R show large gaps between languages and score much lower outside English. To narrow this gap, the authors propose multilingual contrastive pre-training (MCP), a sentence-level objective in which the model picks the true assertion among multilingual alternatives. MCP improves accuracy and makes fine-tuning more sample-efficient, which offers a practical route to CSR for lower-resource languages.

## Key points
- MickeyCorpus has 561k sentences in 11 languages. It serves both as data for Mickey Probe and, after cross-lingual resampling, as the pre-training corpus for MCP.
- X-CSQA (question answering) and X-CODAH (scene completion) cover 16 languages, including low-resource Swahili (sw) and Urdu (ur). Before translation, the authors filtered out sentiment-laden and social-behavior examples to reduce cultural bias.
- The evaluated ML-LMs are mBERT, distil-mBERT, XLM-100 and XLM-R. They differ strongly in which languages they handle best, and all perform worse beyond English.
- MCP raises XLM-R L accuracy on X-CSQA by 2.7%. It also gives a 1.2-point average gain on XNLI-10% across 15 languages, and it speeds up convergence.
- Code and data are available at https://inklab.usc.edu/XCSR/. Users are asked to cite the original OMCS, CSQA and CODAH papers as well.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 104 works · 33,876 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (4): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 465 works

## Cite

Cite as: Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao and Xiang Ren. 2021. Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lin2021common,
  title = {{Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning}},
  author = {Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao and Xiang Ren},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.102},
  url = {https://doi.org/10.18653/v1/2021.acl-long.102}
}
```

- [← previous CDLM: Cross-Document Language Modeling](https://md.research.elamur.ai/free/en/papers/caciularu-2021-cdlm-cross-document-language-modeling/)
- [next → NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints](https://md.research.elamur.ai/free/en/papers/lu-2021-neurologic-decoding-un-supervised-neural/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

30 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Acharya, Kartik Talamadupula, and Mark A. Finlayson. 2020. An atlas of cultural commonsense for machine reasoning. ArXiv, abs/2009.05664. — [arXiv:2009.05664](https://arxiv.org/abs/2009.05664) [find](https://scholar.google.com/scholar?q=A%2E+Acharya%2C+Kartik+Talamadupula%2C+and+Mark+A%2E+Finlayson%2E+2020%2E+An+atlas+of+cultural+commonsense+for+machine+reasoning%2E+ArXiv%2C+abs%2F2009%2E05664%2E+%E2%80%94+%5BarXiv%3A2009%2E05664%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2009%2E05664%29)
2. Michael Chen, Mike D’Arcy, Alisa Liu, Jared Fernandez, and Doug Downey. 2019. [CODAH: An adversarially-authored question answering dataset for common sense](https://md.research.elamur.ai/free/en/papers/chen-2019-codah-adversarially-authored-question-answering/). In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pages 63–69, Minneapolis, USA. Association for Computatio…
3. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. [TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics, 8:454– 47…
4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
5. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057–7067. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+Advances+in+Neural+Information+Processing+Systems+32%3A+Annual+Conference+on+Neural+Information+Processing+Systems)
6. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
7. Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58(9):92–103. [find](https://scholar.google.com/scholar?q=Ernest+Davis+and+Gary+Marcus%2E+2015%2E+Commonsense+reasoning+and+commonsense+knowledge+in+artificial+intelligence%2E+Communications+of+the+ACM%2C+58%289%29%3A92%E2%80%93103%2E)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
9. Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. [Scalable multi-hop relational reasoning for knowledge-aware question answering](https://md.research.elamur.ai/free/en/papers/feng-2020-scalable-multi-hop-relational-reasoning/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1295–1309, Online. Associati…
10. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/). Technical report.
11. Bill Yuchen Lin, Frank F. Xu, Kenny Zhu, and Seung-won Hwang. 2018. Mining cross-cultural differences and similarities in social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 709–719, Melbourne, Australia. Association… [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Frank+F%2E+Xu%2C+Kenny+Zhu%2C+and+Seung%2Dwon+Hwang%2E+2018%2E+Mining+cross%2Dcultural+differences+and+similarities+in+social+media%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for)
12. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020. X-FACTR: Multilingual factual knowledge retrieval from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943–5959, Online. Ass… [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Antonios+Anastasopoulos%2C+Jun+Araki%2C+Haibo+Ding%2C+and+Graham+Neubig%2E+2020%2E+X%2DFACTR%3A+Multilingual+factual+knowledge+retrieval+from+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2020+)
13. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. [Marian: Fast neural machine translation in C++](https://md.research.elamur.ai/free/en/papers/junczys-dowmunt-2018-marian-fast-neural-machine-translation/). In Proceedings of ACL 2018, System…
14. Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. [BERT-ATTACK: Adversarial attack against BERT using BERT](https://md.research.elamur.ai/free/en/papers/li-2020-bert-attack-adversarial-attack-against/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202, Online. Association for Computational Linguistics.
15. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Mi… [find](https://scholar.google.com/scholar?q=Yaobo+Liang%2C+Nan+Duan%2C+Yeyun+Gong%2C+Ning+Wu%2C+Fenfei+Guo%2C+Weizhen+Qi%2C+Ming+Gong%2C+Linjun+Shou%2C+Daxin+Jiang%2C+Guihong+Cao%2C+Xiaodong+Fan%2C+Ruofei+Zhang%2C+Rahul+Agrawal%2C+Edward+Cui%2C+Sining+Wei%2C+Taroon+Bharti%2C+)
16. Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-… [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Xinyue+Chen%2C+Jamin+Chen%2C+and+Xiang+Ren%2E+2019%2E+KagNet%3A+Knowledge%2Daware+graph+networks+for+commonsense+reasoning%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+L)
17. Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models](https://md.research.elamur.ai/free/en/papers/lin-2020-birds-have-four-legs-numersense/). In Proceedings of Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemo…
18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
19. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
20. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. [XCOPA: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association…
21. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. [Stanza: A python natural language processing toolkit for many human languages](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101– 108, Online.…
22. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
23. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. [Masked language model scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics.
24. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+ArXiv%2C+abs%2F1910%2E01108%2E)
25. Push Singh, Thomas Lin, Erik T Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. 2002. Open mind common sense: Knowledge acquisition from the general public. In OTM Confederated International Conferences" On the Move to Meaningful Internet Systems", pages 1223–1237. Springer. [find](https://scholar.google.com/scholar?q=Push+Singh%2C+Thomas+Lin%2C+Erik+T+Mueller%2C+Grace+Lim%2C+Travell+Perkins%2C+and+Wan+Li+Zhu%2E+2002%2E+Open+mind+common+sense%3A+Knowledge+acquisition+from+the+general+public%2E+In+OTM+Confederated+International+Confe)
26. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
27. Jörg Tiedemann. 2016. OPUS – parallel corpora for everyone. In Proceedings of the 19th Annual Conference of the European Association for Machine Translation: Projects/Products, Riga, Latvia. Baltic Journal of Modern Computing. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2016%2E+OPUS+%E2%80%93+parallel+corpora+for+everyone%2E+In+Proceedings+of+the+19th+Annual+Conference+of+the+European+Association+for+Machine+Translation%3A+Projects%2FProducts%2C+Riga%2C+Latvia%2E+Baltic+Jo)
28. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
29. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104, Brussels, Belgium. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018%2E+SWAG%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+)
30. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+BERT%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+2)

*Updated 24 October 2026*