---
title: "IndoCulture — paper by Koto et al. (2024) — summary and ideas"
description: "The resource offers a regionally fine-grained way to probe cultural bias in multilingual… 50% accuracy. 55 references, cited by 42 papers here."
authors: ["Fajri Koto", "Rahmad Mahendra", "Nurul Aisyah", "Timothy Baldwin"]
year: 2024
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2024.tacl-1.92/"
doi: "10.1162/tacl_a_00726"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/koto-2024-indoculture-exploring-geographically-influenced-cultural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/koto-2024-indoculture-exploring-geographically-influenced-cultural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces

# IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces

**Authors:** [Fajri Koto](https://md.research.elamur.ai/free/en/authors/fajri-koto/), [Rahmad Mahendra](https://md.research.elamur.ai/free/en/authors/rahmad-mahendra/), Nurul Aisyah and [Timothy Baldwin](https://md.research.elamur.ai/free/en/authors/timothy-baldwin/) · **Year:** 2024 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2024](https://md.research.elamur.ai/free/en/venues/tacl/2024/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.tacl-1.92/) · [PDF](https://aclanthology.org/2024.tacl-1.92.pdf) · [DOI 10.1162/tacl\_a\_00726](https://doi.org/10.1162/tacl_a_00726) · [OpenAlex](https://openalex.org/W4405738703)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [42](https://md.research.elamur.ai/free/en/papers/koto-2024-indoculture-exploring-geographically-influenced-cultural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most commonsense reasoning benchmarks are built in English and ignore where the reasoner lives, so they tend to reward Anglocentric knowledge. IndoCulture addresses this gap for Indonesia, a country with very high ethnic and linguistic diversity. The authors recruited local residents in eleven provinces, from Aceh to Papua, to write Indonesian sentence-completion items by hand. Each item has a one-sentence premise about local custom and three plausible continuations, only one of which is correct. Items follow twelve predefined topics, such as food, weddings, art, pregnancy and family ties, and passed a two-stage verification. Because the data was neither templated nor scraped from the web, it is less likely to overlap with LLM training sets. The authors tested 27 language models. Only Llama–3 comes close to GPT–4, while most other open-weight models stay under 50% accuracy, even though local people answer every item correctly. Results vary by province and are stronger for Bali and West Java. Adding the location to the prompt helps, especially for large models. The resource offers a regionally fine-grained way to probe cultural bias in multilingual models.

## Key points
- The dataset has 2,429 three-option items covering 11 provinces and 12 topics. It was written by 22 local annotators and is released on Hugging Face (indolem/IndoCulture) under CC BY-NC-SA 4.0.
- The evaluation covered 19 open-weight multilingual models, 6 Indonesian-centric models and 2 closed models (GPT–3.5 and GPT–4). Llama–3 Instruct (70B) is roughly on par with GPT–4.
- In a manual check of 200 generated answers, Merak (7B) fell from 53.2% (probability-based scoring) to 29.5%. Only 4.5% of its answers came with a valid explanation. Llama–3 and GPT–4 reached 70.5% and 69.5%, yet 29% and 34% of their respective correct answers were accompanied by flawed explanations.
- Machine translation into English lowered most models' scores, partly because local terms were mistranslated. On 100 samples, GPT–4 scored 77.0 on the Indonesian original, 68.0 on machine-translated English and 72.0 on human-corrected English.
- Including province- or country-level location in the prompt improves accuracy, with the largest gains for GPT–3.5 and GPT–4.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) — University, Melbourne, Australia · 102 works · 22,618 citations
- [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 138 works · 6,590 citations
- [RMIT University](https://md.research.elamur.ai/free/en/universities/rmit-university/) — University, Melbourne, Australia · 19 works · 704 citations
- [University of Indonesia](https://md.research.elamur.ai/free/en/universities/university-of-indonesia/) — University, Depok, Indonesia · 8 works · 589 citations

Also on the byline: [Quantic School of Business and Technology](https://ror.org/03dnnmg11).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Fajri Koto, Rahmad Mahendra, Nurul Aisyah and Timothy Baldwin. 2024. IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{koto2024indoculture,
  title = {{IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces}},
  author = {Fajri Koto, Rahmad Mahendra, Nurul Aisyah and Timothy Baldwin},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00726},
  url = {https://doi.org/10.1162/tacl_a_00726}
}
```

- [← previous Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?](https://md.research.elamur.ai/free/en/papers/lee-2024-long-context-language-models-subsume/)
- [next → Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning](https://md.research.elamur.ai/free/en/papers/wang-2024-q-improving-multi-step-reasoning/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

55 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alham Fikri Aji, Genta Indra Winata, Fajri Koto, Samuel Cahyawijaya, Ade Romadhony, Rahmad Mahendra, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Timothy Baldwin, Jey Han Lau, and Sebastian Ruder. 2022. One country, 700+ languages: NLP challenges for underrepresented languages and dialects… [find](https://scholar.google.com/scholar?q=Alham+Fikri+Aji%2C+Genta+Indra+Winata%2C+Fajri+Koto%2C+Samuel+Cahyawijaya%2C+Ade+Romadhony%2C+Rahmad+Mahendra%2C+Kemal+Kurniawan%2C+David+Moeljadi%2C+Radityo+Eko+Prasojo%2C+Timothy+Baldwin%2C+Jey+Han+Lau%2C+and+Sebastian+R)
2. Roger E. Axtell and Mike Fornwald. 1998. Gestures: The do’s and Taboos of Body Language Around the World. Wiley. [find](https://scholar.google.com/scholar?q=Roger+E%2E+Axtell+and+Mike+Fornwald%2E+1998%2E+Gestures%3A+The+do%E2%80%99s+and+Taboos+of+Body+Language+Around+the+World%2E+Wiley%2E)
3. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. [PIQA: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference,…
4. Educational Advances in Artificial Intelligence, EAAI 2020, pages 7432–7439. AAAI Press. https://doi.org/10.1609/aaai .v34i05.6239 — [doi:10.1609/aaai](https://doi.org/10.1609/aaai) [↗](https://doi.org/10.1609/aaai)
5. Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Khodra, Ayu Purwarianti, and Pascale Fung. 2021. IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation. In Proce… [find](https://scholar.google.com/scholar?q=Samuel+Cahyawijaya%2C+Genta+Indra+Winata%2C+Bryan+Wilie%2C+Karissa+Vincentio%2C+Xiaohong+Li%2C+Adhiguna+Kuncoro%2C+Sebastian+Ruder%2C+Zhi+Yuan+Lim%2C+Syafri+Bahar%2C+Masayu+Khodra%2C+Ayu+Purwarianti%2C+and+Pascale+Fung%2E+20)
6. Joe Davison, Joshua Feldman, and Alexander Rush. 2019. [Commonsense knowledge mining from pretrained models](https://md.research.elamur.ai/free/en/papers/davison-2019-commonsense-knowledge-mining-pretrained-models/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–1…
7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
8. Yi Fung, Tuhin Chakrabarty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. 2023. NORMSAGE: Multi-lingual multi-cultural norm discovery from conversations on-the-fly. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15217–15230, Singapore. Associa… [find](https://scholar.google.com/scholar?q=Yi+Fung%2C+Tuhin+Chakrabarty%2C+Hao+Guo%2C+Owen+Rambow%2C+Smaranda+Muresan%2C+and+Heng+Ji%2E+2023%2E+NORMSAGE%3A+Multi%2Dlingual+multi%2Dcultural+norm+discovery+from+conversations+on%2Dthe%2Dfly%2E+In+Proceedings+of+the+2023+C)
9. Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. [Massively multi-cultural knowledge acquisition & lm benchmarking](https://md.research.elamur.ai/free/en/papers/fung-2024-massively-multi-cultural-knowledge-acquisition/). arXiv preprint arXiv:2402.09369.
10. Anthony Giddens and Philip W. Sutton. 2021. Essential Concepts in Sociology. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Anthony+Giddens+and+Philip+W%2E+Sutton%2E+2021%2E+Essential+Concepts+in+Sociology%2E+John+Wiley+%26+Sons%2E)
11. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. [Challenges and strategies in cross-cultural N…](https://md.research.elamur.ai/free/en/papers/hershcovich-2022-challenges-strategies-cross-cultural-nlp/)
12. Jie Huang and Kevin Chen-Chuan Chang. 2023. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, Toronto, Canada. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-acl.67 [↗](https://doi.org/10.18653/v1/2023.findings-acl.67)
13. Muhammad Ichsan. 2023. Merak-7b: The LLM for Bahasa Indonesia. Hugging Face Repository. [find](https://scholar.google.com/scholar?q=Muhammad+Ichsan%2E+2023%2E+Merak%2D7b%3A+The+LLM+for+Bahasa+Indonesia%2E+Hugging+Face+Repository%2E)
14. Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). Singapore. Association for Computati… [find](https://scholar.google.com/scholar?q=Fajri+Koto%2C+Nurul+Aisyah%2C+Haonan+Li%2C+and+Timothy+Baldwin%2E+2023%2E+Large+language+models+only+pass+primary+school+exams+in+Indonesia%3A+A+comprehensive+test+on+IndoMMLU%2E+In+Proceedings+of+the+2023+Conferen)
15. Fajri Koto, Timothy Baldwin, and Jey Han Lau. 2022. Cloze evaluation for deeper understanding of commonsense stories in Indonesian. In Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022), pages 8–16, Dublin, Ireland. Association for Computational Linguistics. ht… [find](https://scholar.google.com/scholar?q=Fajri+Koto%2C+Timothy+Baldwin%2C+and+Jey+Han+Lau%2E+2022%2E+Cloze+evaluation+for+deeper+understanding+of+commonsense+stories+in+Indonesian%2E+In+Proceedings+of+the+First+Workshop+on+Commonsense+Representation+a)
16. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. [ArabicMMLU: Assessing massive multitask language understanding in Arabic](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/). In Findings of th…
17. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024 .findings-acl.334 sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2016–2039, Mexico Ci… [↗](https://doi.org/10.18653/v1/2024)
18. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E)
19. Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. [Bactrian-X: A multilingual replicable instruction-following model with low-rank adaptation](https://md.research.elamur.ai/free/en/papers/li-2023-bactrian-x-multilingual-replicable-instruction/). arXiv preprint arXiv:2305.15011.
20. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. [CMMLU: Measuring massive multitask language understanding in Chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/). In Findings of the Association for Computational Linguistics ACL 2024, pages 11260–11285, Bangkok, Thailand and virtual me…
21. Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. [Birds have four legs?! NumerSense: Probing numerical commonsense knowledge of pre-trained language models](https://md.research.elamur.ai/free/en/papers/lin-2020-birds-have-four-legs-numersense/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862–6868, Online. Assoc…
22. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… — [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
23. Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual LLMs culturally-diverse reasoners? An investigation into multicultural proverbs and Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, R… [find](https://scholar.google.com/scholar?q=Chen+Liu%2C+Fajri+Koto%2C+Timothy+Baldwin%2C+and+Iryna+Gurevych%2E+2024%2E+Are+multilingual+LLMs+culturally%2Ddiverse+reasoners%3F+An+investigation+into+multicultural+proverbs+and+Zhengzhong+Liu%2C+Aurick+Qiao%2C+Willi)
24. John J. Macionis. 2012. Sociology: Fourteenth Edition. Pearson. [find](https://scholar.google.com/scholar?q=John+J%2E+Macionis%2E+2012%2E+Sociology%3A+Fourteenth+Edition%2E+Pearson%2E)
25. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. [Language models of code are few-shot commonsense learners](https://md.research.elamur.ai/free/en/papers/madaan-2022-language-models-code-few-shot/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1384–1403, Abu Dhabi, United Arab Emirates. Association for Computatio…
26. Rahmad Mahendra, Alham Fikri Aji, Samuel Louvan, Fahrurrozi Rahman, and Clara Vania. 2021. IndoNLI: A natural language inference dataset for Indonesian. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10511–10527, Online and Punta Cana, Dominican Repu… [find](https://scholar.google.com/scholar?q=Rahmad+Mahendra%2C+Alham+Fikri+Aji%2C+Samuel+Louvan%2C+Fahrurrozi+Rahman%2C+and+Clara+Vania%2E+2021%2E+IndoNLI%3A+A+natural+language+inference+dataset+for+Indonesian%2E+In+Proceedings+of+the+2021+Conference+on+Empiri)
27. Alex Mesoudi. 2016. Cultural evolution: Integrating psychology, evolution and culture. Current Opinion in Psychology, 7:17–22. https://doi.org/10.1016/j .copsyc.2015.07.001 — [doi:10.1016/j](https://doi.org/10.1016/j) [find](https://scholar.google.com/scholar?q=Alex+Mesoudi%2E+2016%2E+Cultural+evolution%3A+Integrating+psychology%2C+evolution+and+culture%2E+Current+Opinion+in+Psychology%2C+7%3A17%E2%80%9322%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1016%2Fj+%2Ecopsyc%2E2015%2E07%2E001+%E2%80%94+%5Bdoi%3A10%2E1016%2Fj%5D%28https%3A%2F%2Fd)
28. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+Allen%2E+2016%2E+A+corpus+and+cloze+evaluation+for+deeper+understanding+of+commo)
29. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… — [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
30. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Vı́ctor Gutiérrez-Basulto, Yazmı́n Ibáñez-Garcı́a, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid… — [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://md.research.elamur.ai/free/en/papers/myung-2024-blend-benchmark-llms-everyday-knowledge/)
31. Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023. [Extracting cultural commonsense knowledge at scale](https://md.research.elamur.ai/free/en/papers/nguyen-2022-extracting-cultural-commonsense-knowledge-scale/). In Proceedings of the ACM Web Conference 2023, pages 1907–1917. https://doi.org/10 .1145/3543507.3583535
32. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024. [SeaLLMs -large language models for Southeast Asia](https://md.research.elamur.ai/free/en/papers/nguyen-2023-seallms-large-language-models-southeast/). In Proceedings…
33. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv, abs/2303.08774.
34. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Tr… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
35. Louis Owen, Vishesh Tripathi, Abhay Kumar, and Biddwan Ahmed. 2024. Komodo: A linguistic expedition into Indonesia’s regional languages. arXiv preprint arXiv:2403.09362. — [arXiv:2403.09362](https://arxiv.org/abs/2403.09362) [find](https://scholar.google.com/scholar?q=Louis+Owen%2C+Vishesh+Tripathi%2C+Abhay+Kumar%2C+and+Biddwan+Ahmed%2E+2024%2E+Komodo%3A+A+linguistic+expedition+into+Indonesia%E2%80%99s+regional+languages%2E+arXiv+preprint+arXiv%3A2403%2E09362%2E+%E2%80%94+%5BarXiv%3A2403%2E09362%5D%28https%3A%2F%2Fa)
36. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. [XCOPA: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Associatio…
37. Hadi Syah Putra, Rahmad Mahendra, and Fariz Darari. 2019. Budayakb: Extraction of cultural heritage entities from heterogeneous formats. In Proceedings of the 9th International Conference on Web Intelligence, Mining and Semantics, WIMS 2019, pages 6:1–6:9. ACM. [https://doi.org](https://doi.org) /10.1145/3326467.33264… — [doi:10.1145/3326467.3326487](https://doi.org/10.1145/3326467.3326487) [↗](https://doi.org)
38. Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, and Alice Oh. 2024. [Can LLM generate culturally relevant commonsense QA data? Case study in Indonesian and Sundanese](https://md.research.elamur.ai/free/en/papers/putri-2024-can-llm-generate-culturally-relevant/). arXiv e-prints, arXiv–2402.
39. Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021. TIMEDIAL: Temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… [find](https://scholar.google.com/scholar?q=Lianhui+Qin%2C+Aditya+Gupta%2C+Shyam+Upadhyay%2C+Luheng+He%2C+Yejin+Choi%2C+and+Manaal+Faruqui%2E+2021%2E+TIMEDIAL%3A+Temporal+commonsense+reasoning+in+dialog%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Associa)
40. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S%2E+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+2011+AAAI+Spring+Symposium+Series%2E)
41. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. https://doi.org/10.1145/3474381 [↗](https://doi.org/10.1145/3474381)
42. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Hannah+Rashkin%2C+Derek+Chen%2C+Ronan+Le+Bras%2C+and+Yejin+Choi%2E+2019%2E+Social+IQa%3A+Commonsense+reasoning+about+social+interactions%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in)
43. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alha… — [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)
44. Vered Shwartz. 2022. Good night at 4 pm?! Time expressions in different cultures. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2842–2853, Dublin, Ireland. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2022.findings-acl.224 [find](https://scholar.google.com/scholar?q=Vered+Shwartz%2E+2022%2E+Good+night+at+4+pm%3F%21+Time+expressions+in+different+cultures%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2022%2C+pages+2842%E2%80%932853%2C+Dublin%2C+Ireland%2E+Association+)
45. AI Singapore. 2023. SEA-LION (Southeast Asian languages in one network): A family of large language models for Southeast Asia. [https://github.com/aisingapore/sealion](https://github.com/aisingapore/sealion) Jenny Thomas. 1983. Cross-cultural pragmatic failure. Applied Linguistics, 4(2):91–112. https://doi.org/10.1093/applin/4.2.91 — [doi:10.1093/applin](https://doi.org/10.1093/applin) [↗](https://github.com/aisingapore/sealion)
46. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
47. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. [‘‘My answer is C’’: First-token probabilities do not match text answers in instruction-tuned language models](https://md.research.elamur.ai/free/en/papers/wang-2024-my-answer-is-c-first/). In Findings of the Association for Computational Linguistics ACL 20…
48. Haryo Wibowo, Erland Fuadi, Made Nityasya, Radityo Eko Prasojo, and Alham Aji. 2024. COPAL-ID: Indonesian language reasoning with local culture and nuances. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi… [find](https://scholar.google.com/scholar?q=Haryo+Wibowo%2C+Erland+Fuadi%2C+Made+Nityasya%2C+Radityo+Eko+Prasojo%2C+and+Alham+Aji%2E+2024%2E+COPAL%2DID%3A+Indonesian+language+reasoning+with+local+culture+and+nuances%2E+In+Proceedings+of+the+2024+Conference+of+th)
49. Raymond Williams. 2014. Keywords: A Vocabulary of Culture and Society. Oxford University Press. [find](https://scholar.google.com/scholar?q=Raymond+Williams%2E+2014%2E+Keywords%3A+A+Vocabulary+of+Culture+and+Society%2E+Oxford+University+Press%2E)
50. Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2020.emnlp[-demos.6](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
51. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. GeoMLAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039–2055, Abu Dhabi, United… [find](https://scholar.google.com/scholar?q=Da+Yin%2C+Hritik+Bansal%2C+Masoud+Monajatipoor%2C+Liunian+Harold+Li%2C+and+Kai%2DWei+Chang%2E+2022%2E+GeoMLAMA%3A+Geo%2Ddiverse+commonsense+probing+on+multilingual+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+202)
52. Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2023. [NusaX: Multilingual parallel sentiment dataset for 10 Indon…](https://md.research.elamur.ai/free/en/papers/winata-2023-nusax-multilingual-parallel-sentiment-dataset/)
53. Habib Zarbaliyev. 2017. Multiculturalism in globalization era: History and challenge for Indonesia. Journal of Social Studies (JSS), 13(1):1–16. https://doi.org/10 .21831/jss.v13i1.16966 [find](https://scholar.google.com/scholar?q=Habib+Zarbaliyev%2E+2017%2E+Multiculturalism+in+globalization+era%3A+History+and+challenge+for+Indonesia%2E+Journal+of+Social+Studies+%28JSS%29%2C+13%281%29%3A1%E2%80%9316%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E21831%2Fjss%2Ev13i1%2E16966)
54. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — [NormBank: A Knowledge Bank of Situational Social Norms](https://md.research.elamur.ai/free/en/papers/ziems-2023-normbank-knowledge-bank-situational-social/)
55. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)

*Updated 2 November 2026*