---
title: "CulturalBench — paper by Chiu et al. (2025) — summary and ideas"
description: "The authors propose CulturalTeaming, a human-AI red-teaming pipeline. 42 references, cited by 31 papers here."
authors: ["Yu Ying Chiu", "Liwei Jiang", "Bill Yuchen Lin", "Chan Young Park", "Shuyue Stella Li", "Sahithya Ravi", "Mehar Bhatia", "Maria Antoniak", "Yulia Tsvetkov", "Vered Shwartz", "Yejin Choi"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.1247/"
doi: "10.18653/v1/2025.acl-long.1247"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chiu-2025-culturalbench-robust-diverse-challenging-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chiu-2025-culturalbench-robust-diverse-challenging-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs' Cultural Knowledge Through Human-AI Red-Teaming

# CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs' Cultural Knowledge Through Human-AI Red-Teaming

**Authors:** [Yu Ying Chiu](https://md.research.elamur.ai/free/en/authors/yu-ying-chiu/), [Liwei Jiang](https://md.research.elamur.ai/free/en/authors/liwei-jiang/), [Bill Yuchen Lin](https://md.research.elamur.ai/free/en/authors/bill-yuchen-lin/), [Chan Young Park](https://md.research.elamur.ai/free/en/authors/chan-young-park/), [Shuyue Stella Li](https://md.research.elamur.ai/free/en/authors/shuyue-stella-li/), [Sahithya Ravi](https://md.research.elamur.ai/free/en/authors/sahithya-ravi/), Mehar Bhatia, [Maria Antoniak](https://md.research.elamur.ai/free/en/authors/maria-antoniak/), [Yulia Tsvetkov](https://md.research.elamur.ai/free/en/authors/yulia-tsvetkov/), [Vered Shwartz](https://md.research.elamur.ai/free/en/authors/vered-shwartz/) and [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.1247/) · [PDF](https://aclanthology.org/2025.acl-long.1247.pdf) · [DOI 10.18653/v1/2025.acl-long.1247](https://doi.org/10.18653/v1/2025.acl-long.1247) · [OpenAlex](https://openalex.org/W4412889941)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [31](https://md.research.elamur.ai/free/en/papers/chiu-2025-culturalbench-robust-diverse-challenging-benchmark/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Measuring how well language models understand everyday cultural practice worldwide is difficult. Static hand-written test sets soon become too easy. Benchmarks built from web pages, value surveys or synthetic generation may overlap with training data or carry over model biases.

The authors propose CulturalTeaming, a human-AI red-teaming pipeline. People describe customs from their own lives, an AI helper turns each description into a four-option question, and contributors keep revising it until a model answers incorrectly. Five Prolific annotators from the target country then checked every question and were allowed to mark several valid answers. Only items where some option received at least four of five votes were kept.

The result is CulturalBench: 1,696 questions covering 45 regions and 17 topics. It comes in two versions:
- **Easy:** multiple choice.
- **Hard:** each question becomes four true/false judgments.

Strong models nearly match humans on Easy, but Hard remains difficult. The best model reaches 61.4% against 92.4% for humans. Models do especially poorly on questions with several correct answers and on underrepresented regions. This makes the resource a more demanding test of cultural competence.

## Key points
- **Data collection and release:** Over 3,600 questions were collected and filtered to more than 3,000 cultural ones. After majority-vote filtering, 1,696 remain. They span 45 regions, including Bangladesh, Zimbabwe and Peru, and 17 topics such as food, etiquette and celebrations. The data is released under CC BY 4.0.
- **Easy vs. Hard results:**
  - CulturalBench-Hard contains 6,784 binary items, and 29 models were evaluated.
  - On Easy, the top score is 89.6%.
  - On Hard, scores range from 28.7% (Aya-8b) to 61.4% (OpenAI o1; the abstract gives 61.5%).
- **Weakness of the Easy format:** A trivial baseline ignores the question entirely and picks the option most similar in embedding space to the culture's name. It still scores 40% on Easy, against 25% chance. This suggests the multiple-choice format can be gamed.
- **Multiple correct answers:** On questions with more than one correct answer, accuracy falls by 28.7 points. This indicates that models tend to collapse onto a single answer.
- **Regional gaps:** Models perform worst on North Africa, South America, the Middle East and Eastern Europe. GPT-4o beats regional providers even on their home cultures, for example Mistral on European questions and DeepSeek/Qwen on Chinese and East Asian ones.
- **Stated limitations:** The benchmark is English-only, and there are only five verifiers per question.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [University of Copenhagen](https://md.research.elamur.ai/free/en/universities/university-of-copenhagen/) — University, Copenhagen, Denmark · 116 works · 15,292 citations
- [University of British Columbia](https://md.research.elamur.ai/free/en/universities/university-of-british-columbia/) — University, Vancouver, Canada · 104 works · 37,402 citations
- [McGill University](https://md.research.elamur.ai/free/en/universities/mcgill-university/) — University, Montreal, Canada · 91 works · 11,178 citations
- [Vector Institute](https://md.research.elamur.ai/free/en/labs/vector-institute/) — Research institute, Toronto, Canada · 37 works · 5,703 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz and Yejin Choi. 2025. CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs' Cultural Knowledge Through Human-AI Red-Teaming. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chiu2025culturalbench,
  title = {{CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs' Cultural Knowledge Through Human-AI Red-Teaming}},
  author = {Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz and Yejin Choi},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1247},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1247}
}
```

- [← previous ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents](https://md.research.elamur.ai/free/en/papers/wang-2025-vidorag-visual-document-retrieval-augmented/)
- [next → InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery](https://md.research.elamur.ai/free/en/papers/cao-2025-instructmol-multi-modal-integration-building/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

42 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Ashutosh Dwivedi, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. 2024. [Towards measuring and modeling" culture" in llms: A survey](https://md.research.elamur.ai/free/en/papers/adilazuarda-2024-measuring-modeling-culture-llms-survey/). arXiv preprint arXiv:2403.15412.
2. Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augenstein. 2023. [Probing pre-trained language models for cross-cultural differences in values](https://md.research.elamur.ai/free/en/papers/arora-2023-probing-pre-trained-language-models/). In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114–130, Dubrovnik, Croatia. Association for Computational Linguis…
3. Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. 2022. [Why exposure bias matters: An imitation learning perspective of error accumulation in language generation](https://md.research.elamur.ai/free/en/papers/arora-2022-why-exposure-bias-matters-imitation/). In Findings.
4. Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, and Eunsol Choi. 2024. [Calmqa: Exploring culturally specific long-form question answering across 23 languages](https://md.research.elamur.ai/free/en/papers/arora-2025-calmqa-exploring-culturally-specific-long/). Preprint, arXiv:2406.17761.
5. Michael Boratko, Xiang Li, Tim O’Gorman, Rajarshi Das, Dan Le, and Andrew McCallum. 2020. ProtoQA: A question answering dataset for prototypical common-sense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1122–1136, Online. Associa… [find](https://scholar.google.com/scholar?q=Michael+Boratko%2C+Xiang+Li%2C+Tim+O%E2%80%99Gorman%2C+Rajarshi+Das%2C+Dan+Le%2C+and+Andrew+McCallum%2E+2020%2E+ProtoQA%3A+A+question+answering+dataset+for+prototypical+common%2Dsense+reasoning%2E+In+Proceedings+of+the+2020+Conf)
6. Nicholas Joseph, et al. 2023a. [Towards measuring the representation of subjective global opinions in language models](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/). arXiv preprint arXiv:2306.16388.
7. Esin Durmus, Karina Nyugen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2023b. [Towards measuring the…](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/)
8. Ashutosh Dwivedi, Pradhyumna Lavania, and Ashutosh Modi. 2023. Eticor: Corpus for analyzing llms for etiquettes. arXiv preprint arXiv:2310.18974. [find](https://scholar.google.com/scholar?q=Ashutosh+Dwivedi%2C+Pradhyumna+Lavania%2C+and+Ashutosh+Modi%2E+2023%2E+Eticor%3A+Corpus+for+analyzing+llms+for+etiquettes%2E+arXiv+preprint+arXiv%3A2310%2E18974%2E)
9. Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lacalle, and Mikel Artetxe. 2024. Bertaqa: How much do language models know about local culture? Advances in Neural Information Processing Systems, 37:34077–34097. [find](https://scholar.google.com/scholar?q=Julen+Etxaniz%2C+Gorka+Azkune%2C+Aitor+Soroa%2C+Oier+Lacalle%2C+and+Mikel+Artetxe%2E+2024%2E+Bertaqa%3A+How+much+do+language+models+know+about+local+culture%3F+Advances+in+Neural+Information+Processing+Systems%2C+37%3A34)
10. Yi Ren Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. [Massively multi-cultural knowledge acquisition & lm benchmarking](https://md.research.elamur.ai/free/en/papers/fung-2024-massively-multi-cultural-knowledge-acquisition/). ArXiv, abs/2402.09369.
11. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. [Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/). arXiv preprint arXiv:2209.07858.
12. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
13. Shreya Havaldar, Sunny Rai, Bhumika Singhal, Langchen Liu, Sharath Chandra Guntuku, and Lyle Ungar. 2023. [Multilingual language models are not multicultural: A case study in emotion](https://md.research.elamur.ai/free/en/papers/havaldar-2023-multilingual-language-models-are-not/). arXiv preprint arXiv:2307.01370.
14. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. [Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study](https://md.research.elamur.ai/free/en/papers/cao-2023-assessing-cross-cultural-alignment-between/). In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 53–67, Dubrovnik, Croatia. As…
15. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova Dassarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Herna… [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zachary+Dodds%2C+Nova+Dassarma%2C+Eli+Tran%2DJohnson%2C+Scott+Johnston%2C+Sheer+El%2DShowk%2C+Andy+Jones%2C+Nelso)
16. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. [Chatbot arena: An open platform for evaluating llms by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/). arXiv preprint arXiv:2403.04132.
17. Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkid… — [Challenges and Strategies in Cross-Cultural NLP](https://md.research.elamur.ai/free/en/papers/hershcovich-2022-challenges-strategies-cross-cultural-nlp/)
18. Amr Keleg and Walid Magdy. 2023. Dlama: A framework for curating culturally diverse facts for probing the knowledge of pretrained language models. arXiv preprint arXiv:2306.05076. [find](https://scholar.google.com/scholar?q=Amr+Keleg+and+Walid+Magdy%2E+2023%2E+Dlama%3A+A+framework+for+curating+culturally+diverse+facts+for+probing+the+knowledge+of+pretrained+language+models%2E+arXiv+preprint+arXiv%3A2306%2E05076%2E)
19. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104.14337. [find](https://scholar.google.com/scholar?q=Douwe+Kiela%2C+Max+Bartolo%2C+Yixin+Nie%2C+Divyansh+Kaushik%2C+Atticus+Geiger%2C+Zhengxuan+Wu%2C+Bertie+Vidgen%2C+Grusha+Prasad%2C+Amanpreet+Singh%2C+Pratik+Ringshia%2C+et+al%2E+2021%2E+Dynabench%3A+Rethinking+benchmarking+in+)
20. Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024. [Culturellm: Incorporating cultural differences into large language models](https://md.research.elamur.ai/free/en/papers/li-2024-culturellm-incorporating-cultural-differences-large/). ArXiv, abs/2402.10946.
21. Chen Cecilia Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2023. [Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings](https://md.research.elamur.ai/free/en/papers/liu-2023-multilingual-llms-culturally-diverse-reasoners/). arXiv preprint arXiv:2309.08591.
22. Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. 2024. [Best practices and lessons learned on synthetic data](https://md.research.elamur.ai/free/en/papers/liu-2024-best-practices-lessons-learned-synthetic/) for language models. arXiv preprint arXiv:2404.07503.
23. Himanshi Lohchab and Annapurna Roy. 2024. Indian gig workers toil at frontlines of ai revolution. [find](https://scholar.google.com/scholar?q=Himanshi+Lohchab+and+Annapurna+Roy%2E+2024%2E+Indian+gig+workers+toil+at+frontlines+of+ai+revolution%2E)
24. Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. 2023. [A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity](https://md.research.elamur.ai/free/en/papers/longpre-2023-pretrainer-s-guide-training-data/). Preprint, arXiv:2305.13…
25. Jun-Hee Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Pérez-Almendros, Abinew Ali Ayele, V’ictor Guti’errez-Basulto, Yazm’in Ib’anez-Garc’ia, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Rzayev, Nina White, Seid Muhie… — [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://md.research.elamur.ai/free/en/papers/myung-2024-blend-benchmark-llms-everyday-knowledge/)
26. Tarek Naous, Michael Joseph Ryan, and Wei Xu. 2023. [Having beer after prayer? measuring cultural bias in large language models](https://md.research.elamur.ai/free/en/papers/naous-2023-having-beer-after-prayer-measuring/). ArXiv, abs/2305.14456.
27. Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Karolina Stanczak, and Aishwarya Agrawal. 2024. [Benchmarking vision language models for cultural understanding](https://md.research.elamur.ai/free/en/papers/nayak-2024-benchmarking-vision-language-models-cultural/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,…
28. Tuan-Phong Nguyen, Simon Razniewski, Aparna S. Varde, and Gerhard Weikum. 2022. [Extracting cultural commonsense knowledge at scale](https://md.research.elamur.ai/free/en/papers/nguyen-2022-extracting-cultural-commonsense-knowledge-scale/). Proceedings of the ACM Web Conference 2023.
29. Shramay Palta and Rachel Rudinger. 2023a. [FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models](https://md.research.elamur.ai/free/en/papers/palta-2023-fork-bite-sized-test-set/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 9952–9962, Toronto, Canada. Association for Computational Linguistics.
30. Shramay Palta and Rachel Rudinger. 2023b. [Fork: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models](https://md.research.elamur.ai/free/en/papers/palta-2023-fork-bite-sized-test-set/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 9952–9962.
31. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Conference on Empirical Methods in Natural Language Processing.
32. Aida Ramezani and Yang Xu. 2023. [Knowledge of cultural moral norms in large language models](https://md.research.elamur.ai/free/en/papers/ramezani-2023-knowledge-cultural-moral-norms-large/). arXiv preprint arXiv:2306.01857.
33. Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A benchmark for measuring the cultural adaptability of large language models. ArXiv, abs/2404.12464. — [arXiv:2404.12464](https://arxiv.org/abs/2404.12464) [find](https://scholar.google.com/scholar?q=Abhinav+Rao%2C+Akhila+Yerukola%2C+Vishwa+Shah%2C+Katharina+Reinecke%2C+and+Maarten+Sap%2E+2024%2E+Normad%3A+A+benchmark+for+measuring+the+cultural+adaptability+of+large+language+models%2E+ArXiv%2C+abs%2F2404%2E12464%2E+%E2%80%94+%5Bar)
34. David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. 2024. [Cvqa: Culturally-diverse multilingual visual question answering benchmark](https://md.research.elamur.ai/free/en/papers/romero-2024-cvqa-culturally-diverse-multilingual-visual/). arXiv preprint arXiv:2406.05967.
35. Sebastin Santy, Jenny T Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. [Nlpositionality: Characterizing design biases of datasets and models](https://md.research.elamur.ai/free/en/papers/santy-2023-nlpositionality-characterizing-design-biases-datasets/). arXiv preprint arXiv:2306.01943.
36. Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Raya Horesh, Rogério Abreu de Paula, Diyi Yang, et al. 2024. [Culturebank: An online community-driven knowledge base towards culturally aware language technologies](https://md.research.elamur.ai/free/en/papers/shi-2024-culturebank-online-community-driven-knowledge/). arXiv preprint arXiv:2404.15238.
37. Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. 2024. Preference fine-tuning of llms should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14367. [find](https://scholar.google.com/scholar?q=Fahim+Tajwar%2C+Anikait+Singh%2C+Archit+Sharma%2C+Rafael+Rafailov%2C+Jeff+Schneider%2C+Tengyang+Xie%2C+Stefano+Ermon%2C+Chelsea+Finn%2C+and+Aviral+Kumar%2E+2024%2E+Preference+fine%2Dtuning+of+llms+should+leverage+suboptima)
38. Yan Tao, Olga Viberg, Ryan S Baker, and Rene F Kizilcec. 2023. [Auditing and mitigating cultural bias in llms](https://md.research.elamur.ai/free/en/papers/tao-2023-cultural-bias-cultural-alignment-large/). arXiv preprint arXiv:2311.14096.
39. Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. 2023. [Rethinking benchmark and contamination for language models with rephrased samples](https://md.research.elamur.ai/free/en/papers/yang-2023-rethinking-benchmark-contamination-language-models/). ArXiv, abs/2311.04850.
40. Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. [Low-resource languages jailbreak gpt-4](https://md.research.elamur.ai/free/en/papers/yong-2023-low-resource-languages-jailbreak-gpt/). arXiv preprint arXiv:2310.02446.
41. Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024. [Generative verifiers: Reward modeling as next-token prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/). arXiv preprint arXiv:2408.15240.
42. Human Quality Check for each culture Data Collection via red-teaming Team Human: Real-world cultural knowledge and observations Team AI Writing: Formulate Question Team AI Verifier: Being challenged Diverse Challenging Human & AI: Filter out non-cultural Team question and grammar check 👤👤 Five human… [find](https://scholar.google.com/scholar?q=Human+Quality+Check+for+each+culture+Data+Collection+via+red%2Dteaming+Team+Human%3A+Real%2Dworld+cultural+knowledge+and+observations+Team+AI+Writing%3A+Formulate+Question+Team+AI+Verifier%3A+Being+challenged+D)

*Updated 2 November 2026*