---
title: "Conceptual Cultural Index — paper by Ohashi and Iyatomi (2026)"
description: "The authors propose the Conceptual Cultural Index (CCI) to fill this gap. 3.1. AUC. 21 references."
authors: ["Takumi Ohashi", "Hitoshi Iyatomi"]
year: 2026
venue: "Proceedings of the First Workshop on Multilingual Multicultural Evaluation"
source: "https://aclanthology.org/2026.mme-main.5/"
doi: "10.18653/v1/2026.mme-main.5"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ohashi-2026-conceptual-cultural-index-metric-cultural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ohashi-2026-conceptual-cultural-index-metric-cultural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/) / Conceptual Cultural Index: A Metric for Cultural Specificity via Relative Generality

# Conceptual Cultural Index: A Metric for Cultural Specificity via Relative Generality

**Authors:** [Takumi Ohashi](https://md.research.elamur.ai/free/en/authors/takumi-ohashi/) and [Hitoshi Iyatomi](https://md.research.elamur.ai/free/en/authors/hitoshi-iyatomi/) · **Year:** 2026 · **Venue:** [Proceedings of the First Workshop on Multilingual Multicultural Evaluation](https://md.research.elamur.ai/free/en/venues/mme/) ([Workshop on Multilingual Multicultural Evaluation 2026](https://md.research.elamur.ai/free/en/venues/mme/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.mme-main.5/) · [PDF](https://aclanthology.org/2026.mme-main.5.pdf) · [DOI 10.18653/v1/2026.mme-main.5](https://doi.org/10.18653/v1/2026.mme-main.5) · [OpenAlex](https://openalex.org/W7140107508)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Cultural benchmarks for LLMs usually report a single accuracy score. That score hides how strongly each item depends on one particular culture, and existing cultural-knowledge corpora have no sentence-level specificity labels. The authors propose the Conceptual Cultural Index (CCI) to fill this gap. An LLM rates how common a sentence is in each of several cultures. CCI is the target culture's rating minus the mean rating of the other cultures. Choosing which cultures to compare against controls what "culture" means in a given analysis.

The method was tested with Japan as the target culture. The test set had 400 GPT-5-generated, manually filtered sentences, half Japan-specific and half general. Five LLMs were used, and CCI was compared with directly asking each model for a specificity score. CCI separated the two classes more clearly, with especially large AUC gains for Japanese-specialized models. The authors also annotated two Japanese benchmarks with CCI. Model accuracy fell as cultural specificity rose.

The work offers an interpretable, automatic way to stratify benchmarks, analyse errors and filter cultural data.

## Key points
- **Definition:** CCI lies in [−1, 1].
  - Near 0 means the sentence is cross-culturally general.
  - Near 1 means it is specific to the target culture.
  - Near −1 means it is specific to the comparison cultures.
  - Scores are averaged over N = 3 runs.
- **Comparison settings:**
  - Global mode uses the 19 G20 member countries.
  - Custom mode tests neighbor control: China/Korea/USA/Japan versus Brazil/France/USA/Japan.
- **Models tested:** Qwen 2.5, Llama 3.1, gpt-oss, Llama 3.1 Swallow and llm-jp 3.1.
- **AUC results (Global CCI vs. baseline):**
  - Llama-3.1-Swallow-8B rose from 0.842 to 0.945.
  - llm-jp-3.1-13b rose from 0.768 to 0.908.
  - gpt-oss-20b scored about 0.96 under both methods.
- **Benchmark stratification:** JCommonsenseQA (1,119 dev items) and JCommonsenseMorality (3,992 test items) were scored with gpt-oss CCI.
  - Most items have low CCI.
  - Accuracy tends to drop in high-CCI bins.
  - llm-jp 3.1 had the best overall accuracy (0.958 and 0.918) and the smallest drop.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hosei University](https://md.research.elamur.ai/free/en/universities/hosei-university/) — University, Tokyo, Japan · 3 works · 4 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 870 works

## Cite

Cite as: Takumi Ohashi and Hitoshi Iyatomi. 2026. Conceptual Cultural Index: A Metric for Cultural Specificity via Relative Generality. Proceedings of the First Workshop on Multilingual Multicultural Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ohashi2026conceptual,
  title = {{Conceptual Cultural Index: A Metric for Cultural Specificity via Relative Generality}},
  author = {Takumi Ohashi and Hitoshi Iyatomi},
  booktitle = {Proceedings of the First Workshop on Multilingual Multicultural Evaluation},
  year = {2026},
  doi = {10.18653/v1/2026.mme-main.5},
  url = {https://doi.org/10.18653/v1/2026.mme-main.5}
}
```

- [← previous Concept rather than Document: Context Compression via AMR-based Conceptual Entropy](https://md.research.elamur.ai/free/en/papers/shi-2026-concept-rather-than-document-context/)
- [next → Conceptualising love in Amharic](https://md.research.elamur.ai/free/en/papers/kacprzak-2026-conceptualising-love-amharic/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/25/)

## References

21 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, and 261 ot… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
2. Awantee Deshpande, Dana Ruiter, Marius Mosbach, and Dietrich Klakow. 2022. StereoKG: Data-Driven Knowledge Graph Construction For Cultural Knowledge and Stereotypes. In Proceedings of the Sixth Workshop on Online Abuse and Harms, pages 67–78. [find](https://scholar.google.com/scholar?q=Awantee+Deshpande%2C+Dana+Ruiter%2C+Marius+Mosbach%2C+and+Dietrich+Klakow%2E+2022%2E+StereoKG%3A+Data%2DDriven+Knowledge+Graph+Construction+For+Cultural+Knowledge+and+Stereotypes%2E+In+Proceedings+of+the+Sixth+Worksh)
3. Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Shivdutt Singh, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. 2024. [Towards Measuring and Modeling “Culture” in LLMs: A Survey](https://md.research.elamur.ai/free/en/papers/adilazuarda-2024-measuring-modeling-culture-llms-survey/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan…
4. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. [Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities](https://md.research.elamur.ai/free/en/papers/fujii-2024-continual-pre-training-cross-lingual/). In Proceedings of the First Conference on Langua…
5. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, and 106 others. 2025. [gpt-oss-120b & gpt-oss…](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/)
6. Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. [Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking](https://md.research.elamur.ai/free/en/papers/fung-2024-massively-multi-cultural-knowledge-acquisition/). arXiv preprint arXiv:2402.09369.
7. Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazuki Fujii, Kensuke Fukumoto, Takuya Fukushima, Namgi Han, Yuto Harada, Chikara Hashimoto, Tatsuya Hiraoka, Shohei Hisada, Sosuke Hosokawa, Lu Jie, Keisuke Kamata, Teruhito Kanazawa, Hiroki Kanezashi, and 62 othe… — [arXiv:2407.03963](https://arxiv.org/abs/2407.03963) [find](https://scholar.google.com/scholar?q=Akiko+Aizawa%2C+Eiji+Aramaki%2C+Bowen+Chen%2C+Fei+Cheng%2C+Hiroyuki+Deguchi%2C+Rintaro+Enomoto%2C+Kazuki+Fujii%2C+Kensuke+Fukumoto%2C+Takuya+Fukushima%2C+Namgi+Han%2C+Yuto+Harada%2C+Chikara+Hashimoto%2C+Tatsuya+Hiraoka%2C+Shoh)
8. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
9. Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. 2024. CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ips… [find](https://scholar.google.com/scholar?q=Eunsu+Kim%2C+Juyoung+Suk%2C+Philhoon+Oh%2C+Haneul+Yoo%2C+James+Thorne%2C+and+Alice+Oh%2E+2024%2E+CLIcK%3A+A+Benchmark+Dataset+of+Cultural+and+Linguistic+Intelligence+in+Korean%2E+In+Proceedings+of+the+2024+Joint+Intern)
10. Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Sunny Yu, Raya Horesh, Rogério Abreu De Paula, and Diyi Yang. 2024. [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://md.research.elamur.ai/free/en/papers/shi-2024-culturebank-online-community-driven-knowledge/). In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4…
11. Fajri Koto, Rahmad Mahendra, Nurul Aisyah, and Timothy Baldwin. 2024. [IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces](https://md.research.elamur.ai/free/en/papers/koto-2024-indoculture-exploring-geographically-influenced-cultural/). Transactions of the Association for Computational Linguistics, 12:1703–1719.
12. Jiaxing Sun, Weiquan Huang, Jiang Wu, Chenya Gu, Wei Li, Songyang Zhang, Hang Yan, and Conghui He. 2024. Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations. In Proceedings of the 62nd Annual Meeting of the Association for Computational L… [find](https://scholar.google.com/scholar?q=Jiaxing+Sun%2C+Weiquan+Huang%2C+Jiang+Wu%2C+Chenya+Gu%2C+Wei+Li%2C+Songyang+Zhang%2C+Hang+Yan%2C+and+Conghui+He%2E+2024%2E+Benchmarking+Chinese+Commonsense+Reasoning+of+LLMs%3A+From+Chinese%2DSpecifics+to+Reasoning%2DMemoriz)
13. Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. [JGLUE: Japanese General Language Understanding Evaluation](https://md.research.elamur.ai/free/en/papers/kurihara-2022-jglue-japanese-general-language-understanding/). In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2957–2966.
14. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Víctor Gutiérrez-Basulto, Yazmín Ibáñez García, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid Muhie… — [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://md.research.elamur.ai/free/en/papers/myung-2024-blend-benchmark-llms-everyday-knowledge/)
15. Masashi Takeshita and Rafal Rzepka. 2025. JETHICS: Japanese Ethics Understanding Evaluation Dataset. arXiv preprint arXiv:2506.16187. — [arXiv:2506.16187](https://arxiv.org/abs/2506.16187) [find](https://scholar.google.com/scholar?q=Masashi+Takeshita+and+Rafal+Rzepka%2E+2025%2E+JETHICS%3A+Japanese+Ethics+Understanding+Evaluation+Dataset%2E+arXiv+preprint+arXiv%3A2506%2E16187%2E+%E2%80%94+%5BarXiv%3A2506%2E16187%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2506%2E16187%29)
16. Yuhang Wang, Yanxu Zhu, Chao Kong, Shuyu Wei, Xiaoyuan Yi, Xing Xie, and Jitao Sang. 2024. CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models. In Proceedings of the 2nd Workshop on Cross-Cultural Considerations in NLP, pages 1–16. [find](https://scholar.google.com/scholar?q=Yuhang+Wang%2C+Yanxu+Zhu%2C+Chao+Kong%2C+Shuyu+Wei%2C+Xiaoyuan+Yi%2C+Xing+Xie%2C+and+Jitao+Sang%2E+2024%2E+CDEval%3A+A+Benchmark+for+Measuring+the+Cultural+Dimensions+of+Large+Language+Models%2E+In+Proceedings+of+the+2nd)
17. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2024. Qwen2.5 Technical Report. arXiv preprint arXi… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
18. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://md.research.elamur.ai/free/en/papers/naous-2023-having-beer-after-prayer-measuring/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393.
19. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. GeoM-LAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039–2055. [find](https://scholar.google.com/scholar?q=Da+Yin%2C+Hritik+Bansal%2C+Masoud+Monajatipoor%2C+Liunian+Harold+Li%2C+and+Kai%2DWei+Chang%2E+2022%2E+GeoM%2DLAMA%3A+Geo%2DDiverse+Commonsense+Probing+on+Multilingual+Pre%2DTrained+Language+Models%2E+In+Proceedings+of+the+20)
20. Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023. Extracting Cultural Commonsense Knowledge at Scale. In Proceedings of the ACM Web Conference 2023, page 1907–1917. [find](https://scholar.google.com/scholar?q=Tuan%2DPhong+Nguyen%2C+Simon+Razniewski%2C+Aparna+Varde%2C+and+Gerhard+Weikum%2E+2023%2E+Extracting+Cultural+Commonsense+Knowledge+at+Scale%2E+In+Proceedings+of+the+ACM+Web+Conference+2023%2C+page+1907%E2%80%931917%2E)
21. Tuan-Phong Nguyen, Simon Razniewski, and Gerhard Weikum. 2024. [Cultural Commonsense Knowledge for Intercultural Dialogues](https://md.research.elamur.ai/free/en/papers/nguyen-2024-cultural-commonsense-knowledge-intercultural-dialogues/). In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, page 1774–1784.

*Updated 1 November 2026*