---
title: "RoleConflictBench — paper by Shin et al. (2026)"
description: "The authors present RoleConflictBench, which uses situational urgency as an objective reference: a critical situation should… 48 references."
authors: ["Jisu Shin", "Hoyun Song", "Juhyun Oh", "Changgeon Ko", "Eunsu Kim", "Chani Jung", "Alice Oh"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1695/"
doi: "10.18653/v1/2026.findings-acl.1695"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shin-2026-roleconflictbench-benchmark-role-conflict-scenarios/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shin-2026-roleconflictbench-benchmark-role-conflict-scenarios/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/41/) / RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity

# RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity

**Authors:** Jisu Shin, [Hoyun Song](https://md.research.elamur.ai/free/en/authors/hoyun-song/), [Juhyun Oh](https://md.research.elamur.ai/free/en/authors/juhyun-oh/), [Changgeon Ko](https://md.research.elamur.ai/free/en/authors/changgeon-ko/), [Eunsu Kim](https://md.research.elamur.ai/free/en/authors/eunsu-kim/), [Chani Jung](https://md.research.elamur.ai/free/en/authors/chani-jung/) and [Alice Oh](https://md.research.elamur.ai/free/en/authors/alice-oh/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1695/) · [PDF](https://aclanthology.org/2026.findings-acl.1695.pdf) · [DOI 10.18653/v1/2026.findings-acl.1695](https://doi.org/10.18653/v1/2026.findings-acl.1695) · [OpenAlex](https://openalex.org/W7166852158)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Role conflicts arise when obligations tied to two social roles, such as parent and employee, cannot both be met. They have no single correct answer, which makes it hard to tell whether LLMs actually weigh the situation or fall back on built-in preferences. The authors present RoleConflictBench, which uses situational urgency as an objective reference: a critical situation should outweigh a routine one, whichever roles are involved. The dataset comes from a three-stage LLM pipeline. First, typical expectations are generated for each role. Each expectation is then turned into situations at three urgency levels. Finally, two such situations are merged into a short first-person story in which the narrator must choose between the roles. The authors reviewed the urgency labels manually and validated them with human annotators. Across 10 LLMs, how often models side with the more urgent situation falls below a random baseline. Instead, their choices track stable preferences for certain roles. This matters for LLMs used as advisors or in social simulations, where ignoring the context of a dilemma can produce skewed or biased guidance.

## Key points
- RoleConflictBench contains 13,914 first-person scenarios of 100–200 words, built on 65 roles from five social domains.
- Urgency is scored 1 (routine), 2 (important but deferrable) or 3 (critical). Pairing two roles yields nine urgency combinations, so neither side always has trivially higher stakes.
- Models are evaluated with a Sensitivity Score, which measures how well their decisions follow the urgency baseline. All 10 tested LLMs perform worse than random on this measure.
- The models show a fixed ranking that favors Family and Occupation roles, as well as male and certain religious roles, regardless of urgency.
- Code and data are released on GitHub (ddindidu/RoleConflictBench). The authors note that the urgency criteria may reflect Western norms and that the benchmark covers only single-turn binary choices.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Korea Advanced Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/korea-advanced-institute-of-science-and-technology/) — University, Daejeon, South Korea · 51 works · 3,167 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Jisu Shin, Hoyun Song, Juhyun Oh, Changgeon Ko, Eunsu Kim, Chani Jung and Alice Oh. 2026. RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shin2026roleconflictbench,
  title = {{RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity}},
  author = {Jisu Shin, Hoyun Song, Juhyun Oh, Changgeon Ko, Eunsu Kim, Chani Jung and Alice Oh},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1695},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1695}
}
```

- [← previous Role-Adapted Clinical Report Generation for Ultrasound Measurements in Low-Resource Settings](https://md.research.elamur.ai/free/en/papers/nainia-2026-role-adapted-clinical-report-generation/)
- [next → Roles of Predictability and Acoustic Distance in Sound Discrimination via Contrastive Learning](https://md.research.elamur.ai/free/en/papers/zhang-2026-roles-predictability-acoustic-distance-sound/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/41/)

## References

48 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 120–128, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=setting%2E+In+Proceedings+of+the+Conference+on+Fairness%2C+Accountability%2C+and+Transparency%2C+FAT%2A+%E2%80%9919%2C+page+120%E2%80%93128%2C+New+York%2C+NY%2C+USA%2E+Association+for+Computing+Machinery%2E)
2. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. [Aligning ai with shared human values](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-aligning-ai-shared-human-values/). In International Conference on Learning Representations.
3. Soyeong Jeong, Aparna Elangovan, Emine Yilmaz, and Oleg Rokhlenko. 2025. Adaptive multi-agent response refinement in conversational systems. arXiv preprint arXiv:2511.08319. — [arXiv:2511.08319](https://arxiv.org/abs/2511.08319) [find](https://scholar.google.com/scholar?q=Soyeong+Jeong%2C+Aparna+Elangovan%2C+Emine+Yilmaz%2C+and+Oleg+Rokhlenko%2E+2025%2E+Adaptive+multi%2Dagent+response+refinement+in+conversational+systems%2E+arXiv+preprint+arXiv%3A2511%2E08319%2E+%E2%80%94+%5BarXiv%3A2511%2E08319%5D%28https)
4. Jianchao Ji, Yutong Chen, Mingyu Jin, Wujiang Xu, Wenyue Hua, and Yongfeng Zhang. 2025. Moralbench: Moral evaluation of llms. ACM SIGKDD Explorations Newsletter, 27(1):62–71. [find](https://scholar.google.com/scholar?q=Jianchao+Ji%2C+Yutong+Chen%2C+Mingyu+Jin%2C+Wujiang+Xu%2C+Wenyue+Hua%2C+and+Yongfeng+Zhang%2E+2025%2E+Moralbench%3A+Moral+evaluation+of+llms%2E+ACM+SIGKDD+Explorations+Newsletter%2C+27%281%29%3A62%E2%80%9371%2E)
5. Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, and 1 others. 2021. Can machines learn morality? the delphi experiment. arXiv preprint arXiv:2110.07574. [find](https://scholar.google.com/scholar?q=Liwei+Jiang%2C+Jena+D+Hwang%2C+Chandra+Bhagavatula%2C+Ronan+Le+Bras%2C+Jenny+Liang%2C+Jesse+Dodge%2C+Keisuke+Sakaguchi%2C+Maxwell+Forbes%2C+Jon+Borchardt%2C+Saadia+Gabriel%2C+and+1+others%2E+2021%2E+Can+machines+learn+morali)
6. Jiho Jin, Woosung Kang, Junho Myung, and Alice Oh. 2025. Social bias benchmark for generation: A comparison of generation and QA-based evaluations. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11215–11228, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiho+Jin%2C+Woosung+Kang%2C+Junho+Myung%2C+and+Alice+Oh%2E+2025%2E+Social+bias+benchmark+for+generation%3A+A+comparison+of+generation+and+QA%2Dbased+evaluations%2E+In+Findings+of+the+Association+for+Computational+Lin)
7. Zhijing Jin, Sydney Levine, Fernando Gonzalez Adauto, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Schölkopf. 2022. When to make exceptions: Exploring language models as accounts of human moral judgment. Advances in neural information processing systems, 35:… [find](https://scholar.google.com/scholar?q=Zhijing+Jin%2C+Sydney+Levine%2C+Fernando+Gonzalez+Adauto%2C+Ojasv+Kamal%2C+Maarten+Sap%2C+Mrinmaya+Sachan%2C+Rada+Mihalcea%2C+Josh+Tenenbaum%2C+and+Bernhard+Sch%C3%B6lkopf%2E+2022%2E+When+to+make+exceptions%3A+Exploring+languag)
8. David Jurgens, Agrima Seth, Jackson Sargent, Athena Aghighi, and Michael Geraci. 2023. Your spouse needs professional help: Determining the contextual appropriateness of messages through modeling social relationships. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin… [find](https://scholar.google.com/scholar?q=David+Jurgens%2C+Agrima+Seth%2C+Jackson+Sargent%2C+Athena+Aghighi%2C+and+Michael+Geraci%2E+2023%2E+Your+spouse+needs+professional+help%3A+Determining+the+contextual+appropriateness+of+messages+through+modeling+soci)
9. Mahammed Kamruzzaman and Gene Louis Kim. 2025a. Exploring changes in nation perception with nationality-assigned personas in LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 3660–3678, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mahammed+Kamruzzaman+and+Gene+Louis+Kim%2E+2025a%2E+Exploring+changes+in+nation+perception+with+nationality%2Dassigned+personas+in+LLMs%2E+In+Proceedings+of+the+2025+Conference+on+Empirical+Methods+in+Natural)
10. Mahammed Kamruzzaman and Gene Louis Kim. 2025b. The impact of name age perception on job recommendations in LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 15033– 15058, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mahammed+Kamruzzaman+and+Gene+Louis+Kim%2E+2025b%2E+The+impact+of+name+age+perception+on+job+recommendations+in+LLMs%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2025%2C+pages+15033%E2%80%93+1)
11. Aditi Khandelwal, Utkarsh Agarwal, Kumar Tanmay, and Monojit Choudhury. 2024. Do moral judgment and reasoning capability of LLMs change with language? a study using the multilingual defining issues test. In Proceedings of the 18th Conference of the European Chapter of the Association for Computation… [find](https://scholar.google.com/scholar?q=Aditi+Khandelwal%2C+Utkarsh+Agarwal%2C+Kumar+Tanmay%2C+and+Monojit+Choudhury%2E+2024%2E+Do+moral+judgment+and+reasoning+capability+of+LLMs+change+with+language%3F+a+study+using+the+multilingual+defining+issues+te)
12. Julia Kharchenko, Tanya Roosta, Aman Chadha, and Chirag Shah. 2024. How well do llms represent values across cultures? empirical analysis of llm responses based on hofstede cultural dimensions. arXiv preprint arXiv:2406.14805. — [arXiv:2406.14805](https://arxiv.org/abs/2406.14805) [find](https://scholar.google.com/scholar?q=Julia+Kharchenko%2C+Tanya+Roosta%2C+Aman+Chadha%2C+and+Chirag+Shah%2E+2024%2E+How+well+do+llms+represent+values+across+cultures%3F+empirical+analysis+of+llm+responses+based+on+hofstede+cultural+dimensions%2E+arXiv+)
13. Eunsu Kim, Junyeong Park, Juhyun Oh, Kiwoong Park, Seyoung Song, A Seza Doğruöz, Najoung Kim, and Alice Oh. 2025a. Are they lovers or friends? evaluating llms’ social reasoning in english and korean dialogues. arXiv preprint arXiv:2510.19028. [find](https://scholar.google.com/scholar?q=Eunsu+Kim%2C+Junyeong+Park%2C+Juhyun+Oh%2C+Kiwoong+Park%2C+Seyoung+Song%2C+A+Seza+Dog%CC%86ru%C3%B6z%2C+Najoung+Kim%2C+and+Alice+Oh%2E+2025a%2E+Are+they+lovers+or+friends%3F+evaluating+llms%E2%80%99+social+reasoning+in+english+and+korean+)
14. Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, and 1 others. 2023. [Soda: Million-scale dialogue distillation with social commonsense contextualization](https://md.research.elamur.ai/free/en/papers/kim-2023-soda-million-scale-dialogue-distillation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural…
15. Jiseon Kim, Jea Kwon, Luiz Felipe Vecchietti, Alice Oh, and Meeyoung Cha. 2025b. Exploring persona-dependent llm alignment for the moral machine experiment. arXiv preprint arXiv:2504.10886. — [arXiv:2504.10886](https://arxiv.org/abs/2504.10886) [find](https://scholar.google.com/scholar?q=Jiseon+Kim%2C+Jea+Kwon%2C+Luiz+Felipe+Vecchietti%2C+Alice+Oh%2C+and+Meeyoung+Cha%2E+2025b%2E+Exploring+persona%2Ddependent+llm+alignment+for+the+moral+machine+experiment%2E+arXiv+preprint+arXiv%3A2504%2E10886%2E+%E2%80%94+%5BarXiv%3A2)
16. Minseo Kim, Sujeong Im, Junseong Choi, Junhee Lee, Chaeeun Shim, and Edward Choi. 2026. Pi-con: A multi-turn interrogation framework for evaluating persona agent consistency. arXiv preprint arXiv:2603.25620. — [arXiv:2603.25620](https://arxiv.org/abs/2603.25620) [find](https://scholar.google.com/scholar?q=Minseo+Kim%2C+Sujeong+Im%2C+Junseong+Choi%2C+Junhee+Lee%2C+Chaeeun+Shim%2C+and+Edward+Choi%2E+2026%2E+Pi%2Dcon%3A+A+multi%2Dturn+interrogation+framework+for+evaluating+persona+agent+consistency%2E+arXiv+preprint+arXiv%3A2603)
17. Changgeon Ko, Jisu Shin, Hoyun Song, Jeongyeon Seo, and Jong C Park. 2024. Different bias under different criteria: Assessing bias in llms with a fact-based approach. arXiv preprint arXiv:2411.17338. — [arXiv:2411.17338](https://arxiv.org/abs/2411.17338) [find](https://scholar.google.com/scholar?q=Changgeon+Ko%2C+Jisu+Shin%2C+Hoyun+Song%2C+Jeongyeon+Seo%2C+and+Jong+C+Park%2E+2024%2E+Different+bias+under+different+criteria%3A+Assessing+bias+in+llms+with+a+fact%2Dbased+approach%2E+arXiv+preprint+arXiv%3A2411%2E17338%2E+)
18. Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2018%2E+Content+analysis%3A+An+introduction+to+its+methodology%2E+Sage+publications%2E)
19. Ayoung Lee, Ryan Sungmo Kwon, Peter Railton, and Lu Wang. 2025. Clash: Evaluating language models on judging high-stakes dilemmas from multiple perspectives. arXiv preprint arXiv:2504.10823. — [arXiv:2504.10823](https://arxiv.org/abs/2504.10823) [find](https://scholar.google.com/scholar?q=Ayoung+Lee%2C+Ryan+Sungmo+Kwon%2C+Peter+Railton%2C+and+Lu+Wang%2E+2025%2E+Clash%3A+Evaluating+language+models+on+judging+high%2Dstakes+dilemmas+from+multiple+perspectives%2E+arXiv+preprint+arXiv%3A2504%2E10823%2E+%E2%80%94+%5BarXiv%3A)
20. Jiyoung Lee, Minwoo Kim, Seungho Kim, Junghwan Kim, Seunghyun Won, Hwaran Lee, and Edward Choi. 2024. KorNAT: LLM alignment benchmark for Korean social values and common knowledge. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11177–11213, Bangkok, Thailand. Associati… [find](https://scholar.google.com/scholar?q=Jiyoung+Lee%2C+Minwoo+Kim%2C+Seungho+Kim%2C+Junghwan+Kim%2C+Seunghyun+Won%2C+Hwaran+Lee%2C+and+Edward+Choi%2E+2024%2E+KorNAT%3A+LLM+alignment+benchmark+for+Korean+social+values+and+common+knowledge%2E+In+Findings+of+the+)
21. Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2021. Scruples: A corpus of community ethical judgments on 32,000 real-life anecdotes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13470–13479. [find](https://scholar.google.com/scholar?q=Nicholas+Lourie%2C+Ronan+Le+Bras%2C+and+Yejin+Choi%2E+2021%2E+Scruples%3A+A+corpus+of+community+ethical+judgments+on+32%2C000+real%2Dlife+anecdotes%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C)
22. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. [StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
23. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. [CrowS-pairs: A challenge dataset for measuring social biases in masked language models](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for…
24. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, and 1 others. 2024. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656. [find](https://scholar.google.com/scholar?q=Team+OLMo%2C+Pete+Walsh%2C+Luca+Soldaini%2C+Dirk+Groeneveld%2C+Kyle+Lo%2C+Shane+Arora%2C+Akshita+Bhagia%2C+Yuling+Gu%2C+Shengyi+Huang%2C+Matt+Jordan%2C+and+1+others%2E+2024%2E+2+olmo+2+furious%2E+arXiv+preprint+arXiv%3A2501%2E0065)
25. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=Arjun+Panickssery%2C+Samuel+R%2E+Bowman%2C+and+Shi+Feng%2E+2024%2E+Llm+evaluators+recognize+and+favor+their+own+generations%2E+In+Proceedings+of+the+38th+International+Conference+on+Neural+Information+Processing+)
26. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22. [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+O%E2%80%99Brien%2C+Carrie+Jun+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S+Bernstein%2E+2023%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+In+Proceedings+of+the+36)
27. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. [BBQ: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Asso…
28. Valentina Pyatkin, Jena D. Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi, and Chandra Bhagavatula. 2023. ClarifyDelphi: Reinforced clarification questions with defeasibility rewards for social and moral situations. In Proceedings of the 61st Annual Meeting of the Association for Computat… [find](https://scholar.google.com/scholar?q=Valentina+Pyatkin%2C+Jena+D%2E+Hwang%2C+Vivek+Srikumar%2C+Ximing+Lu%2C+Liwei+Jiang%2C+Yejin+Choi%2C+and+Chandra+Bhagavatula%2E+2023%2E+ClarifyDelphi%3A+Reinforced+clarification+questions+with+defeasibility+rewards+for+so)
29. Haoyi Qiu, Alexander Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien-Sheng Wu. 2025. Evaluating cultural and social awareness of LLM web agents. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3978–4005, Albuquerque, New Mexico. Associat… [find](https://scholar.google.com/scholar?q=Haoyi+Qiu%2C+Alexander+Fabbri%2C+Divyansh+Agarwal%2C+Kung%2DHsiang+Huang%2C+Sarah+Tan%2C+Nanyun+Peng%2C+and+Chien%2DSheng+Wu%2E+2025%2E+Evaluating+cultural+and+social+awareness+of+LLM+web+agents%2E+In+Findings+of+the+Assoc)
30. Donya Rooein, Vilém Zouhar, Debora Nozza, and Dirk Hovy. 2025. Biased tales: Cultural and topic bias in generating children’s stories. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 52–72, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Donya+Rooein%2C+Vil%C3%A9m+Zouhar%2C+Debora+Nozza%2C+and+Dirk+Hovy%2E+2025%2E+Biased+tales%3A+Cultural+and+topic+bias+in+generating+children%E2%80%99s+stories%2E+In+Proceedings+of+the+2025+Conference+on+Empirical+Methods+in+Nat)
31. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. [Gender bias in coreference resolution](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1…
32. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le-Bras, and Yejin Choi. 2019. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). arXiv preprint arXiv:1904.09728.
33. Shalom H Schwartz. 1992. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Advances in experimental social psychology, volume 25, pages 1–65. Elsevier. [find](https://scholar.google.com/scholar?q=Shalom+H+Schwartz%2E+1992%2E+Universals+in+the+content+and+structure+of+values%3A+Theoretical+advances+and+empirical+tests+in+20+countries%2E+In+Advances+in+experimental+social+psychology%2C+volume+25%2C+pages+1%E2%80%93)
34. Shalom H Schwartz. 1994. Are there universal aspects in the structure and contents of human values? Journal of social issues, 50(4):19–45. [find](https://scholar.google.com/scholar?q=Shalom+H+Schwartz%2E+1994%2E+Are+there+universal+aspects+in+the+structure+and+contents+of+human+values%3F+Journal+of+social+issues%2C+50%284%29%3A19%E2%80%9345%2E)
35. Shalom H Schwartz, Jan Cieciuch, Michele Vecchione, Eldad Davidov, Ronald Fischer, Constanze Beierlein, Alice Ramos, Markku Verkasalo, Jan-Erik Lönnqvist, Kursad Demirutku, and 1 others. 2012. Refining the theory of basic individual values. Journal of personality and social psychology, 103(4):663. y… [find](https://scholar.google.com/scholar?q=Shalom+H+Schwartz%2C+Jan+Cieciuch%2C+Michele+Vecchione%2C+Eldad+Davidov%2C+Ronald+Fischer%2C+Constanze+Beierlein%2C+Alice+Ramos%2C+Markku+Verkasalo%2C+Jan%2DErik+L%C3%B6nnqvist%2C+Kursad+Demirutku%2C+and+1+others%2E+2012%2E+Refinin)
36. Taylor Sorensen, Liwei Jiang, Jena D Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, and 1 others. 2024. [Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties](https://md.research.elamur.ai/free/en/papers/sorensen-2023-value-kaleidoscope-engaging-ai-pluralistic/). In Proceedings of the AAAI Conference on Artificial…
37. Takehiro Takayanagi, Kiyoshi Izumi, Javier Sanz-Cruzado, Richard McCreadie, and Iadh Ounis. 2025. Are generative ai agents effective personalized financial advisors? In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 286–295. [find](https://scholar.google.com/scholar?q=Takehiro+Takayanagi%2C+Kiyoshi+Izumi%2C+Javier+Sanz%2DCruzado%2C+Richard+McCreadie%2C+and+Iadh+Ounis%2E+2025%2E+Are+generative+ai+agents+effective+personalized+financial+advisors%3F+In+Proceedings+of+the+48th+Interna)
38. Kumar Tanmay, Aditi Khandelwal, Utkarsh Agarwal, and Monojit Choudhury. 2023. Probing the moral development of large language models through defining issues test. arXiv preprint arXiv:2309.13356. — [arXiv:2309.13356](https://arxiv.org/abs/2309.13356) [find](https://scholar.google.com/scholar?q=Kumar+Tanmay%2C+Aditi+Khandelwal%2C+Utkarsh+Agarwal%2C+and+Monojit+Choudhury%2E+2023%2E+Probing+the+moral+development+of+large+language+models+through+defining+issues+test%2E+arXiv+preprint+arXiv%3A2309%2E13356%2E+%E2%80%94+%5Ba)
39. U.S. Bureau of Labor Statistics. 2025. Occupational employment and wages. [https://www.bls.gov/](https://www.bls.gov/) news.release/ocwage.htm. Online; accessed 22 September 2025. [↗](https://www.bls.gov/)
40. Alexander Sasha Vezhnevets, John P Agapiou, Avia Aharon, Ron Ziv, Jayd Matyas, Edgar A Duéñez-Guzmán, William A Cunningham, Simon Osindero, Danny Karmon, and Joel Z Leibo. 2023. Generative agent-based modeling with actions grounded in physical, social, or digital space using concordia. arXiv preprin… — [arXiv:2312.03664](https://arxiv.org/abs/2312.03664) [find](https://scholar.google.com/scholar?q=Alexander+Sasha+Vezhnevets%2C+John+P+Agapiou%2C+Avia+Aharon%2C+Ron+Ziv%2C+Jayd+Matyas%2C+Edgar+A+Du%C3%A9%C3%B1ez%2DGuzm%C3%A1n%2C+William+A+Cunningham%2C+Simon+Osindero%2C+Danny+Karmon%2C+and+Joel+Z+Leibo%2E+2023%2E+Generative+agent%2Dbased)
41. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2024. [Self-preference bias in llm-as-a-judge](https://md.research.elamur.ai/free/en/papers/wataoka-2024-self-preference-bias-llm-judge/). arXiv preprint arXiv:2410.21819.
42. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
43. Ye Yuan, Kexin Tang, Jianhao Shen, Ming Zhang, and Chenguang Wang. 2024. Measuring social norms of large language models. arXiv preprint arXiv:2404.02491. [find](https://scholar.google.com/scholar?q=Ye+Yuan%2C+Kexin+Tang%2C+Jianhao+Shen%2C+Ming+Zhang%2C+and+Chenguang+Wang%2E+2024%2E+Measuring+social+norms+of+large+language+models%2E+arXiv+preprint+arXiv%3A2404%2E02491%2E)
44. Jisu Shin, Juhyun Oh, Eunsu Kim, Hoyun Song, and Alice Oh. 2025. Spotting out-of-character behavior: Atomic-level evaluation of persona fidelity in open-ended generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 26312–26332, Vienna, Austria. Association for Compu… [find](https://scholar.google.com/scholar?q=Jisu+Shin%2C+Juhyun+Oh%2C+Eunsu+Kim%2C+Hoyun+Song%2C+and+Alice+Oh%2E+2025%2E+Spotting+out%2Dof%2Dcharacter+behavior%3A+Atomic%2Dlevel+evaluation+of+persona+fidelity+in+open%2Dended+generation%2E+In+Findings+of+the+Associatio)
45. Haolan Zhan, Zhuang Li, Yufei Wang, Linhao Luo, Tao Feng, Xiaoxi Kang, Yuncheng Hua, Lizhen Qu, Lay-Ki Soon, Suraj Sharma, and 1 others. 2023. Socialdial: A benchmark for socially-aware dialogue systems. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Inf… [find](https://scholar.google.com/scholar?q=Haolan+Zhan%2C+Zhuang+Li%2C+Yufei+Wang%2C+Linhao+Luo%2C+Tao+Feng%2C+Xiaoxi+Kang%2C+Yuncheng+Hua%2C+Lizhen+Qu%2C+Lay%2DKi+Soon%2C+Suraj+Sharma%2C+and+1+others%2E+2023%2E+Socialdial%3A+A+benchmark+for+socially%2Daware+dialogue+syste)
46. Jisu Shin, Hoyun Song, Huije Lee, Soyeong Jeong, and Jong Park. 2024. Ask LLMs directly, “what shapes Jifan Zhang, Henry Sleight, Andi Peng, John Schulman, and Esin Durmus. 2025. Stress-testing model specs reveals character differences among language models. arXiv preprint arXiv:2510.07686. [find](https://scholar.google.com/scholar?q=Jisu+Shin%2C+Hoyun+Song%2C+Huije+Lee%2C+Soyeong+Jeong%2C+and+Jong+Park%2E+2024%2E+Ask+LLMs+directly%2C+%E2%80%9Cwhat+shapes+Jifan+Zhang%2C+Henry+Sleight%2C+Andi+Peng%2C+John+Schulman%2C+and+Esin+Durmus%2E+2025%2E+Stress%2Dtesting+model+)
47. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. [Gender bias in coreference resolution: Evaluation and debiasing methods](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
48. Jingyan Zhou, Minda Hu, Junan Li, Xiaoying Zhang, Xixin Wu, Irwin King, and Helen Meng. 2023. Rethinking machine ethics–can llms perform moral reasoning through the lens of moral theories? arXiv preprint arXiv:2308.15399. [find](https://scholar.google.com/scholar?q=Jingyan+Zhou%2C+Minda+Hu%2C+Junan+Li%2C+Xiaoying+Zhang%2C+Xixin+Wu%2C+Irwin+King%2C+and+Helen+Meng%2E+2023%2E+Rethinking+machine+ethics%E2%80%93can+llms+perform+moral+reasoning+through+the+lens+of+moral+theories%3F+arXiv+prepr)

*Updated 1 November 2026*