---
title: "John vs. Ahmed: Debate-Induced Bias in Multilingual LLMs (2024)"
description: "This paper asks whether popular chat LLMs carry identity-related preferences that alignment hides… 22 references, cited by 8 papers here."
authors: ["Anastasiia Demidova", "Hanin Atwany", "Nour Rabih", "Sanad Sha'ban", "Muhammad Abdul-Mageed"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.18/"
doi: "10.18653/v1/2024.arabicnlp-1.18"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/demidova-2024-john-vs-ahmed-debate-induced/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/demidova-2024-john-vs-ahmed-debate-induced/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / John vs. Ahmed: Debate-Induced Bias in Multilingual LLMs

# John vs. Ahmed: Debate-Induced Bias in Multilingual LLMs

**Authors:** [Anastasiia Demidova](https://md.research.elamur.ai/free/en/authors/anastasiia-demidova/), [Hanin Atwany](https://md.research.elamur.ai/free/en/authors/hanin-atwany/), [Nour Rabih](https://md.research.elamur.ai/free/en/authors/nour-rabih/), [Sanad Sha'ban](https://md.research.elamur.ai/free/en/authors/sanad-sha-ban/) and [Muhammad Abdul-Mageed](https://md.research.elamur.ai/free/en/authors/muhammad-abdul-mageed/) · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.18/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.18.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.18](https://doi.org/10.18653/v1/2024.arabicnlp-1.18) · [OpenAlex](https://openalex.org/W4402671356)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/demidova-2024-john-vs-ahmed-debate-induced/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This paper asks whether popular chat LLMs carry identity-related preferences that alignment hides, and whether those preferences shift with the prompt language. The authors built argument-based prompts in Arabic, English and Russian. In these prompts, GPT-3.5 and Gemini write a debate between characters who differ in culture, politics, race, religion or gender, and then decide who won. Counting how often each identity is declared the winner turns a free-form story task into a measurable bias signal.

Across all five domains, both models showed systematic preferences. These preferences mostly stayed hidden unless the debate framing was used. A recurring pattern was that the winner tended to be the character tied to the language of the prompt. Arabic prompts produced the strongest skew regardless of topic, and Russian prompts gave the most balanced outcomes for both models.

The work matters because it extends safety evaluation beyond English. It also suggests that DPO/RLHF-style alignment may mask bias rather than remove it.

## Key points
- Models tested: GPT-3.5 (gpt-3.5-turbo) and Gemini. Prompt languages: Arabic, English, Russian. Domains: cultural, political, racial, religious, gender.
- Bias is measured as win percentages of paired identities in model-generated debates where the model picks the victor.
- Race was probed further on two topics, immigration and child labor:
  - On immigration with English prompts, Gemini let the Black debater win 3 of 5 debates, while GPT-3.5 gave Black debaters 0 of 5.
  - On child labor, English prompts gave win rates of roughly 55%, while Arabic prompts pushed them above 95%.
- Some patterns differed between models. With Arabic prompts, GPT-3.5 tended to favor Asian characters and Middle Easterners scored higher than in English. With Russian prompts, Gemini showed no preference among Asian, White and Black characters but gave Middle Easterners lower success.
- The authors list several limitations: only two models, three languages and five domains were studied, and the causes of the bias were not investigated.

## Interesting ideas
- Making the model judge a contest it narrated itself exposes preferences that direct questions never reveal, because the judgment is indirect and forced.
- Keeping the debate topic fixed and changing only the identities of the debaters works as a counterfactual test: any difference in outcome comes from the identity alone.
- Treating the prompt language as an experimental variable revealed effects that a single-language test would miss.
- Analyzing results per topic showed that the strength of the bias depends on the subject, not only on the identity.
- Win rates over every identity pair, arranged as a matrix, give a compact and comparable picture of who is favored over whom.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Before trusting an LLM teacher tier for silver labels, run forced-choice probes on it to spot systematic preferences that could leak into training data.
- **I2** (learn): Add negative controls that swap character names or nationalities in fables and word problems; the answers should stay identical.
- **I3** (back): When translating through the meta-world, check that the same text in different source languages yields the same world graph and the same downstream answers.
- **I4** (answer): Report QA and test accuracy sliced by topic and domain, so that weaknesses confined to one theme are not averaged away.
- **I5** (frames): When frames detect contests or disputes, record winner and loser roles explicitly so that outcome patterns across a corpus can be tabulated and inspected.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of British Columbia](https://md.research.elamur.ai/free/en/universities/university-of-british-columbia/) — University, Vancouver, Canada · 98 works · 34,805 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Anastasiia Demidova, Hanin Atwany, Nour Rabih, Sanad Sha'ban and Muhammad Abdul-Mageed. 2024. John vs. Ahmed: Debate-Induced Bias in Multilingual LLMs. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{demidova2024john,
  title = {{John vs. Ahmed: Debate-Induced Bias in Multilingual LLMs}},
  author = {Anastasiia Demidova, Hanin Atwany, Nour Rabih, Sanad Sha'ban and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.18},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.18}
}
```

- [← previous Halwasa: Quantify and Analyze Hallucinations in Large Language Models: Arabic as a Case Study](https://md.research.elamur.ai/free/en/papers/mubarak-2024-halwasa-quantify-analyze-hallucinations-large/)
- [next → Measuring and Modifying the Readability of English Texts with GPT-4](https://md.research.elamur.ai/free/en/papers/trott-2024-measuring-modifying-readability-english-texts/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Azalden Alakrot, Muftah Fraifer, and Nikola S Nikolov. 2021. Machine learning approach to detection of offensive language in online communication in arabic. In 2021 IEEE 1st International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering MI-STA, pages 244–249. IEEE. [find](https://scholar.google.com/scholar?q=Azalden+Alakrot%2C+Muftah+Fraifer%2C+and+Nikola+S+Nikolov%2E+2021%2E+Machine+learning+approach+to+detection+of+offensive+language+in+online+communication+in+arabic%2E+In+2021+IEEE+1st+International+Maghreb+Meet)
2. Emad A. Alghamdi, Reem I. Masoud, Deema Alnuhait, Afnan Y. Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2024. [Aratrust: An evaluation of trustworthiness for llms in arabic](https://md.research.elamur.ai/free/en/papers/alghamdi-2025-aratrust-evaluation-trustworthiness-llms-arabic/). Preprint, arXiv:2403.09017. [↗](https://arxiv.org/abs/2403.09017)
3. Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augenstein. 2023. [Probing pre-trained language models for cross-cultural differences in values](https://md.research.elamur.ai/free/en/papers/arora-2023-probing-pre-trained-language-models/). In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114–130, Dubrovnik, Croatia. Association for Computational Linguistics.
4. Noel F Ayoub, Karthik Balakrishnan, Marc S Ayoub, Thomas F Barrett, Abel P David, and Stacey T Gray. 2024. Inherent bias in large language models: A random sampling analysis. Mayo Clinic Proceedings: Digital Health, 2(2):186–191. [↗](https://doi.org/10.1016/j.mcpdig.2024.03.003)
5. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862–872. [find](https://scholar.google.com/scholar?q=Jwala+Dhamala%2C+Tony+Sun%2C+Varun+Kumar%2C+Satyapriya+Krishna%2C+Yada+Pruksachatkun%2C+Kai%2DWei+Chang%2C+and+Rahul+Gupta%2E+2021%2E+Bold%3A+Dataset+and+metrics+for+measuring+biases+in+open%2Dended+language+generation%2E+In)
6. Xiangjue Dong, Yibo Wang, Philip S. Yu, and James Caverlee. 2023. Probing explicit and implicit gender bias through llm conditional text generation. Preprint, arXiv:2311.00306. [↗](https://arxiv.org/abs/2311.00306)
7. Ashutosh Dwivedi, Pradhyumna Lavania, and Ashutosh Modi. 2023. Eticor: Corpus for analyzing llms for etiquettes. arXiv preprint arXiv:2310.18974. [↗](https://arxiv.org/abs/2310.18974)
8. Gemini Google Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805. [↗](https://arxiv.org/abs/2312.11805)
9. Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. Bias runs deep: Implicit reasoning biases in persona-assigned LLMs. In The Twelfth International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2311.04892)
10. Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference, CI ’23. ACM. [find](https://scholar.google.com/scholar?q=Hadas+Kotek%2C+Rikker+Dockum%2C+and+David+Sun%2E+2023%2E+Gender+bias+and+stereotypes+in+large+language+models%2E+In+Proceedings+of+The+ACM+Collective+Intelligence+Conference%2C+CI+%E2%80%9923%2E+ACM%2E)
11. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with confidence: Uncertainty quantification for black-box large language models. Preprint, arXiv:2305.19187. [↗](https://arxiv.org/abs/2305.19187)
12. Mengjie Zhao and Hinrich Schütze. 2021. Discrete and soft prompting for multilingual models. arXiv preprint arXiv:2109.03630. [↗](https://arxiv.org/abs/2109.03630)
13. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. 2022. Quantifying and alleviating political bias in language models. Artificial Intelligence, 304:103654. [↗](https://doi.org/10.1016/j.artint.2021.103654)
14. Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2024. More human than human: Measuring chatgpt political bias. Public Choice, 198(1):3–23. [↗](https://doi.org/10.1007/s11127-023-01097-2)
15. Tarek Naous, Michael J Ryan, and Wei Xu. 2023. [Having beer after prayer? measuring cultural bias in large language models](https://md.research.elamur.ai/free/en/papers/naous-2023-having-beer-after-prayer-measuring/). arXiv preprint arXiv:2305.14456. [↗](https://arxiv.org/abs/2305.14456)
16. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. [Camel tools: An open source python toolkit for arabic natural language processing](https://md.research.elamur.ai/free/en/papers/obeid-2020-camel-tools-open-source-python/). In Proceedings of the twelfth language resources and evaluation conference, pages 7022–7032.
17. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Preprint, arXiv:2203.02155. [↗](https://arxiv.org/abs/2203.02155)
18. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.2305.18290)
19. Aida Ramezani and Yang Xu. 2023. [Knowledge of cultural moral norms in large language models](https://md.research.elamur.ai/free/en/papers/ramezani-2023-knowledge-cultural-moral-norms-large/). Preprint, arXiv:2306.01857. [↗](https://arxiv.org/abs/2306.01857)
20. Karolina Stańczak, Sagnik Ray Choudhury, Tiago Pimentel, Ryan Cotterell, and Isabelle Augenstein. 2023. Quantifying gender bias towards politicians in cross-lingual language models. Preprint, arXiv:2104.07505. [↗](https://arxiv.org/abs/2104.07505)
21. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2024. [Decodingtrust: A comprehensive assessment of trustworthiness in gpt models](https://md.research.elamur.ai/free/en/papers/wang-2023-decodingtrust-comprehensive-assessment-trustworthiness-gpt/). Preprint, arXiv:2306.11698. [↗](https://arxiv.org/abs/2306.11698)
22. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R Lyu. 2023. All languages matter: On the multilingual safety of large language models. arXiv preprint arXiv:2310.00905. [↗](https://arxiv.org/abs/2310.00905)

*Updated 1 November 2026*