---
title: "Multilingual Disparities in LLM-Based Safety Judgments:"
description: "The paper tests a common assumption about multilingual LLM judges: that the same content gets the same verdict whatever… 23 references."
authors: ["Songjiang Liu", "Riley Grossman", "Mike Smith", "Cristian Borcea", "Yi Chen"]
year: 2026
venue: "Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)"
source: "https://aclanthology.org/2026.mellm-1.26/"
doi: "10.18653/v1/2026.mellm-1.26"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2026-multilingual-disparities-llm-based-safety/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2026-multilingual-disparities-llm-based-safety/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/37/) / Multilingual Disparities in LLM-Based Safety Judgments: Evidence from Brand Safety Applications

# Multilingual Disparities in LLM-Based Safety Judgments: Evidence from Brand Safety Applications

**Authors:** Songjiang Liu, Riley Grossman, Mike Smith, [Cristian Borcea](https://md.research.elamur.ai/free/en/authors/cristian-borcea/) and [Yi Chen](https://md.research.elamur.ai/free/en/authors/yi-chen/) · **Year:** 2026 · **Venue:** [Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)](https://md.research.elamur.ai/free/en/venues/mellm/) ([Workshop on Multilinguality in the Era of Large Language Models 2026](https://md.research.elamur.ai/free/en/venues/mellm/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.mellm-1.26/) · [PDF](https://aclanthology.org/2026.mellm-1.26.pdf) · [DOI 10.18653/v1/2026.mellm-1.26](https://doi.org/10.18653/v1/2026.mellm-1.26) · [OpenAlex](https://openalex.org/W7166852367)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper tests a common assumption about multilingual LLM judges: that the same content gets the same verdict whatever its language. The authors study this in brand safety, where automated ratings decide whether advertisers place ads next to news. That decision affects publisher income and advertiser reputation.

They collected Euronews stories that were professionally translated into several languages. This gave them parallel content without relying on machine translation. Each language version was scored three times by MiniMax-M2.5 on the 15 GARM categories, with a safety-floor verdict and a graded suitability level (0–3). A smaller sample was re-rated with two other models to check robustness.

Disagreement across languages turned out to be very common. A fixed-effects ANOVA showed that language significantly shifts suitability scores even when article, category and run are controlled. With the primary model, some languages were systematically judged more harshly and others more leniently. The direction of these shifts changed with the model, but significant language effects held for all three. The work shows that LLM safety ratings can treat equivalent content unequally, with consequences for content moderation more broadly.

## Key points
- **Dataset and resource:** 10,467 Euronews articles with 70,041 language versions in 13 languages. English has the most versions (9,666) and Arabic the fewest (2,374). Ukrainian is not covered, though Russian is. The ratings benchmark, including prompts, URLs and reasoning traces, is released at github.com/sl947/langdiff.
- **How common disagreement is:** about 25.8% of article–category pairs show any cross-language disagreement. Among pairs where at least one version received a non-zero risk, the figure rises to 96.8%. Even rare categories such as Tobacco/Vaping and Online Piracy reach 85–98% under this condition.
- **Statistical tests:** in the ANOVA, the language effect is significant (F = 8.115, p \< 0.001), and so is the language × category interaction. Repeated runs show no significant effect (p = 0.374).
- **Direction of the bias in MiniMax-M2.5:**
  - Rated more strictly: English, German and French content.
  - Rated more leniently: Polish, Hungarian, Greek, Turkish and Persian content.
  - Robustness checks used gemini-3.1-flash-lite-preview and deepseek-v4-pro; language effects persisted, but directional patterns differed by model.
- **Recommendations:**
  - Evaluate systems on aligned multilingual benchmarks, reporting results by language and category.
  - Send high-disagreement cases to human review.
  - Make models and prompts transparent to advertisers and publishers.
  - Let publishers contest ratings.
  - Causal mechanisms (cultural bias vs. data imbalance) are left open.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Indiana University Bloomington](https://md.research.elamur.ai/free/en/universities/indiana-university-bloomington/) — University, Bloomington, United States · 37 works · 3,685 citations
- [New Jersey Institute of Technology](https://md.research.elamur.ai/free/en/universities/new-jersey-institute-of-technology/) — University, Newark, United States · 15 works · 1,333 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (2): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: Songjiang Liu, Riley Grossman, Mike Smith, Cristian Borcea and Yi Chen. 2026. Multilingual Disparities in LLM-Based Safety Judgments: Evidence from Brand Safety Applications. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2026multilingual,
  title = {{Multilingual Disparities in LLM-Based Safety Judgments: Evidence from Brand Safety Applications}},
  author = {Songjiang Liu, Riley Grossman, Mike Smith, Cristian Borcea and Yi Chen},
  booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mellm-1.26},
  url = {https://doi.org/10.18653/v1/2026.mellm-1.26}
}
```

- [← previous Multilingual Chain-of-Thought Compression via Cross-Lingual Distillation](https://md.research.elamur.ai/free/en/papers/wan-2026-multilingual-chain-thought-compression-cross/)
- [next → Multilingual KokoroChat: A Multi-LLM Ensemble Translation Method for Creating a Multilingual Counseling Dialogue Dataset](https://md.research.elamur.ai/free/en/papers/suzuki-2026-multilingual-kokorochat-multi-llm-ensemble/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/37/)

## References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Isaac Chung and Linda Freienthal. 2026. Cross-lingual stability of LLM judges under controlled generation: Evidence from Finno-Ugric languages. In Proceedings of the First Workshop on Multilingual Multicultural Evaluation, pages 133–148, Rabat, Morocco. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Isaac+Chung+and+Linda+Freienthal%2E+2026%2E+Cross%2Dlingual+stability+of+LLM+judges+under+controlled+generation%3A+Evidence+from+Finno%2DUgric+languages%2E+In+Proceedings+of+the+First+Workshop+on+Multilingual+Mul)
2. Neil Fasching and Yphtach Lelkes. 2025. Model-dependent moderation: Inconsistencies in hate speech detection across LLM-based systems. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22271–22285, Vienna, Austria. Association for Computational Linguistics. on Empirical M… [find](https://scholar.google.com/scholar?q=Neil+Fasching+and+Yphtach+Lelkes%2E+2025%2E+Model%2Ddependent+moderation%3A+Inconsistencies+in+hate+speech+detection+across+LLM%2Dbased+systems%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL)
3. Dev Pragad. 2024. Don’t cancel brand safety – improve it. AdExchanger. [find](https://scholar.google.com/scholar?q=Dev+Pragad%2E+2024%2E+Don%E2%80%99t+cancel+brand+safety+%E2%80%93+improve+it%2E+AdExchanger%2E)
4. Allison Schiff. 2025. Ai is helping brand safety break free from blocklists. AdExchanger. [find](https://scholar.google.com/scholar?q=Allison+Schiff%2E+2025%2E+Ai+is+helping+brand+safety+break+free+from+blocklists%2E+AdExchanger%2E)
5. Michael Smith, Riley Grossman, Antonio Torres-Aguero, Pritam Sen, Cristian Borcea, and Yi Chen. 2026. Inconsistencies in classification of online news articles: A call for common standards in brand safety services. Preprint, arXiv:2601.01303. — [arXiv:2601.01303](https://arxiv.org/abs/2601.01303) [find](https://scholar.google.com/scholar?q=Michael+Smith%2C+Riley+Grossman%2C+Antonio+Torres%2DAguero%2C+Pritam+Sen%2C+Cristian+Borcea%2C+and+Yi+Chen%2E+2026%2E+Inconsistencies+in+classification+of+online+news+articles%3A+A+call+for+common+standards+in+brand+sa)
6. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9):pgae346. [find](https://scholar.google.com/scholar?q=Yan+Tao%2C+Olga+Viberg%2C+Ryan+S+Baker%2C+and+Ren%C3%A9+F+Kizilcec%2E+2024%2E+Cultural+bias+and+cultural+alignment+of+large+language+models%2E+PNAS+Nexus%2C+3%289%29%3Apgae346%2E)
7. Soheil Feizi, MohammadTaghi Hajiaghayi, Keivan Rezaei, and Suho Shin. 2024. Online advertisements with llms: Opportunities and challenges. Preprint, arXiv:2311.07601. — [arXiv:2311.07601](https://arxiv.org/abs/2311.07601) [find](https://scholar.google.com/scholar?q=Soheil+Feizi%2C+MohammadTaghi+Hajiaghayi%2C+Keivan+Rezaei%2C+and+Suho+Shin%2E+2024%2E+Online+advertisements+with+llms%3A+Opportunities+and+challenges%2E+Preprint%2C+arXiv%3A2311%2E07601%2E+%E2%80%94+%5BarXiv%3A2311%2E07601%5D%28https%3A%2F%2Farxi)
8. Xiyan Fu and Wei Liu. 2025. How reliable is multilingual LLM-as-a-judge? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11040–11053, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiyan+Fu+and+Wei+Liu%2E+2025%2E+How+reliable+is+multilingual+LLM%2Das%2Da%2Djudge%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2025%2C+pages+11040%E2%80%9311053%2C+Suzhou%2C+China%2E+Association+for+Com)
9. Jiahui He, Yiluo Wei, and Gareth Tyson. 2026. Enhancing content moderation with llms: A reddit case study on evaluating and refining human decisions. In Proceedings of the ACM Web Conference 2026, WWW ’26, page 9823–9834, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Jiahui+He%2C+Yiluo+Wei%2C+and+Gareth+Tyson%2E+2026%2E+Enhancing+content+moderation+with+llms%3A+A+reddit+case+study+on+evaluating+and+refining+human+decisions%2E+In+Proceedings+of+the+ACM+Web+Conference+2026%2C+WWW)
10. Alice Hunsberger. 2025. How to use llms for content moderation. Musubi. [find](https://scholar.google.com/scholar?q=Alice+Hunsberger%2E+2025%2E+How+to+use+llms+for+content+moderation%2E+Musubi%2E)
11. Integral Ad Science. 2024. Our brand safety and suitability overview. Published March 6, 2024. [find](https://scholar.google.com/scholar?q=Integral+Ad+Science%2E+2024%2E+Our+brand+safety+and+suitability+overview%2E+Published+March+6%2C+2024%2E)
12. Ross W. Johnson, Clay Voorhees, and Farnoosh Kho-dakarami. 2023. Is your brand protected? assessing brand safety risks in digital campaigns. Journal of Advertising Research, 63(3):205–221. [find](https://scholar.google.com/scholar?q=Ross+W%2E+Johnson%2C+Clay+Voorhees%2C+and+Farnoosh+Kho%2Ddakarami%2E+2023%2E+Is+your+brand+protected%3F+assessing+brand+safety+risks+in+digital+campaigns%2E+Journal+of+Advertising+Research%2C+63%283%29%3A205%E2%80%93221%2E)
13. Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha. 2024. Llm-mod: Can large language models assist content moderation? In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA ’24, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Mahi+Kolla%2C+Siddharth+Salunkhe%2C+Eshwar+Chandrasekharan%2C+and+Koustuv+Saha%2E+2024%2E+Llm%2Dmod%3A+Can+large+language+models+assist+content+moderation%3F+In+Extended+Abstracts+of+the+CHI+Conference+on+Human+Facto)
14. Adi Levi, Or Levi, Sardhendu Mishra, and Jonathan Morra. 2025. Ai vs. human moderators: A comparative evaluation of multimodal llms in content moderation for brand safety. Preprint, arXiv:2508.05527. — [arXiv:2508.05527](https://arxiv.org/abs/2508.05527) [find](https://scholar.google.com/scholar?q=Adi+Levi%2C+Or+Levi%2C+Sardhendu+Mishra%2C+and+Jonathan+Morra%2E+2025%2E+Ai+vs%2E+human+moderators%3A+A+comparative+evaluation+of+multimodal+llms+in+content+moderation+for+brand+safety%2E+Preprint%2C+arXiv%3A2508%2E05527%2E+)
15. Marco Lewis. 2022. The netzdg and the avia law: How two different legal systems created two different outcomes from similar laws. Wis. Int’l LJ, 40:491. [find](https://scholar.google.com/scholar?q=Marco+Lewis%2E+2022%2E+The+netzdg+and+the+avia+law%3A+How+two+different+legal+systems+created+two+different+outcomes+from+similar+laws%2E+Wis%2E+Int%E2%80%99l+LJ%2C+40%3A491%2E)
16. Bibek Upadhayay and Vahid Behzadan. 2025. X-guard: Multilingual guard agent for content moderation. In Proceedings of the First Workshop on LLM Security (LLMSEC), pages 54–86, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bibek+Upadhayay+and+Vahid+Behzadan%2E+2025%2E+X%2Dguard%3A+Multilingual+guard+agent+for+content+moderation%2E+In+Proceedings+of+the+First+Workshop+on+LLM+Security+%28LLMSEC%29%2C+pages+54%E2%80%9386%2C+Vienna%2C+Austria%2E+Associa)
17. Yash Vekaria, Rishab Nithyanand, and Zubair Shafiq. 2024. The inventory is dark and full of misinformation: Understanding ad inventory pooling in the ad-tech supply chain. In 2024 IEEE Symposium on Security and Privacy (SP), pages 1590–1608. [find](https://scholar.google.com/scholar?q=Yash+Vekaria%2C+Rishab+Nithyanand%2C+and+Zubair+Shafiq%2E+2024%2E+The+inventory+is+dark+and+full+of+misinformation%3A+Understanding+ad+inventory+pooling+in+the+ad%2Dtech+supply+chain%2E+In+2024+IEEE+Symposium+on+Se)
18. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. 2024. All languages matter: On the multilingual safety of LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5865– 5877, Bangkok, Thailand. Association for Computatio… [find](https://scholar.google.com/scholar?q=Wenxuan+Wang%2C+Zhaopeng+Tu%2C+Chang+Chen%2C+Youliang+Yuan%2C+Jen%2Dtse+Huang%2C+Wenxiang+Jiao%2C+and+Michael+Lyu%2E+2024%2E+All+languages+matter%3A+On+the+multilingual+safety+of+LLMs%2E+In+Findings+of+the+Association+for+)
19. World Federation of Advertisers. 2024. Wfa discontinues garm. Published August 9, 2024. [find](https://scholar.google.com/scholar?q=World+Federation+of+Advertisers%2E+2024%2E+Wfa+discontinues+garm%2E+Published+August+9%2C+2024%2E)
20. World Population Review. 2026. Alcohol consumption by country 2026. Accessed: 05-01-2026. [find](https://scholar.google.com/scholar?q=World+Population+Review%2E+2026%2E+Alcohol+consumption+by+country+2026%2E+Accessed%3A+05%2D01%2D2026%2E)
21. Zheng Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen Bach, and Julia Kreutzer. 2025. [The state of multilingual LLM safety research: From measuring the language gap to mitigating it](https://md.research.elamur.ai/free/en/papers/yong-2025-state-multilingual-llm-safety-research/). In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15845–15860, Suzhou,…
22. ZEFR. 2026. Zefr is the first third-party partner to receive mrc accreditation for content-level brand safety and suitability reporting on youtube. ZEFR Press Release. [find](https://scholar.google.com/scholar?q=ZEFR%2E+2026%2E+Zefr+is+the+first+third%2Dparty+partner+to+receive+mrc+accreditation+for+content%2Dlevel+brand+safety+and+suitability+reporting+on+youtube%2E+ZEFR+Press+Release%2E)
23. Sarah Masud, Sahajpreet Singh, Viktor Hangya, Alexander Fraser, and Tanmoy Chakraborty. 2024. Hate personified: Investigating the role of LLMs in content moderation. In Proceedings of the 2024 Conference [find](https://scholar.google.com/scholar?q=Sarah+Masud%2C+Sahajpreet+Singh%2C+Viktor+Hangya%2C+Alexander+Fraser%2C+and+Tanmoy+Chakraborty%2E+2024%2E+Hate+personified%3A+Investigating+the+role+of+LLMs+in+content+moderation%2E+In+Proceedings+of+the+2024+Confere)

*Updated 1 November 2026*