--- title: "A Multilingual Red Teaming–Driven Safety Analysis of LLMs — Pandeiro 2026" description: "The paper examines how well large language models resist harmful requests in more than one language, with a focus on English and Portuguese. 15 references." authors: ["Patrícia Pandeiro", "Vera Cabarrão", "Helena Moniz"] year: 2026 venue: "Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)" source: "https://aclanthology.org/2026.eamt-1.46/" licence: "CC BY-ND 4.0 (EAMT)" licence_url: "https://creativecommons.org/licenses/by-nd/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/pandeiro-2026-multilingual-red-teaming-driven-safety/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pandeiro-2026-multilingual-red-teaming-driven-safety/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / A Multilingual Red Teaming–Driven Safety Analysis of LLMs # A Multilingual Red Teaming–Driven Safety Analysis of LLMs **Authors:** Patrícia Pandeiro, Vera Cabarrão and [Helena Moniz](https://md.research.elamur.ai/free/en/authors/helena-moniz/) · **Year:** 2026 · **Venue:** Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1) · **Licence:** [CC BY-ND 4.0 (EAMT)](https://creativecommons.org/licenses/by-nd/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.eamt-1.46/) · [PDF](https://aclanthology.org/2026.eamt-1.46.pdf) ## Summary The paper examines how well large language models resist harmful requests in more than one language, with a focus on English and Portuguese. The authors built a bilingual test set of 170 prompts per language. It combines handpicked items from two public red teaming collections with a smaller in-house set that imitates ordinary users probing the model, for example through euphemisms. Prompts were machine-translated into Portuguese with GPT-4o and then checked by a native speaker. A single annotator labelled every response. Results are reported as a binary safety percentage and as a finer-grained safety score built from +1/−1 categories. Three experiments were run: a five-model comparison, tests with guardrailing and content-moderation system prompts, and a study of how token limits and temperature affect TowerLLM 3.0 models. TowerLLM variants outperformed GPT-4o on safety. Portuguese results were consistently weaker than English, and LLM-based content moderation performed poorly. The work argues for multilingual safety testing and suggests red teaming as a new professional role for translators. ## Key points - **Data:** 72 requests come from Haize Labs' Redteaming Resistance Leaderboard dataset, 50 from Babelscape's ALERT, and 48 were newly written, giving 340 requests in total across EN and PT. Twenty of them are two-part prompts. The set is released as `Unbabel/Multilingual-Red-Teaming` on Hugging Face. - **Model comparison:** Five models were tested: GPT-4o, Vesuvius 3.0, Sugarloaf 3.1, Sugarloaf 4.0 and Vesuvius 4.0. Sugarloaf 3.1 was safest overall, while Vesuvius 4.0 led on the Portuguese safety score. Both beat GPT-4o. - **Guardrailing prompt:** Mistral AI's Safe Prompt improved safety, with the clearest gains in Portuguese. - **Content moderation:** Mistral's Self-reflection Prompt and a Llama Guard 3-8B prompt both gave substandard harm classification. GPT-4o did best on this task, and Vesuvius 3.0 surprisingly outperformed the newer Sugarloaf 3.1. - **Decoding settings:** TowerLLM 3.0 models (Anthill, Sugarloaf, Vesuvius) were tested at 512/1000/2048 tokens with temperature 0 or 1. The 1000-token, temperature-1 setting was safest, while 512 tokens was markedly poor. Higher temperature reduced harmful content but caused more failed requests. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) ## Cite Cite as: Patrícia Pandeiro, Vera Cabarrão and Helena Moniz. 2026. A Multilingual Red Teaming–Driven Safety Analysis of LLMs. Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{pandeiro2026multilingual, title = {{A Multilingual Red Teaming–Driven Safety Analysis of LLMs}}, author = {Patrícia Pandeiro, Vera Cabarrão and Helena Moniz}, booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)}, year = {2026}, url = {https://aclanthology.org/2026.eamt-1.46/} } ``` - [← previous A Method for Learning Large-Scale Computational Construction Grammars from Semantically Annotated Corpora](https://md.research.elamur.ai/free/en/papers/eecke-2026-method-learning-large-scale-computational/) - [next → A Novel Dataset and Three Ways to Approach Automatic Metaphor Detection in German Religious Online Forums](https://md.research.elamur.ai/free/en/papers/reimann-2026-novel-dataset-three-ways-approach/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) ## References 15 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Alves, Duarte M., José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza & André F. T. Martins. (2024). Tower: An open Multilingual Large Language Model for Translation-Related Tasks.… [our card](https://md.research.elamur.ai/free/en/papers/alves-2024-tower-open-multilingual-large-language/) 2. Angwin, J., Jeff Larson, Surya Mattu, & Lauren Kirchner. (2016). Machine Bias. ProPublica. Retrieved March 31, 2025, from [https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing) [↗](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing) 3. Ayyamperumal, S. G., & Limin Ge. (2024). Current state of LLM Risks and AI Guardrails. arXiv. https://doi.org/10.48550/arXiv.2406.12934 — [doi:10.48550/arxiv.2406.12934](https://doi.org/10.48550/arxiv.2406.12934) [↗](https://doi.org/10.48550/arXiv.2406.12934) 4. Babelscape. (2024). ALERT: A Comprehensive Benchmark for Assessing Large Language Models’ Safety through Red Teaming. GitHub. Retrieved November 5, 2024, from [https://github.com/Babelscape/ALERT](https://github.com/Babelscape/ALERT) [↗](https://github.com/Babelscape/ALERT) 5. Content moderation with self-reflection. (n.d.). Mistral AI. Retrieved May 13, 2025, from [https://docs.mistral.ai/capabilities/guardrailing/\#conten](https://docs.mistral.ai/capabilities/guardrailing/\#conten) t-moderation-with-self-reflection European Union. (2024). Regulation - EU - 2024/1689 -EN - EUR-Lex. EUR-Lex. Retrieved January 28, 2026, from https://e… [↗](https://docs.mistral.ai/capabilities/guardrailing/\#conten) 6. Haize Labs. (2024). Redteaming Resistance Benchmark. GitHub. Retrieved November 5, 2024, from [https://github.com/haizelabs/redteaming-resistance-benchmark](https://github.com/haizelabs/redteaming-resistance-benchmark) [↗](https://github.com/haizelabs/redteaming-resistance-benchmark) 7. Heikkilä, M. (2022). Dutch scandal serves as a warning for Europe over risks of using algorithms. POLITICO. Retrieved March 31, 2025, from [https://www.politico.eu/article/dutch-scandal-serves-as-a-warning-for-europe-over-risks-of-using-algorithms/](https://www.politico.eu/article/dutch-scandal-serves-as-a-warning-for-europe-over-risks-of-using-algorithms/) [↗](https://www.politico.eu/article/dutch-scandal-serves-as-a-warning-for-europe-over-risks-of-using-algorithms/) 8. Hello GPT-4o. (2024). OpenAI. Retrieved March 5, 2026, from [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/) [↗](https://openai.com/index/hello-gpt-4o/) 9. Moniz, H., & Carla Parra Escartín. (Eds.). (2023). Towards Responsible Machine Translation: Ethical and Legal Considerations in Machine Translation (Vol. 4). Springer International Publishing. https://doi.org/10.1007/978-3-031-14689-3 — [doi:10.1007/978-3-031-14689-3](https://doi.org/10.1007/978-3-031-14689-3) [↗](https://doi.org/10.1007/978-3-031-14689-3) 10. Moorkens, J., Andy Way, & Séamus Lankford. (2024). Automating Translation (1st ed.). Routledge. https://doi.org/10.4324/9781003381280 — [doi:10.4324/9781003381280](https://doi.org/10.4324/9781003381280) [↗](https://doi.org/10.4324/9781003381280) 11. Meta Llama. (n.d.). Llama Guard 3-8B Model Card. GitHub. Retrieved February 15, 2025, from [https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard3/8B/MODEL_CARD.md](https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard3/8B/MODEL_CARD.md) [↗](https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard3/8B/MODEL_CARD.md) 12. Obermeyer, Z., Brian Powers, Christine Vogeli, & Sendhil Mullainathan. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342 — [doi:10.1126/science.aax2342](https://doi.org/10.1126/science.aax2342) [↗](https://doi.org/10.1126/science.aax2342) 13. System prompt to enforce guardrails. (n.d.). Mistral AI. Retrieved May 13, 2025, from [https://docs.mistral.ai/capabilities/guardrailing/\#system](https://docs.mistral.ai/capabilities/guardrailing/\#system) -prompt-to-enforce-guardrails Rei, Ricardo, Nuno M. Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, & André F. T. Martins. (2025). Tow… [↗](https://docs.mistral.ai/capabilities/guardrailing/\#system) 14. The White House. (2023). Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. The White House. Retrieved August 14, 2025, from [https://bidenwhitehouse.archives.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-tru…](https://bidenwhitehouse.archives.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-tru…) [↗](https://bidenwhitehouse.archives.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-tru…) 15. Zenko, M. (2015). Red team: How to succeed by thinking like the enemy. Basic Books. [find](https://scholar.google.com/scholar?q=Zenko%2C+M%2E+%282015%29%2E+Red+team%3A+How+to+succeed+by+thinking+like+the+enemy%2E+Basic+Books%2E)