---
title: "Code-Switching Red-Teaming — paper by Yoo et al. (2025)"
description: "The authors propose CSRT, a cheap and fully automated framework for building code-switched… 61 references, cited by 19 papers here."
authors: ["Haneul Yoo", "Yongjin Yang", "Hwaran Lee"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.657/"
doi: "10.18653/v1/2025.acl-long.657"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yoo-2025-code-switching-red-teaming-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yoo-2025-code-switching-red-teaming-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding

# Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding

**Authors:** [Haneul Yoo](https://md.research.elamur.ai/free/en/authors/haneul-yoo/), [Yongjin Yang](https://md.research.elamur.ai/free/en/authors/yongjin-yang/) and [Hwaran Lee](https://md.research.elamur.ai/free/en/authors/hwaran-lee/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.657/) · [PDF](https://aclanthology.org/2025.acl-long.657.pdf) · [DOI 10.18653/v1/2025.acl-long.657](https://doi.org/10.18653/v1/2025.acl-long.657) · [OpenAlex](https://openalex.org/W4412945237)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [19](https://md.research.elamur.ai/free/en/papers/yoo-2025-code-switching-red-teaming-llm/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Multilingual LLMs tend to be less safe in non-English languages, but no one had tested whether code-switching, the natural habit of mixing languages within one utterance, can also break safety guardrails. The authors propose CSRT, a cheap and fully automated framework for building code-switched red-teaming prompts. They use the 315 adversarial prompts of MultiJail, available as parallel versions in 10 languages, and ask gpt-4o to merge each set of translations into a single mixed-language query. The resulting prompts are used to evaluate ten open and proprietary LLMs on two axes: safety (attack success rate and refusal rate) and how well each model understands the mixed input (comprehension).

Code-switched prompts produced 46.7% more successful attacks than the same prompts in English alone. Token-level mixing proved more harmful than sentence-level mixing or a single non-English language. Using more languages, and especially low-resource ones, made attacks more successful. From this, the authors argue that safety alignment in current LLMs is unintentionally tied to how well-resourced a language is. This matters for any low-resource language community.

## Key points
- **Dataset:** The seed data is MultiJail (315 prompts). The languages are English plus three resource tiers:
  - high: Zh, It, Vi
  - mid: Ar, Ko, Th
  - low: Bn, Sw, Jv
- **Language mix:** Each CSRT query mixes on average 7.83 ± 2.19 languages. English makes up about 20.9% of tokens, and each other language makes up roughly 7.5–10.4%.
- **Attack ranking:** CSRT beats the Sandwich Attack (sentence-level mixing), which in turn beats monolingual non-English attacks from MultiJail.
- **Comprehension:**
  - Qwen 1.5 and Claude 3 understood code-switched text best.
  - On non-adversarial MMMLU-based queries, comprehension stayed above 90% for high- and mid-resource languages.
  - It fell to about 61–80% for Bengali and 77–88% for code-switched input.
- **Scaling and availability:**
  - The method also works on monolingual seed data without human translation.
  - Ablation results held on a set of more than 16K samples.
  - Code and data are released at github.com/haneul-yoo/csrt.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Korea Advanced Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/korea-advanced-institute-of-science-and-technology/) — University, Daejeon, South Korea · 51 works · 3,167 citations
- [NAVER Cloud](https://md.research.elamur.ai/free/en/companies/naver-cloud/) — Company, Seongnam-si, South Korea · 15 works · 10,707 citations
- [Sogang University](https://md.research.elamur.ai/free/en/universities/sogang-university/) — University, Seoul, South Korea · 3 works · 55 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 517 works

## Cite

Cite as: Haneul Yoo, Yongjin Yang and Hwaran Lee. 2025. Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yoo2025code,
  title = {{Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding}},
  author = {Haneul Yoo, Yongjin Yang and Hwaran Lee},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.657},
  url = {https://doi.org/10.18653/v1/2025.acl-long.657}
}
```

- [← previous Binary Classifier Optimization for Large Language Model Alignment](https://md.research.elamur.ai/free/en/papers/jung-2025-binary-classifier-optimization-large-language/)
- [next → Distillation Scaling Laws](https://md.research.elamur.ai/free/en/papers/busbridge-2025-distillation-scaling-laws/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

61 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. [LinCE: A centralized benchmark for linguistic code-switching evaluation](https://md.research.elamur.ai/free/en/papers/aguilar-2020-lince-centralized-benchmark-linguistic-code/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1803–1813, Marseille, France. European Language Resources Association.
2. Gabriel Alon and Michael Kamfonas. 2023. [Detecting language model attacks with perplexity](https://md.research.elamur.ai/free/en/papers/alon-2023-detecting-language-model-attacks-perplexity/).
3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
4. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. [The belebele benchmark: a parallel reading comprehension dataset in 122 language variants](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/). In Proceedings of the 62nd Annual M…
5. Anab Maulana Barik, Rahmad Mahendra, and Mirna Adriani. 2019. Normalization of Indonesian-English code-mixed Twitter data. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 417–424, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anab+Maulana+Barik%2C+Rahmad+Mahendra%2C+and+Mirna+Adriani%2E+2019%2E+Normalization+of+Indonesian%2DEnglish+code%2Dmixed+Twitter+data%2E+In+Proceedings+of+the+5th+Workshop+on+Noisy+User%2Dgenerated+Text+%28W%2DNUT+2019%29%2C)
6. Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. Red… — [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/)
7. Alejandro E. Brice and Roanne G. Brice. 2009. Language Development: Monolingual and Bilingual Acquisition. Pearson. [find](https://scholar.google.com/scholar?q=Alejandro+E%2E+Brice+and+Roanne+G%2E+Brice%2E+2009%2E+Language+Development%3A+Monolingual+and+Bilingual+Acquisition%2E+Pearson%2E)
8. Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John Philip McCrae. 2020. [Corpus creation for sentiment analysis in code-mixed Tamil-English text](https://md.research.elamur.ai/free/en/papers/chakravarthi-2020-corpus-creation-sentiment-analysis-code/). In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collabor…
9. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2025. [Jailbreaking Black Box Large Language Models in Twenty Queries](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/) . In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 23–42, Los Alamitos, CA, USA. IEEE Computer Society.
10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+X)
11. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try ARC, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/).
12. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. [Multilingual jailbreak challenges in large language models](https://md.research.elamur.ai/free/en/papers/deng-2023-multilingual-jailbreak-challenges-large-language/). In The Twelfth International Conference on Learning Representations.
13. Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. 2024. A wolf in sheep‘s clothing: Generalized nested jailbreak prompts can fool large language models easily. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computationa… [find](https://scholar.google.com/scholar?q=Peng+Ding%2C+Jun+Kuang%2C+Dan+Ma%2C+Xuezhi+Cao%2C+Yunsen+Xian%2C+Jiajun+Chen%2C+and+Shujian+Huang%2E+2024%2E+A+wolf+in+sheep%E2%80%98s+clothing%3A+Generalized+nested+jailbreak+prompts+can+fool+large+language+models+easily%2E+In+)
14. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wort… [find](https://scholar.google.com/scholar?q=Deep+Ganguli%2C+Liane+Lovitt%2C+Jackson+Kernion%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Saurav+Kadavath%2C+Ben+Mann%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Kamal+Ndousse%2C+Andy+Jones%2C+Sam+Bowman%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+D)
15. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. [RealToxicityPrompts: Evaluating neural toxic degeneration in language models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti…
16. Brian Goldstein and Kathryn Kohnert. 2005. Speech, language, and hearing in developing bilingual children. Language, Speech, and Hearing Services in Schools, 36(3):264–267. [find](https://scholar.google.com/scholar?q=Brian+Goldstein+and+Kathryn+Kohnert%2E+2005%2E+Speech%2C+language%2C+and+hearing+in+developing+bilingual+children%2E+Language%2C+Speech%2C+and+Hearing+Services+in+Schools%2C+36%283%29%3A264%E2%80%93267%2E)
17. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
18. Vera F. Gutierrez-Clellen. 1999. Language choice in intervention with bilingual children. American Journal of Speech-Language Pathology, 8(4):291–302. [find](https://scholar.google.com/scholar?q=Vera+F%2E+Gutierrez%2DClellen%2E+1999%2E+Language+choice+in+intervention+with+bilingual+children%2E+American+Journal+of+Speech%2DLanguage+Pathology%2C+8%284%29%3A291%E2%80%93302%2E)
19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
20. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. [Llama guard: LLM-based input-output safeguard for human-AI conversations](https://md.research.elamur.ai/free/en/papers/inan-2023-llama-guard-llm-based-input/).
21. Devansh Jain, Priyanshu Kumar, Samuel Gehman, Xuhui Zhou, Thomas Hartvigsen, and Maarten Sap. 2024. PolygloToxicityPrompts: Multilingual evaluation of neural toxic degeneration in large language models. In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Devansh+Jain%2C+Priyanshu+Kumar%2C+Samuel+Gehman%2C+Xuhui+Zhou%2C+Thomas+Hartvigsen%2C+and+Maarten+Sap%2E+2024%2E+PolygloToxicityPrompts%3A+Multilingual+evaluation+of+neural+toxic+degeneration+in+large+language+model)
22. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. [Baseline defenses for adversarial attacks against aligned language models](https://md.research.elamur.ai/free/en/papers/jain-2023-baseline-defenses-adversarial-attacks-against/).
23. Sai Muralidhar Jayanthi, Kavya Nerella, Khyathi Raghavi Chandu, and Alan W Black. 2021. CodemixedNLP: An extensible and open NLP toolkit for code-mixing. In Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, pages 113–118, Online. Association for Computationa… [find](https://scholar.google.com/scholar?q=Sai+Muralidhar+Jayanthi%2C+Kavya+Nerella%2C+Khyathi+Raghavi+Chandu%2C+and+Alan+W+Black%2E+2021%2E+CodemixedNLP%3A+An+extensible+and+open+NLP+toolkit+for+code%2Dmixing%2E+In+Proceedings+of+the+Fifth+Workshop+on+Comput)
24. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
25. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
26. Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. 2024b. [ArtPrompt: ASCII art-based jailbreak attacks against aligned LLMs](https://md.research.elamur.ai/free/en/papers/jiang-2024-artprompt-ascii-art-based-jailbreak/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag…
27. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
28. Simran Khanuja, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2020. A new dataset for natural language inference from code-mixed conversations. In Proceedings of the 4th Workshop on Computational Approaches to Code Switching, pages 9–16, Marseille, France. European Language Resources A… [find](https://scholar.google.com/scholar?q=Simran+Khanuja%2C+Sandipan+Dandapat%2C+Sunayana+Sitaram%2C+and+Monojit+Choudhury%2E+2020%2E+A+new+dataset+for+natural+language+inference+from+code%2Dmixed+conversations%2E+In+Proceedings+of+the+4th+Workshop+on+Comp)
29. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). In The Twelfth International Conference on Learning Representations.
30. Kathryn Kohnert, Dongsun Yim, Kelly Nett, Pui Fong Kan, and Lillian Duran. 2005. Intervention with linguistically diverse preschool children. Language, Speech, and Hearing Services in Schools, 36(3):251– 263. [find](https://scholar.google.com/scholar?q=Kathryn+Kohnert%2C+Dongsun+Yim%2C+Kelly+Nett%2C+Pui+Fong+Kan%2C+and+Lillian+Duran%2E+2005%2E+Intervention+with+linguistically+diverse+preschool+children%2E+Language%2C+Speech%2C+and+Hearing+Services+in+Schools%2C+36%283%29%3A2)
31. Chengfei Li, Shuhao Deng, Yaoping Wang, Guangjing Wang, Yaguang Gong, Changbin Chen, and Jinfeng Bai. 2022. TALCS: An open-source Mandarin-English code-switching corpus and a speech recognition baseline. In Proc. [find](https://scholar.google.com/scholar?q=Chengfei+Li%2C+Shuhao+Deng%2C+Yaoping+Wang%2C+Guangjing+Wang%2C+Yaguang+Gong%2C+Changbin+Chen%2C+and+Jinfeng+Bai%2E+2022%2E+TALCS%3A+An+open%2Dsource+Mandarin%2DEnglish+code%2Dswitching+corpus+and+a+speech+recognition+baseli)
32. Interspeech 2022, pages 1741–1745. [find](https://scholar.google.com/scholar?q=Interspeech+2022%2C+pages+1741%E2%80%931745%2E)
33. Ying Li, Yue Yu, and Pascale Fung. 2012. A Mandarin-English code-switching corpus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2515– 2519, Istanbul, Turkey. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Ying+Li%2C+Yue+Yu%2C+and+Pascale+Fung%2E+2012%2E+A+Mandarin%2DEnglish+code%2Dswitching+corpus%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+pages+2515%E2%80%93+251)
34. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. [AutoDAN: Generating stealthy jailbreak prompts on aligned large language models](https://md.research.elamur.ai/free/en/papers/liu-2023-autodan-generating-stealthy-jailbreak-prompts/). In The Twelfth International Conference on Learning Representations.
35. Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Peng Xu, Yan Xu, Zihan Liu, Rita Frieske, Tiezheng Yu, Wenliang Dai, Elham J. Barezi, Qifeng Chen, Xiaojuan Ma, Bertram Shi, and Pascale Fung. 2022. ASCEND: A spontaneous Chinese-English dataset for code-switching in multi-turn conversation. In Proceed… [find](https://scholar.google.com/scholar?q=Holy+Lovenia%2C+Samuel+Cahyawijaya%2C+Genta+Winata%2C+Peng+Xu%2C+Yan+Xu%2C+Zihan+Liu%2C+Rita+Frieske%2C+Tiezheng+Yu%2C+Wenliang+Dai%2C+Elham+J%2E+Barezi%2C+Qifeng+Chen%2C+Xiaojuan+Ma%2C+Bertram+Shi%2C+and+Pascale+Fung%2E+2022%2E+ASC)
36. Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
37. Dau-Cheng Lyu, Tien-Ping Tan, Eng Siong Chng, and Haizhou Li. 2010. SEAME: a Mandarin-English code-switching speech corpus in south-east asia. In Proc. [find](https://scholar.google.com/scholar?q=Dau%2DCheng+Lyu%2C+Tien%2DPing+Tan%2C+Eng+Siong+Chng%2C+and+Haizhou+Li%2E+2010%2E+SEAME%3A+a+Mandarin%2DEnglish+code%2Dswitching+speech+corpus+in+south%2Deast+asia%2E+In+Proc%2E)
38. Interspeech 2010, pages 1986–1989. [find](https://scholar.google.com/scholar?q=Interspeech+2010%2C+pages+1986%E2%80%931989%2E)
39. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. [HarmBench: a standardized evaluation framework for automated red teaming and robust refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/). In Proceedings of the 41st International Con…
40. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… — [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
41. Li Nguyen and Christopher Bryant. 2020. CanVEC - the canberra Vietnamese-English code-switching natural speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4121–4129, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Li+Nguyen+and+Christopher+Bryant%2E+2020%2E+CanVEC+%2D+the+canberra+Vietnamese%2DEnglish+code%2Dswitching+natural+speech+corpus%2E+In+Proceedings+of+the+Twelfth+Language+Resources+and+Evaluation+Conference%2C+pages)
42. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… [find](https://scholar.google.com/scholar?q=OpenAI%2C+Josh+Achiam%2C+Steven+Adler%2C+Sandhini+Agarwal%2C+Lama+Ahmad%2C+Ilge+Akkaya%2C+Florencia+Leoni+Aleman%2C+Diogo+Almeida%2C+Janko+Altenschmidt%2C+Sam+Altman%2C+Shyamal+Anadkat%2C+Red+Avila%2C+Igor+Babuschkin%2C+Suchir)
43. Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, and Lizhuang Ma. 2024. CodeAttack: Revealing safety generalization challenges of large language models via code completion. In Findings of the Association for Computational Linguistics ACL 2024, pages 11437–11452, Bangkok, Thailand and… [find](https://scholar.google.com/scholar?q=Qibing+Ren%2C+Chang+Gao%2C+Jing+Shao%2C+Junchi+Yan%2C+Xin+Tan%2C+Wai+Lam%2C+and+Lizhuang+Ma%2E+2024%2E+CodeAttack%3A+Revealing+safety+generalization+challenges+of+large+language+models+via+code+completion%2E+In+Findings+)
44. Mohd Sanad Zaki Rizvi, Anirudh Srinivasan, Tanuja Ganu, Monojit Choudhury, and Sunayana Sitaram. 2021. GCM: A toolkit for generating synthetic code-mixed text. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages… [find](https://scholar.google.com/scholar?q=Mohd+Sanad+Zaki+Rizvi%2C+Anirudh+Srinivasan%2C+Tanuja+Ganu%2C+Monojit+Choudhury%2C+and+Sunayana+Sitaram%2E+2021%2E+GCM%3A+A+toolkit+for+generating+synthetic+code%2Dmixed+text%2E+In+Proceedings+of+the+16th+Conference+of)
45. Sheikh Shafayat, H Hasan, Minhajur Mahim, Rifki Putri, James Thorne, and Alice Oh. 2024. BEnQA: A question answering benchmark for Bengali and English. In Findings of the Association for Computational Linguistics ACL 2024, pages 1158–1177, Bangkok, Thailand and virtual meeting. Association for Compu… [find](https://scholar.google.com/scholar?q=Sheikh+Shafayat%2C+H+Hasan%2C+Minhajur+Mahim%2C+Rifki+Putri%2C+James+Thorne%2C+and+Alice+Oh%2E+2024%2E+BEnQA%3A+A+question+answering+benchmark+for+Bengali+and+English%2E+In+Findings+of+the+Association+for+Computational)
46. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. ["Do Anything Now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models](https://md.research.elamur.ai/free/en/papers/shen-2023-anything-now-characterizing-evaluating-wild/). In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, page 1671–1685, New…
47. Kushagra Singh, Indira Sen, and Ponnurangam Kumaraguru. 2018. A Twitter corpus for Hindi-English code mixed POS tagging. In Proceedings of the Sixth International Workshop on Natural Language Processing for Social Media, pages 12–17, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kushagra+Singh%2C+Indira+Sen%2C+and+Ponnurangam+Kumaraguru%2E+2018%2E+A+Twitter+corpus+for+Hindi%2DEnglish+code+mixed+POS+tagging%2E+In+Proceedings+of+the+Sixth+International+Workshop+on+Natural+Language+Processi)
48. Bibek Upadhayay and Vahid Behzadan. 2024. Sandwich attack: Multi-language mixture adaptive attack on LLMs. In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024), pages 208–226, Mexico City, Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bibek+Upadhayay+and+Vahid+Behzadan%2E+2024%2E+Sandwich+attack%3A+Multi%2Dlanguage+mixture+adaptive+attack+on+LLMs%2E+In+Proceedings+of+the+4th+Workshop+on+Trustworthy+Natural+Language+Processing+%28TrustNLP+2024%29)
49. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. 2024. All languages matter: On the multilingual safety of LLMs. In Findings of the Association for Computational Linguistics ACL 2024, pages 5865–5877, Bangkok, Thailand and virtual meeting. Associat… [find](https://scholar.google.com/scholar?q=Wenxuan+Wang%2C+Zhaopeng+Tu%2C+Chang+Chen%2C+Youliang+Yuan%2C+Jen%2Dtse+Huang%2C+Wenxiang+Jiao%2C+and+Michael+Lyu%2E+2024%2E+All+languages+matter%3A+On+the+multilingual+safety+of+LLMs%2E+In+Findings+of+the+Association+for+)
50. Barack W. Wanjawa, Lilian D. A. Wanzare, Florence Indede, Owen Mconyango, Lawrence Muchemi, and Edward Ombui. 2023. KenSwQuAD—a question answering dataset for swahili low-resource language. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(4). [find](https://scholar.google.com/scholar?q=Barack+W%2E+Wanjawa%2C+Lilian+D%2E+A%2E+Wanzare%2C+Florence+Indede%2C+Owen+Mconyango%2C+Lawrence+Muchemi%2C+and+Edward+Ombui%2E+2023%2E+KenSwQuAD%E2%80%94a+question+answering+dataset+for+swahili+low%2Dresource+language%2E+ACM+Trans%2E)
51. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. [Jailbroken: how does llm safety training fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.
52. Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and Thamar Solorio. 2023. The decades progress on code-switching research in NLP: A systematic survey on trends and challenges. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2936–2978, Toronto, Canada. Association for Com… [find](https://scholar.google.com/scholar?q=Genta+Winata%2C+Alham+Fikri+Aji%2C+Zheng+Xin+Yong%2C+and+Thamar+Solorio%2E+2023%2E+The+decades+progress+on+code%2Dswitching+research+in+NLP%3A+A+systematic+survey+on+trends+and+challenges%2E+In+Findings+of+the+Associ)
53. Rong Xiang, Mingyu Wan, Qi Su, Chu-Ren Huang, and Qin Lu. 2020. Sina Mandarin alphabetical words:a web-driven code-mixing lexical resource. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on… [find](https://scholar.google.com/scholar?q=Rong+Xiang%2C+Mingyu+Wan%2C+Qi+Su%2C+Chu%2DRen+Huang%2C+and+Qin+Lu%2E+2020%2E+Sina+Mandarin+alphabetical+words%3Aa+web%2Ddriven+code%2Dmixing+lexical+resource%2E+In+Proceedings+of+the+1st+Conference+of+the+Asia%2DPacific+Cha)
54. Zheng Xin Yong, Ruochen Zhang, Jessica Forde, Skyler Wang, Arjun Subramonian, Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Yin Lin Tan, Long Phan, Long Phan, Rowena Garcia, Thamar Solorio, and Alham Fikri Aji. 2023. [Prompting multilingual large languag…](https://md.research.elamur.ai/free/en/papers/yong-2023-prompting-multilingual-large-language-models/)
55. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. [GPTFUZZER: Red teaming large language models with auto-generated jailbreak prompts](https://md.research.elamur.ai/free/en/papers/yu-2023-gptfuzzer-red-teaming-large-language/).
56. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2024. [GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher](https://md.research.elamur.ai/free/en/papers/yuan-2023-gpt-4-too-smart-safe/). In The Twelfth International Conference on Learning Representations.
57. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
58. Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata, and Alham Fikri Aji. 2023. [Multilingual large language models are not (yet) code-switchers](https://md.research.elamur.ai/free/en/papers/zhang-2023-multilingual-large-language-models-are/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12567–12582, Singapore. Association…
59. Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond English: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055. [find](https://scholar.google.com/scholar?q=Jun+Zhao%2C+Zhihao+Zhang%2C+Luhui+Gao%2C+Qi+Zhang%2C+Tao+Gui%2C+and+Xuanjing+Huang%2E+2024%2E+Llama+beyond+English%3A+An+empirical+study+on+language+capability+transfer%2E+arXiv+preprint+arXiv%3A2401%2E01055%2E)
60. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Proceedings of the 37th International Conference on Neu…
61. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. [Universal and transferable adversarial attacks on aligned language models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/).

*Updated 1 November 2026*