---
title: "Multilingual Large Language Models Are Not (Yet) Code-Switchers"
description: "This paper offers a broader benchmark spanning four CSW tasks: classifying sentiment, translating… 61 references, cited by 59 papers here."
authors: ["Ruochen Zhang", "Samuel Cahyawijaya", "Jan Christian Blaise Cruz", "Genta Winata", "Alham Fikri Aji"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.774/"
doi: "10.18653/v1/2023.emnlp-main.774"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2023-multilingual-large-language-models-are/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2023-multilingual-large-language-models-are/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Multilingual Large Language Models Are Not (Yet) Code-Switchers

# Multilingual Large Language Models Are Not (Yet) Code-Switchers

**Authors:** Ruochen Zhang, [Samuel Cahyawijaya](https://md.research.elamur.ai/free/en/authors/samuel-cahyawijaya/), Jan Christian Blaise Cruz, [Genta Winata](https://md.research.elamur.ai/free/en/authors/genta-indra-winata/) and [Alham Fikri Aji](https://md.research.elamur.ai/free/en/authors/alham-fikri-aji/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.774/) · [PDF](https://aclanthology.org/2023.emnlp-main.774.pdf) · [DOI 10.18653/v1/2023.emnlp-main.774](https://doi.org/10.18653/v1/2023.emnlp-main.774) · [OpenAlex](https://openalex.org/W4389519244)

**Citations:** 26 citations (OpenAlex, 2026-10-14) · [59](https://md.research.elamur.ai/free/en/papers/zhang-2023-multilingual-large-language-models-are/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Code-switching (CSW), where speakers alternate between languages within a single utterance, is common in multilingual communities. However, it is unclear whether multilingual LLMs can actually process such mixed text. Earlier studies mostly tested small BERT-style encoders on synthetic data and covered few languages or tasks. This paper offers a broader benchmark spanning four CSW tasks: classifying sentiment, translating text, producing summaries, and identifying the language of each individual word. It covers five language pairs: Hindi-English, Spanish-English, Tamil-English, Malayalam-English and MSA-Egyptian Arabic.

The authors prompt BLOOMZ, mT0, XGLM and ChatGPT (GPT-3.5 turbo) at multiple model sizes in zero- and few-shot settings. They compare the results against much smaller fine-tuned models such as XLM-R, mBERT, mDeBERTa v3, M2M100, mBART-50 and mT0. Prompted performance grows with model size, yet fine-tuned small models still clearly win. ChatGPT reaches roughly fine-tuned quality, but its closed nature makes this hard to interpret.

The work shows that being multilingual does not automatically make a model competent at code-switching. It argues that CSW-specific data and evaluation should become part of LLM development.

## Key points
- **Datasets:**
  - Sentiment analysis: Sentimix Spanish-English, MixSentiment Malayalam and MixSentiment Tamil, reduced to positive/negative labels only.
  - Machine translation: MixMT 2022 Hinglish-English, with 8,060/942/960 train/validation/test pairs.
  - Summarization: Gupshup, derived from SAMSum, with 5,831 training pairs and 500 each for validation and test.
  - Language identification: the LinCE English-Hindi and MSA-EA subsets, evaluated on validation data because the test set is not public.
- **Prompting setup:** 0, 1, 3 and 5 in-context examples, each with five different prompt templates.
- **Model sizes:** mT0 from 300M to 13B, BLOOMZ from 560M to 7.1B (the 175B variant was excluded), and XGLM from 564M to 7.5B.
- **Main finding:** prompted performance follows a scaling trend across tasks and architectures, but fine-tuned models of far smaller size substantially outperform even the largest prompted open model.
- **Stated limitations:** the study covers only five language pairs and four tasks, and Llama-2 and Falcon were left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Brown University](https://md.research.elamur.ai/free/en/universities/brown-university/) — University, Providence, United States · 58 works · 25,310 citations
- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 58 works · 23,058 citations
- [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 47 works · 4,456 citations
- [Bloomberg](https://md.research.elamur.ai/free/en/companies/bloomberg/) — Company, New York, United States · 5 works · 1,546 citations
- [Samsung](https://md.research.elamur.ai/free/en/companies/samsung-south-korea/) — Company, Seoul, South Korea · 3 works · 1,138 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works

## Cite

Cite as: Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata and Alham Fikri Aji. 2023. Multilingual Large Language Models Are Not (Yet) Code-Switchers. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2023multilingual,
  title = {{Multilingual Large Language Models Are Not (Yet) Code-Switchers}},
  author = {Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata and Alham Fikri Aji},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.774},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.774}
}
```

- [← previous Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM](https://md.research.elamur.ai/free/en/papers/bawden-2023-investigating-translation-performance-large-multilingual/)
- [next → NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

61 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Farid Adilazuarda, Samuel Cahyawijaya, Genta Indra Winata, Pascale Fung, and Ayu Purwarianti. 2022. Indorobusta: Towards robustness against diverse code-mixed indonesian local languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, pages 25–34. [find](https://scholar.google.com/scholar?q=Muhammad+Farid+Adilazuarda%2C+Samuel+Cahyawijaya%2C+Genta+Indra+Winata%2C+Pascale+Fung%2C+and+Ayu+Purwarianti%2E+2022%2E+Indorobusta%3A+Towards+robustness+against+diverse+code%2Dmixed+indonesian+local+languages%2E+In+P)
2. Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. [Lince: A centralized benchmark for linguistic code-switching evaluation](https://md.research.elamur.ai/free/en/papers/aguilar-2020-lince-centralized-benchmark-linguistic-code/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1803–1813.
3. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. [Massively multilingual neural machine translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874–3884…
4. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. [A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/).
5. John Baugh. 2005. Linguistic profiling. In Black linguistics, pages 167–180. Routledge. [find](https://scholar.google.com/scholar?q=John+Baugh%2E+2005%2E+Linguistic+profiling%2E+In+Black+linguistics%2C+pages+167%E2%80%93180%2E+Routledge%2E)
6. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. [Findings of the 2014 workshop on statistical machine translation](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/). In Proceedings of the…
7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
8. Samuel Cahyawijaya, Holy Lovenia, Tiezheng Yu, Willy Chung, and Pascale Fung. 2023. Instructalign: Teaching novel languages with to llms through alignment-based cross-lingual instruction. [find](https://scholar.google.com/scholar?q=Samuel+Cahyawijaya%2C+Holy+Lovenia%2C+Tiezheng+Yu%2C+Willy+Chung%2C+and+Pascale+Fung%2E+2023%2E+Instructalign%3A+Teaching+novel+languages+with+to+llms+through+alignment%2Dbased+cross%2Dlingual+instruction%2E)
9. Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Khodra, Ayu Purwarianti, and Pascale Fung. 2021. IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation. In Proce… [find](https://scholar.google.com/scholar?q=Samuel+Cahyawijaya%2C+Genta+Indra+Winata%2C+Bryan+Wilie%2C+Karissa+Vincentio%2C+Xiaohong+Li%2C+Adhiguna+Kuncoro%2C+Sebastian+Ruder%2C+Zhi+Yuan+Lim%2C+Syafri+Bahar%2C+Masayu+Khodra%2C+Ayu+Purwarianti%2C+and+Pascale+Fung%2E+20)
10. Bharathi Raja Chakravarthi, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John P Mc-Crae. 2020a. [A sentiment analysis dataset for code-mixed malayalam-english](https://md.research.elamur.ai/free/en/papers/chakravarthi-2020-sentiment-analysis-dataset-code-mixed/). arXiv preprint arXiv:2006.00210.
11. Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John Philip McCrae. 2020b. [Corpus creation for sentiment analysis in code-mixed Tamil-English text](https://md.research.elamur.ai/free/en/papers/chakravarthi-2020-corpus-creation-sentiment-analysis-code/). In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collabo…
12. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. [Scaling instruction-finetuned language models](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/). arXiv preprint arXiv:2210.11416.
13. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+%C3%89douard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
15. Kaustubh Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahadiran, Simon Mille, Ashish Shrivastava, Samson Tan, Tongshang Wu, Jascha Sohl-Dickstein, Jinho Choi, Eduard Hovy, Ondřej Dušek, Sebastian Ruder, Sajant Anand, Nagender Aneja, Rabin Banjade, Lisa B… [find](https://scholar.google.com/scholar?q=Kaustubh+Dhole%2C+Varun+Gangal%2C+Sebastian+Gehrmann%2C+Aadesh+Gupta%2C+Zhenhao+Li%2C+Saad+Mahamood%2C+Abinaya+Mahadiran%2C+Simon+Mille%2C+Ashish+Shrivastava%2C+Samson+Tan%2C+Tongshang+Wu%2C+Jascha+Sohl%2DDickstein%2C+Jinho+Ch)
16. Mark Dingemanse and Andreas Liesenfeld. 2022. From text to talk: Harnessing conversational corpora for humane and diversity-aware language technology. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5614–5633, Dublin, Ireland.… [find](https://scholar.google.com/scholar?q=Mark+Dingemanse+and+Andreas+Liesenfeld%2E+2022%2E+From+text+to+talk%3A+Harnessing+conversational+corpora+for+humane+and+diversity%2Daware+language+technology%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+)
17. A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2021. [A survey of code-switching: Linguistic and social perspectives for language technologies](https://md.research.elamur.ai/free/en/papers/dogruoz-2021-survey-code-switching-linguistic-social/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internationa…
18. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multiling… [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+Naman+Goyal%2C+Tom+Birch%2C+Vitaliy+Liptchinsky%2C+S)
19. Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. 2023. Scaling laws for multilingual neural machine translation. arXiv preprint arXiv:2302.09650. — [arXiv:2302.09650](https://arxiv.org/abs/2302.09650) [find](https://scholar.google.com/scholar?q=Patrick+Fernandes%2C+Behrooz+Ghorbani%2C+Xavier+Garcia%2C+Markus+Freitag%2C+and+Orhan+Firat%2E+2023%2E+Scaling+laws+for+multilingual+neural+machine+translation%2E+arXiv+preprint+arXiv%3A2302%2E09650%2E+%E2%80%94+%5BarXiv%3A2302%2E0965)
20. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bogdan+Gliwa%2C+Iwona+Mochol%2C+Maciej+Biesek%2C+and+Aleksander+Wawer%2E+2019%2E+SAMSum+corpus%3A+A+human%2Dannotated+dialogue+dataset+for+abstractive+summarization%2E+In+Proceedings+of+the+2nd+Workshop+on+New+Fronti)
21. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. [Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/).
22. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
23. Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. [Gluecos: An evaluation benchmark for code-switched nlp](https://md.research.elamur.ai/free/en/papers/khanuja-2020-gluecos-evaluation-benchmark-code-switched/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3575–3585.
24. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
25. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2E)
26. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Proc… [find](https://scholar.google.com/scholar?q=Hugo+Lauren%C3%A7on%2C+Lucile+Saulnier%2C+Thomas+Wang%2C+Christopher+Akiki%2C+Albert+Villanova+del+Moral%2C+Teven+Le+Scao%2C+Leandro+Von+Werra%2C+Chenghao+Mou%2C+Eduardo+Gonz%C3%A1lez+Ponferrada%2C+Huu+Nguyen%2C+et+al%2E+2022%2E+The+b)
27. Ke Li, Jinyu Li, Guoli Ye, Rui Zhao, and Yifan Gong. 2019. Towards code-switching asr for end-to-end ctc models. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6076–6080. [find](https://scholar.google.com/scholar?q=Ke+Li%2C+Jinyu+Li%2C+Guoli+Ye%2C+Rui+Zhao%2C+and+Yifan+Gong%2E+2019%2E+Towards+code%2Dswitching+asr+for+end%2Dto%2Dend+ctc+models%2E+In+ICASSP+2019+%2D+2019+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Pro)
28. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. [Few-shot learning with multilingual language models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/). arXiv preprint arXiv:2112.10668.
29. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. [What makes good in-context examples for gpt-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) arXiv preprint arXiv:2101.06804.
30. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactio)
31. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/). arXiv preprint arXiv:2301.13688.
32. Laiba Mehnaz, Debanjan Mahata, Rakesh Gosangi, Uma Sushmitha Gunturi, Riya Jain, Gauri Gupta, Amardeep Kumar, Isabelle G. Lee, Anish Acharya, and Rajiv Ratn Shah. 2021. GupShup: Summarizing open-domain code-switched conversations. In Proceedings of the 2021 Conference on Empirical Methods in Natural… [find](https://scholar.google.com/scholar?q=Laiba+Mehnaz%2C+Debanjan+Mahata%2C+Rakesh+Gosangi%2C+Uma+Sushmitha+Gunturi%2C+Riya+Jain%2C+Gauri+Gupta%2C+Amardeep+Kumar%2C+Isabelle+G%2E+Lee%2C+Anish+Acharya%2C+and+Rajiv+Ratn+Shah%2E+2021%2E+GupShup%3A+Summarizing+open%2Ddomai)
33. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. [Crosslingual generalization through multitask finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/). arXiv preprint arXiv:2211.01786.
34. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv preprint arXiv:2203.02155.
35. Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. [Multi-task prompted training enables zero-shot task generaliza…](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
36. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
37. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+International+Conference+on+Machine+Learning%2C+pages+4596%E2%80%934604%2E+PMLR%2E)
38. Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. arXiv preprint arXiv:2101.05779. [find](https://scholar.google.com/scholar?q=Giovanni+Paolini%2C+Ben+Athiwaratkun%2C+Jason+Krone%2C+Jie+Ma%2C+Alessandro+Achille%2C+Rishita+Anubhai%2C+Cicero+Nogueira+dos+Santos%2C+Bing+Xiang%2C+and+Stefano+Soatto%2E+2021%2E+Structured+prediction+as+translation+bet)
39. Vivek Srivastava and Mayank Singh. 2022. Overview and results of MixMT shared-task at WMT 2022. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 806–811, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vivek+Srivastava+and+Mayank+Singh%2E+2022%2E+Overview+and+results+of+MixMT+shared%2Dtask+at+WMT+2022%2E+In+Proceedings+of+the+Seventh+Conference+on+Machine+Translation+%28WMT%29%2C+pages+806%E2%80%93811%2C+Abu+Dhabi%2C+United+)
40. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023%2E+The+RefinedWeb+dataset+for+F)
41. Samson Tan and Shafiq Joty. 2021. Code-mixing on sesame street: Dawn of the adversarial polyglots. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3596–3616. [find](https://scholar.google.com/scholar?q=Samson+Tan+and+Shafiq+Joty%2E+2021%2E+Code%2Dmixing+on+sesame+street%3A+Dawn+of+the+adversarial+polyglots%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computatio)
42. S. Poplack. 2001. Code switching: Linguistic. In International Encyclopedia of the Social & Behavioral Sciences, pages 2062–2065. Elsevier. [find](https://scholar.google.com/scholar?q=S%2E+Poplack%2E+2001%2E+Code+switching%3A+Linguistic%2E+In+International+Encyclopedia+of+the+Social+%26+Behavioral+Sciences%2C+pages+2062%E2%80%932065%2E+Elsevier%2E)
43. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401. [find](https://scholar.google.com/scholar?q=Yuqing+Tang%2C+Chau+Tran%2C+Xian+Li%2C+Peng%2DJen+Chen%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Jiatao+Gu%2C+and+Angela+Fan%2E+2020%2E+Multilingual+translation+with+extensible+multilingual+pretraining+and+finetuning%2E+arXiv)
44. Shana Poplack. 1980. Sometimes i’ll start a sentence in spanish y termino en espanol: toward a typology of code-switching1. [find](https://scholar.google.com/scholar?q=Shana+Poplack%2E+1980%2E+Sometimes+i%E2%80%99ll+start+a+sentence+in+spanish+y+termino+en+espanol%3A+toward+a+typology+of+code%2Dswitching1%2E)
45. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2023. [Scaling speech technology to 1,000+ languages](https://md.research.elamur.ai/free/en/papers/pratap-2023-scaling-speech-technology-1-000/). arXiv preprint arXiv:2305.13516.
46. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv e-prints.
47. Together Computer SambaNova Systems. 2023. BLOOMChat: a New Open Multilingual Chat LLM. [find](https://scholar.google.com/scholar?q=Together+Computer+SambaNova+Systems%2E+2023%2E+BLOOMChat%3A+a+New+Open+Multilingual+Chat+LLM%2E)
48. [Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Da…](https://md.research.elamur.ai/free/en/papers/tang-2021-multilingual-translation-denoising-pre-training/)
49. Eshaan Tanwar, Manish Borthakur, Subhabrata Dutta, and Tanmoy Chakraborty. 2023. [Multilingual llms are better cross-lingual in-context learners with alignment](https://md.research.elamur.ai/free/en/papers/tanwar-2023-multilingual-llms-are-better-cross/). arXiv preprint arXiv:2305.05940.
50. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239.
51. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
52. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+2021%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2109)
53. Ruochen Zhang and Carsten Eickhoff. 2023. Crocosum: A benchmark dataset for cross-lingual code-switched summarization. arXiv preprint arXiv:2303.04092. [find](https://scholar.google.com/scholar?q=Ruochen+Zhang+and+Carsten+Eickhoff%2E+2023%2E+Crocosum%3A+A+benchmark+dataset+for+cross%2Dlingual+code%2Dswitched+summarization%2E+arXiv+preprint+arXiv%3A2303%2E04092%2E)
54. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems.
55. Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preotiuc-Pietro. 2022a. Cross-lingual few-shot learning on unseen languages. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Confere… [find](https://scholar.google.com/scholar?q=Genta+Winata%2C+Shijie+Wu%2C+Mayank+Kulkarni%2C+Thamar+Solorio%2C+and+Daniel+Preotiuc%2DPietro%2E+2022a%2E+Cross%2Dlingual+few%2Dshot+learning+on+unseen+languages%2E+In+Proceedings+of+the+2nd+Conference+of+the+Asia%2DPacif)
56. Genta Indra Winata, Alham Fikri Aji, Zheng-Xin Yong, and Thamar Solorio. 2022b. The decades progress on code-switching research in nlp: A systematic survey on trends and challenges. arXiv preprint arXiv:2212.09660. [find](https://scholar.google.com/scholar?q=Genta+Indra+Winata%2C+Alham+Fikri+Aji%2C+Zheng%2DXin+Yong%2C+and+Thamar+Solorio%2E+2022b%2E+The+decades+progress+on+code%2Dswitching+research+in+nlp%3A+A+systematic+survey+on+trends+and+challenges%2E+arXiv+preprint+arX)
57. Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, Peng Xu, and Pascale Fung. 2020. Meta-transfer learning for code-switched speech recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3770–3776, Online. Association for Computa… [find](https://scholar.google.com/scholar?q=Genta+Indra+Winata%2C+Samuel+Cahyawijaya%2C+Zhaojiang+Lin%2C+Zihan+Liu%2C+Peng+Xu%2C+and+Pascale+Fung%2E+2020%2E+Meta%2Dtransfer+learning+for+code%2Dswitched+speech+recognition%2E+In+Proceedings+of+the+58th+Annual+Meetin)
58. Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2021a. Are multilingual models effective in code-switching? In Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, pages 142–153. [find](https://scholar.google.com/scholar?q=Genta+Indra+Winata%2C+Samuel+Cahyawijaya%2C+Zihan+Liu%2C+Zhaojiang+Lin%2C+Andrea+Madotto%2C+and+Pascale+Fung%2E+2021a%2E+Are+multilingual+models+effective+in+code%2Dswitching%3F+In+Proceedings+of+the+Fifth+Workshop+on+)
59. Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021b. [Language models are few-shot multilingual learners](https://md.research.elamur.ai/free/en/papers/winata-2021-language-models-are-few-shot/). In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 1–15.
60. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
61. Zheng-Xin Yong, Ruochen Zhang, Jessica Zosa Forde, Skyler Wang, Samuel Cahyawijaya, Holy Lovenia, Genta Indra Winata, Lintang Sutawika, Jan Christian Blaise Cruz, Long Phan, et al. 2023. Prompting multilingual large language models to generate code-mixed texts: The case of south east asian languages… [find](https://scholar.google.com/scholar?q=Zheng%2DXin+Yong%2C+Ruochen+Zhang%2C+Jessica+Zosa+Forde%2C+Skyler+Wang%2C+Samuel+Cahyawijaya%2C+Holy+Lovenia%2C+Genta+Indra+Winata%2C+Lintang+Sutawika%2C+Jan+Christian+Blaise+Cruz%2C+Long+Phan%2C+et+al%2E+2023%2E+Prompting+mul)

*Updated 24 October 2026*