---
title: "LLaMA Beyond English — paper by Zhao et al. (2024)"
description: "This study asks which adaptation steps are actually needed to carry LLaMA's text generation and… 58 references, cited by 72 papers here."
authors: ["Jun Zhao", "Zhihao Zhang", "Luhui Gao", "Qi Zhang", "Tao Gui", "Xuanjing Huang"]
year: 2024
venue: "arXiv:2401.01055"
source: "https://arxiv.org/abs/2401.01055"
doi: "10.48550/arxiv.2401.01055"
arxiv: "2401.01055"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2024-llama-english-empirical-study-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2024-llama-english-empirical-study-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / LLaMA Beyond English: An Empirical Study on Language Capability Transfer

# LLaMA Beyond English: An Empirical Study on Language Capability Transfer

**Authors:** [Jun Zhao](https://md.research.elamur.ai/free/en/authors/jun-zhao/), [Zhihao Zhang](https://md.research.elamur.ai/free/en/authors/zhihao-zhang/), Luhui Gao, [Qi Zhang](https://md.research.elamur.ai/free/en/authors/qi-zhang/), [Tao Gui](https://md.research.elamur.ai/free/en/authors/tao-gui/) and [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/) · **Year:** 2024 · **Venue:** arXiv:2401.01055 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.01055) · [PDF](https://arxiv.org/pdf/2401.01055) · [DOI 10.48550/arxiv.2401.01055](https://doi.org/10.48550/arxiv.2401.01055) · [arXiv 2401.01055](https://arxiv.org/abs/2401.01055) · [OpenAlex](https://openalex.org/W4390573446)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/zhao-2024-llama-english-empirical-study-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Most widely used open LLMs, LLaMA among them, learn mainly from English text and perform poorly in other languages. This study asks which adaptation steps are actually needed to carry LLaMA's text generation and instruction-following abilities into a new language, and how much training each step requires. Using Chinese as the main test case, the authors spent over 1440 GPU hours comparing LLaMA, LLaMA2 and public Chinese variants that had received 30B to 100B extra tokens. They varied three factors: vocabulary extension, further pretraining scale and instruction tuning size. Knowledge was measured with C-Eval, MMLU, AGI-Eval and GAOKAO-Bench. Answer quality was judged on LLM-Eval for accuracy, fluency, informativeness, coherence and harmlessness. Extending the vocabulary proved unnecessary and was even counterproductive at modest training scales. A model trained on less than 1% of the usual further-pretraining data matched the state-of-the-art Open Chinese LLaMA. The same pattern appeared across 13 low-resource languages. For teams building models for under-resourced languages, the results point to a low-cost recipe: keep the original tokenizer, invest in instruction data, and train on several languages together to preserve English skills.

## Key points

- **Vocabulary extension did not help.** Further pretraining on just 0.5B Chinese tokens with the original vocabulary beat an extended-vocabulary model (Chinese LLaMA, 49,953-token SentencePiece vocabulary) trained on over 30B tokens.
- **Knowledge gains need very large scale.** Up to roughly 100B tokens of Chinese pretraining barely raised LLaMA's knowledge scores.
- **Response quality is cheap to improve.** Hundreds of thousands of instruction examples were enough, without large-scale pretraining.
- **English skills are at risk.** Transfer training on Chinese alone clearly degraded LLaMA's English ability, and multilingual joint training largely prevented this loss.
- **Setup:** the models compared were LLaMA-7B/13B, LLaMA2-7B, Chinese LLaMA 7B Plus, Chinese LLaMA2 7B and Open Chinese LLaMA. LLM-Eval covers 17 instruction categories. The authors also report code-switching in model outputs (roughly 1.6–4.8% across languages) and read it as a sign of internal cross-lingual alignment.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works

## Cite

Cite as: Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui and Xuanjing Huang. 2024. LLaMA Beyond English: An Empirical Study on Language Capability Transfer. arXiv:2401.01055.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhao2024llama,
  title = {{LLaMA Beyond English: An Empirical Study on Language Capability Transfer}},
  author = {Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui and Xuanjing Huang},
  year = {2024},
  doi = {10.48550/arxiv.2401.01055},
  eprint = {2401.01055},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.01055}
}
```

- [← previous KoBBQ: Korean Bias Benchmark for Question Answering](https://md.research.elamur.ai/free/en/papers/jin-2024-kobbq-korean-bias-benchmark-question/)
- [next → Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps](https://md.research.elamur.ai/free/en/papers/chuang-2024-lookback-lens-detecting-mitigating-contextual/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

58 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anil, R.; Dai, A. M.; Firat, O.; Johnson, M.; and Lepikhin, D. 2023. [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv:2305.10403.
2. Artetxe, M.; Ruder, S.; and Yogatama, D. 2020. [On the Cross-lingual Transferability of Monolingual Representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4623–4637.
3. Online: Association for Computational Linguistics. Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; Nori, H.; Palangi, H.; Ribeiro, M. T.; and Zhang, Y. 2023. [find](https://scholar.google.com/scholar?q=Online%3A+Association+for+Computational+Linguistics%2E+Bubeck%2C+S%2E%3B+Chandrasekaran%2C+V%2E%3B+Eldan%2C+R%2E%3B+Gehrke%2C+J%2E%3B+Horvitz%2C+E%2E%3B+Kamar%2C+E%2E%3B+Lee%2C+P%2E%3B+Lee%2C+Y%2E+T%2E%3B+Li%2C+Y%2E%3B+Lundberg%2C+S%2E%3B+Nori%2C+H%2E%3B+Palangi%2C+H%2E%3B+Ribe)
4. [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv:2303.12712. Cahyawijaya, S.; Lovenia, H.; Aji, A. F.; Winata, G. I.; and Wilie, B. 2023. NusaCrowd: Open Source Initiative for Indonesian NLP Resources. arXiv:2212.09648.
5. Cao, S.; Kitaev, N.; and Klein, D. 2020. Multilingual Alignment of Contextual Word Representations. arXiv:2002.03518. [find](https://scholar.google.com/scholar?q=Cao%2C+S%2E%3B+Kitaev%2C+N%2E%3B+and+Klein%2C+D%2E+2020%2E+Multilingual+Alignment+of+Contextual+Word+Representations%2E+arXiv%3A2002%2E03518%2E)
6. Chen, G.; Ma, S.; Chen, Y.; Zhang, D.; Pan, J.; Wang, W.; and Wei, F. 2022. Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation. arXiv:2110.08547. — [arXiv:2110.08547](https://arxiv.org/abs/2110.08547) [find](https://scholar.google.com/scholar?q=Chen%2C+G%2E%3B+Ma%2C+S%2E%3B+Chen%2C+Y%2E%3B+Zhang%2C+D%2E%3B+Pan%2C+J%2E%3B+Wang%2C+W%2E%3B+and+Wei%2C+F%2E+2022%2E+Towards+Making+the+Most+of+Multilingual+Pretraining+for+Zero%2DShot+Neural+Machine+Translation%2E+arXiv%3A2110%2E08547%2E+%E2%80%94+%5BarXiv%3A211)
7. Chi, E. A.; Hewitt, J.; and Manning, C. D. 2020. [find](https://scholar.google.com/scholar?q=Chi%2C+E%2E+A%2E%3B+Hewitt%2C+J%2E%3B+and+Manning%2C+C%2E+D%2E+2020%2E)
8. Finding Universal Grammatical Relations in Multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5564–5577. Online: Association for Computational Linguistics. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulm… — [Training Verifiers to Solve Math Word Problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
9. Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; and Stoyanov, V. 2020a. [Unsupervised Cross-lingual Representation Learning at Scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). arXiv:1911.02116.
10. Conneau, A.; Wu, S.; Li, H.; Zettlemoyer, L.; and Stoyanov, V. 2020b. [Emerging Cross-lingual Structure in Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/conneau-2020-emerging-cross-lingual-structure-pretrained/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6022– 6034. Online: Association for Computational Linguistics.
11. Conover, M.; Hayes, M.; Mathur, A.; Xie, J.; Wan, J.; Shah, S.; Ghodsi, A.; Wendell, P.; Zaharia, M.; and Xin, R. 2023. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM. [find](https://scholar.google.com/scholar?q=Conover%2C+M%2E%3B+Hayes%2C+M%2E%3B+Mathur%2C+A%2E%3B+Xie%2C+J%2E%3B+Wan%2C+J%2E%3B+Shah%2C+S%2E%3B+Ghodsi%2C+A%2E%3B+Wendell%2C+P%2E%3B+Zaharia%2C+M%2E%3B+and+Xin%2C+R%2E+2023%2E+Free+Dolly%3A+Introducing+the+World%E2%80%99s+First+Truly+Open+Instruction%2DTuned+LLM%2E)
12. Cui, Y.; Yang, Z.; and Yao, X. 2023a. Chinese LLaMA and Alpaca Large Language Models. [find](https://scholar.google.com/scholar?q=Cui%2C+Y%2E%3B+Yang%2C+Z%2E%3B+and+Yao%2C+X%2E+2023a%2E+Chinese+LLaMA+and+Alpaca+Large+Language+Models%2E)
13. Cui, Y.; Yang, Z.; and Yao, X. 2023b. [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://md.research.elamur.ai/free/en/papers/cui-2023-efficient-effective-text-encoding-chinese/). arXiv:2304.08177.
14. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%3B+Chang%2C+M%2E%2DW%2E%3B+Lee%2C+K%2E%3B+and+Toutanova%2C+K%2E+2019%2E)
15. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. Minneapolis, Minnesota: As…
16. Dong, Q.; Li, L.; Dai, D.; Zheng, C.; Wu, Z.; Chang, B.; Sun, X.; Xu, J.; Li, L.; and Sui, Z. 2023. [A Survey on In-context Learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/). arXiv:2301.00234.
17. Dufter, P.; and Schütze, H. 2020. [Identifying Elements Essential for BERT’s Multilinguality](https://md.research.elamur.ai/free/en/papers/dufter-2020-identifying-elements-essential-bert-s/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4423–4437. Online: Association for Computational Linguistics.
18. Eronen, J.; Ptaszynski, M.; and Masui, F. 2023. Zero-shot cross-lingual transfer language selection using linguistic similarity. Information Processing & Management, 60(3): 103250. [find](https://scholar.google.com/scholar?q=Eronen%2C+J%2E%3B+Ptaszynski%2C+M%2E%3B+and+Masui%2C+F%2E+2023%2E+Zero%2Dshot+cross%2Dlingual+transfer+language+selection+using+linguistic+similarity%2E+Information+Processing+%26+Management%2C+60%283%29%3A+103250%2E)
19. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). CoRR, abs/2009.03300.
20. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W. 2021. [LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). CoRR, abs/2106.09685.
21. Huang, H.; Tang, T.; Zhang, D.; Zhao, W. X.; Song, T.; Xia, Y.; and Wei, F. 2023a. [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://md.research.elamur.ai/free/en/papers/huang-2023-all-languages-created-equal-llms/). arXiv:2305.07004.
22. Huang, W.; Abbeel, P.; Pathak, D.; and Mordatch, I. 2022. [find](https://scholar.google.com/scholar?q=Huang%2C+W%2E%3B+Abbeel%2C+P%2E%3B+Pathak%2C+D%2E%3B+and+Mordatch%2C+I%2E+2022%2E)
23. [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/). In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvari, C.; Niu, G.; and Sabato, S., eds., Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Rese…
24. Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; and Zhang, J. 2023b. [find](https://scholar.google.com/scholar?q=Huang%2C+Y%2E%3B+Bai%2C+Y%2E%3B+Zhu%2C+Z%2E%3B+Zhang%2C+J%2E%3B+and+Zhang%2C+J%2E+2023b%2E)
25. [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). arXiv:2305.08322. Ji, Y.; Deng, Y.; Gong, Y.; Peng, Y.; Niu, Q.; Ma, B.; and Li, X. 2023. BELLE: Be Everyone’s Large Language model Engine. [https://github.com/LianjiaTech/BELLE](https://github.com/LianjiaTech/BELLE). [↗](https://github.com/LianjiaTech/BELLE)
26. Jiang, Z.; Anastasopoulos, A.; Araki, J.; Ding, H.; and Neubig, G. 2020. X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 5943–5959. [find](https://scholar.google.com/scholar?q=Jiang%2C+Z%2E%3B+Anastasopoulos%2C+A%2E%3B+Araki%2C+J%2E%3B+Ding%2C+H%2E%3B+and+Neubig%2C+G%2E+2020%2E+X%2DFACTR%3A+Multilingual+Factual+Knowledge+Retrieval+from+Pretrained+Language+Models%2E+In+Proceedings+of+the+2020+Conference+on+Emp)
27. Online: Association for Computational Linguistics. Joshi, P.; Santy, S.; Budhiraja, A.; Bali, K.; and Choudhury, M. 2020. [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6282–6293.…
28. Katz, D. M.; Bommarito, M. J.; Gao, S.; and Arredondo, P. 2023. Gpt-4 passes the bar exam. Available at SSRN 4389233. [find](https://scholar.google.com/scholar?q=Katz%2C+D%2E+M%2E%3B+Bommarito%2C+M%2E+J%2E%3B+Gao%2C+S%2E%3B+and+Arredondo%2C+P%2E+2023%2E+Gpt%2D4+passes+the+bar+exam%2E+Available+at+SSRN+4389233%2E)
29. Khanuja, S.; Dandapat, S.; Srinivasan, A.; Sitaram, S.; and Choudhury, M. 2020. [GLUECoS: An Evaluation Benchmark for Code-Switched NLP](https://md.research.elamur.ai/free/en/papers/khanuja-2020-gluecos-evaluation-benchmark-code-switched/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 3575–3585. Online: Association for Computational Linguistics.
30. Krishnan, J.; Anastasopoulos, A.; Purohit, H.; and Rangwala, H. 2021. Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Filling. arXiv:2103.07792. [find](https://scholar.google.com/scholar?q=Krishnan%2C+J%2E%3B+Anastasopoulos%2C+A%2E%3B+Purohit%2C+H%2E%3B+and+Rangwala%2C+H%2E+2021%2E+Multilingual+Code%2DSwitching+for+Zero%2DShot+Cross%2DLingual+Intent+Prediction+and+Slot+Filling%2E+arXiv%3A2103%2E07792%2E)
31. Li, H.; Koto, F.; Wu, M.; Aji, A. F.; and Baldwin, T. 2023. [Bactrian-X : A Multilingual Replicable Instruction-Following Model with Low-Rank Adaptation](https://md.research.elamur.ai/free/en/papers/li-2023-bactrian-x-multilingual-replicable-instruction/). arXiv:2305.15011.
32. Lin, X. V.; Mihaylov, T.; Artetxe, M.; Wang, T.; Chen, S.; Simig, D.; Ott, M.; Goyal, N.; Bhosale, S.; Du, J.; Pasunuru, R.; Shleifer, S.; Koura, P. S.; Chaudhary, V.; O’Horo, B.; Wang, J.; Zettlemoyer, L.; Kozareva, Z.; Diab, M.; Stoyanov, V.; and Li, X. 2022. [Few-shot Learning with Multilingual La…](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
33. Nguyen, X.-P.; Aljunied, S. M.; Joty, S.; and Bing, L. 2023. [Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts](https://md.research.elamur.ai/free/en/papers/nguyen-2024-democratizing-llms-low-resource-languages/). arXiv:2306.11372.
34. OpenAI. 2022. Introducing ChatGPT. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+Introducing+ChatGPT%2E)
35. OpenLMLab. 2023. Open-Chinese-LLaMA. [find](https://scholar.google.com/scholar?q=OpenLMLab%2E+2023%2E+Open%2DChinese%2DLLaMA%2E)
36. Penedo, G.; Malartic, Q.; Hesslow, D.; Cojocaru, R.; Cappelli, A.; Alobeidli, H.; Pannier, B.; Almazrouei, E.; and Launay, J. 2023. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. arXiv:2306.01116. [find](https://scholar.google.com/scholar?q=Penedo%2C+G%2E%3B+Malartic%2C+Q%2E%3B+Hesslow%2C+D%2E%3B+Cojocaru%2C+R%2E%3B+Cappelli%2C+A%2E%3B+Alobeidli%2C+H%2E%3B+Pannier%2C+B%2E%3B+Almazrouei%2C+E%2E%3B+and+Launay%2C+J%2E+2023%2E+The+RefinedWeb+Dataset+for+Falcon+LLM%3A+Outperforming+Curated+Corpora)
37. Pires, T.; Schlinger, E.; and Garrette, D. 2019. [How Multilingual is Multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4996–5001. Florence, Italy: Association for Computational Linguistics.
38. Ranta, A.; and Goutte, C. 2021. Linguistic Diversity in Natural Language Processing. Traitement Automatique des Langues, 62(3): 7–11. [find](https://scholar.google.com/scholar?q=Ranta%2C+A%2E%3B+and+Goutte%2C+C%2E+2021%2E+Linguistic+Diversity+in+Natural+Language+Processing%2E+Traitement+Automatique+des+Langues%2C+62%283%29%3A+7%E2%80%9311%2E)
39. Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; Ilić, S.; Hesslow, D.; and Castagné, R. 2023. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Scao%2C+T%2E+L%2E%3B+Fan%2C+A%2E%3B+Akiki%2C+C%2E%3B+Pavlick%2C+E%2E%3B+Ilic%CC%81%2C+S%2E%3B+Hesslow%2C+D%2E%3B+and+Castagne%CC%81%2C+R%2E+2023%2E+BLOOM%3A+A+176B%2DParameter+Open%2DAccess+Multilingual+Language+Model%2E+arXiv%3A2211%2E05100%2E)
40. StabilityAI. 2023. Announcing StableCode. [find](https://scholar.google.com/scholar?q=StabilityAI%2E+2023%2E+Announcing+StableCode%2E)
41. Tan, S.; and Joty, S. 2021. Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots. arXiv:2103.09593. [find](https://scholar.google.com/scholar?q=Tan%2C+S%2E%3B+and+Joty%2C+S%2E+2021%2E+Code%2DMixing+on+Sesame+Street%3A+Dawn+of+the+Adversarial+Polyglots%2E+arXiv%3A2103%2E09593%2E)
42. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Alpaca: A Strong, Replicable Instruction-Following Model. [find](https://scholar.google.com/scholar?q=Taori%2C+R%2E%3B+Gulrajani%2C+I%2E%3B+Zhang%2C+T%2E%3B+Dubois%2C+Y%2E%3B+Li%2C+X%2E%3B+Guestrin%2C+C%2E%3B+Liang%2C+P%2E%3B+and+Hashimoto%2C+T%2E+B%2E+2023%2E+Alpaca%3A+A+Strong%2C+Replicable+Instruction%2DFollowing+Model%2E)
43. Team, I. 2023a. Internlm: A multilingual language model with progressively enhanced capabilities. [find](https://scholar.google.com/scholar?q=Team%2C+I%2E+2023a%2E+Internlm%3A+A+multilingual+language+model+with+progressively+enhanced+capabilities%2E)
44. Team, I. 2023b. InternLM: A Multilingual Language Model with Progressively Enhanced Capabilities. [https://github.com/InternLM/InternLM-techreport](https://github.com/InternLM/InternLM-techreport). [↗](https://github.com/InternLM/InternLM-techreport)
45. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023a. [LLaMA: Open and Efficient Foundation Language Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv:2302.13971.
46. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; and Almahairi, A. 2023b. [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv:2307.09288.
47. Winata, G. I.; Cahyawijaya, S.; Liu, Z.; Lin, Z.; Madotto, A.; and Fung, P. 2021a. Are Multilingual Models Effective in Code-Switching? arXiv:2103.13309. [find](https://scholar.google.com/scholar?q=Winata%2C+G%2E+I%2E%3B+Cahyawijaya%2C+S%2E%3B+Liu%2C+Z%2E%3B+Lin%2C+Z%2E%3B+Madotto%2C+A%2E%3B+and+Fung%2C+P%2E+2021a%2E+Are+Multilingual+Models+Effective+in+Code%2DSwitching%3F+arXiv%3A2103%2E13309%2E)
48. Winata, G. I.; Lin, Z.; and Fung, P. 2019. Learning Multilingual Meta-Embeddings for Code-Switching Named Entity Recognition. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), 181–186. Florence, Italy: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Winata%2C+G%2E+I%2E%3B+Lin%2C+Z%2E%3B+and+Fung%2C+P%2E+2019%2E+Learning+Multilingual+Meta%2DEmbeddings+for+Code%2DSwitching+Named+Entity+Recognition%2E+In+Proceedings+of+the+4th+Workshop+on+Representation+Learning+for+NLP+%28Rep)
49. Winata, G. I.; Lin, Z.; Shin, J.; Liu, Z.; and Fung, P. 2019. [find](https://scholar.google.com/scholar?q=Winata%2C+G%2E+I%2E%3B+Lin%2C+Z%2E%3B+Shin%2C+J%2E%3B+Liu%2C+Z%2E%3B+and+Fung%2C+P%2E+2019%2E)
50. Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3541–3547. Hong Kong, China: Association for C… [find](https://scholar.google.com/scholar?q=Hierarchical+Meta%2DEmbeddings+for+Code%2DSwitching+Named+Entity+Recognition%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+International+Joint+Conf)
51. Winata, G. I.; Madotto, A.; Lin, Z.; Liu, R.; Yosinski, J.; and Fung, P. 2021b. [Language Models are Few-shot Multilingual Learners](https://md.research.elamur.ai/free/en/papers/winata-2021-language-models-are-few-shot/). In Proceedings of the 1st Workshop on Multilingual Representation Learning, 1–15. Punta Cana, Dominican Republic: Association for Computational Linguistics.
52. Wu, S.; and Dredze, M. 2019. [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/).
53. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 833–844. Hong Kong, China: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+International+Joint+Conference+on+Natural+Language+Processing+%28EMNLP%2DIJCNLP%29%2C+833%E2%80%93844%2E+Hong+Kong%2C+)
54. Wu, Z.; Papadimitriou, I.; and Tamkin, A. 2022. Oolong: Investigating What Makes Crosslingual Transfer Hard with Controlled Studies. arXiv:2202.12312. — [arXiv:2202.12312](https://arxiv.org/abs/2202.12312) [find](https://scholar.google.com/scholar?q=Wu%2C+Z%2E%3B+Papadimitriou%2C+I%2E%3B+and+Tamkin%2C+A%2E+2022%2E+Oolong%3A+Investigating+What+Makes+Crosslingual+Transfer+Hard+with+Controlled+Studies%2E+arXiv%3A2202%2E12312%2E+%E2%80%94+%5BarXiv%3A2202%2E12312%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2202%2E1)
55. Zhang, M.; Zhang, Q.; Zhang, Y.; and Gui, T. 2023a. [find](https://scholar.google.com/scholar?q=Zhang%2C+M%2E%3B+Zhang%2C+Q%2E%3B+Zhang%2C+Y%2E%3B+and+Gui%2C+T%2E+2023a%2E)
56. LLMEVAL-1 Chinese Large Language Model Evaluation Phase 1. Zhang, X.; Li, C.; Zong, Y.; Ying, Z.; He, L.; and Qiu, X. 2023b. [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](https://md.research.elamur.ai/free/en/papers/zhang-2023-evaluating-performance-large-language/). arXiv:2305.12474.
57. Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2023. [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). arXiv:2304.06364.
58. Zhu, W.; Liu, H.; Dong, Q.; Xu, J.; Huang, S.; Kong, L.; Chen, J.; and Li, L. 2023. [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://md.research.elamur.ai/free/en/papers/zhu-2024-multilingual-machine-translation-large-language/). arXiv:2304.04675.

*Updated 2 November 2026*