---
title: "Extrapolating Large Language Models to Non-English by Aligning"
description: "Large language models such as LLaMA are trained mostly on English text and English instruction… 29 references, cited by 63 papers here."
authors: ["Wenhao Zhu", "Yunzhe Lv", "Qingxiu Dong", "Fei Yuan", "Jingjing Xu", "Shujian Huang", "Lingpeng Kong", "Jiajun Chen", "Lei Li"]
year: 2023
venue: "arXiv:2308.04948"
source: "https://arxiv.org/abs/2308.04948"
doi: "10.48550/arxiv.2308.04948"
arxiv: "2308.04948"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2023-extrapolating-large-language-models-non/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2023-extrapolating-large-language-models-non/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Extrapolating Large Language Models to Non-English by Aligning Languages

# Extrapolating Large Language Models to Non-English by Aligning Languages

**Authors:** [Wenhao Zhu](https://md.research.elamur.ai/free/en/authors/wenhao-zhu/), Yunzhe Lv, [Qingxiu Dong](https://md.research.elamur.ai/free/en/authors/qingxiu-dong/), [Fei Yuan](https://md.research.elamur.ai/free/en/authors/fei-yuan/), [Jingjing Xu](https://md.research.elamur.ai/free/en/authors/jingjing-xu/), [Shujian Huang](https://md.research.elamur.ai/free/en/authors/shujian-huang/), [Lingpeng Kong](https://md.research.elamur.ai/free/en/authors/lingpeng-kong/), [Jiajun Chen](https://md.research.elamur.ai/free/en/authors/jiajun-chen/) and [Lei Li](https://md.research.elamur.ai/free/en/authors/lei-li/) · **Year:** 2023 · **Venue:** arXiv:2308.04948 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.04948) · [PDF](https://arxiv.org/pdf/2308.04948) · [DOI 10.48550/arxiv.2308.04948](https://doi.org/10.48550/arxiv.2308.04948) · [arXiv 2308.04948](https://arxiv.org/abs/2308.04948) · [OpenAlex](https://openalex.org/W4385750324)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/zhu-2023-extrapolating-large-language-models-non/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models such as LLaMA are trained mostly on English text and English instruction data, so they perform much worse in other languages, especially those whose scripts differ from English. The authors transfer English competence to other languages by building semantic alignment during instruction-tuning. This avoids costly continued pre-training on monolingual corpora. In cross-lingual instruction-tuning (CoIT), LLaMA-7B is tuned on two kinds of data:
- the Alpaca instruction set, machine-translated into a target language and kept alongside the English original;
- translation instructions built from public parallel corpora.

CoIT produces one x-LLaMA per language. Pooling all resources in multilingual instruction-tuning (MuIT) yields a single m-LLaMA. The authors also fit a scaling law that relates translation data volume and language similarity to translation quality. They use it to divide a limited data budget across languages. The x-LLaMAs clearly beat Alpaca-7B on non-English QA. m-LLaMA comes close to them while following instructions in several languages. The recipe is cheap and reproducible, so it suits adapting LLMs to languages like Ukrainian.

## Key points
- Six target languages with little alphabet overlap with English: Arabic, Greek, Hindi, Turkish, Vietnamese, Chinese. Translation data comes from WikiMatrix and News Commentary.
- On XQUAD and MLQA, x-LLaMAs improve on Alpaca-7B by 27.83% on average. Their non-English accuracy approaches Alpaca-7B's English accuracy.
- On FLORES-101, x-LLaMAs outperform earlier LLaMA-based models by 18.89% on average. They also beat the supervised M2M-12B in half of the tested directions.
- Translation data with the non-English side as the target (En→X) helps more than X→En.
- The scaling law is S(X) = 100 − α·(γ·X)^β, where γ is similarity to English. Budget allocation based on it outperforms uniform allocation. Representation analysis shows multilingual alignment in m-LLaMA's middle layers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 118 works · 13,496 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 152 works · 13,166 citations
- [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 67 works · 10,550 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 99 works · 8,512 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen and Lei Li. 2023. Extrapolating Large Language Models to Non-English by Aligning Languages. arXiv:2308.04948.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhu2023extrapolating,
  title = {{Extrapolating Large Language Models to Non-English by Aligning Languages}},
  author = {Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen and Lei Li},
  year = {2023},
  doi = {10.48550/arxiv.2308.04948},
  eprint = {2308.04948},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.04948}
}
```

- [← previous Unveiling the Implicit Toxicity in Large Language Models](https://md.research.elamur.ai/free/en/papers/wen-2023-unveiling-implicit-toxicity-large-language/)
- [next → Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks](https://md.research.elamur.ai/free/en/papers/kang-2023-knowledge-augmented-reasoning-distillation-small/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

29 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
2. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. [A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). arXiv preprint arXiv:2302.04023, 2023.
3. Terra Blevins and Luke Zettlemoyer. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. [find](https://scholar.google.com/scholar?q=Terra+Blevins+and+Luke+Zettlemoyer%2E+Language+contamination+helps+explains+the+cross%2Dlingual+capabilities+of+English+pretrained+models%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+)
4. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672, 2022. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2Djussa%CC%80%2C+James+Cross%2C+Onur+C%CC%A7elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+No+language+left+behind%3A+Scaling+human)
5. Yiming Cui, Ziqing Yang, and Xin Yao. [Efficient and effective text encoding for chinese llama and alpaca](https://md.research.elamur.ai/free/en/papers/cui-2023-efficient-effective-text-encoding-chinese/). arXiv preprint arXiv:2304.08177, 2023.
6. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. The Journal of Machine Learning Research (JMLR), 2021. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+et+al%2E+Beyond+english%2Dcentric+multilingual+mac)
7. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational… [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzma%CC%81n%2C+and+Angela+Fan%2E+The+Flores%2D101+evaluation+benchmark+for+)
8. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. [Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting](https://md.research.elamur.ai/free/en/papers/huang-2023-all-languages-created-equal-llms/). arXiv preprint arXiv:2305.07004, 2023.
9. Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Xing Wang, Shuming Shi, and Zhaopeng Tu. Parrot: Translating during chat using large language models. arXiv preprint arXiv:2304.02426, 2023. — [arXiv:2304.02426](https://arxiv.org/abs/2304.02426) [find](https://scholar.google.com/scholar?q=Wenxiang+Jiao%2C+Jen%2Dtse+Huang%2C+Wenxuan+Wang%2C+Xing+Wang%2C+Shuming+Shi%2C+and+Zhaopeng+Tu%2E+Parrot%3A+Translating+during+chat+using+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E02426%2C+2023%2E+%E2%80%94+%5BarXiv%3A2304%2E0)
10. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. [MLQA: Evaluating cross-lingual extractive question answering](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
11. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.[16634, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/).
12. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/). arXiv preprint arXiv:2301.13688, 2023.
13. Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. [Contrastive learning for many-to-many multilingual neural machine translation](https://md.research.elamur.ai/free/en/papers/pan-2021-contrastive-learning-many-many-multilingual/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
14. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2002.
15. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
16. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://md.research.elamur.ai/free/en/papers/schwenk-2019-wikimatrix-mining-135m-parallel-sentences/). In Proceedings of Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2021.
17. Thibault Sellam, Dipanjan Das, and Ankur Parikh. [BLEURT: Learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/). In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
18. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
19. Jörg Tiedemann. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC), 2012. [find](https://scholar.google.com/scholar?q=Jo%CC%88rg+Tiedemann%2E+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%29%2C+2012%2E)
20. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
21. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023a. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023a%2E)
22. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+)
23. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. [How far can camels go? exploring the state of instruction tuning on open resources](https://md.research.elamur.ai/free/en/papers/wang-2023-far-camels-go-exploring-state/). arXiv preprint arXiv:2306.04751, 2023b.
24. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In International Conference on Learning Representations (ICLR), 2022.
25. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistic…
26. Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. Bigtrans: Augmenting large language models with multilingual translation capability over 100 languages. arXiv preprint arXiv:2305.18098, 2023. — [arXiv:2305.18098](https://arxiv.org/abs/2305.18098) [find](https://scholar.google.com/scholar?q=Wen+Yang%2C+Chong+Li%2C+Jiajun+Zhang%2C+and+Chengqing+Zong%2E+Bigtrans%3A+Augmenting+large+language+models+with+multilingual+translation+capability+over+100+languages%2E+arXiv+preprint+arXiv%3A2305%2E18098%2C+2023%2E+%E2%80%94+%5B)
27. Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, et al. [Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-bayling-bridging-cross-lingual-alignment/). arXiv preprint arXiv:2306.10968,…
28. Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023. [find](https://scholar.google.com/scholar?q=Yanli+Zhao%2C+Andrew+Gu%2C+Rohan+Varma%2C+Liang+Luo%2C+Chien%2DChin+Huang%2C+Min+Xu%2C+Less+Wright%2C+Hamid+Shojanazeri%2C+Myle+Ott%2C+Sam+Shleifer%2C+et+al%2E+Pytorch+fsdp%3A+experiences+on+scaling+fully+sharded+data+parallel)
29. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. [Multilingual machine translation with large language models: Empirical results and analysis](https://md.research.elamur.ai/free/en/papers/zhu-2024-multilingual-machine-translation-large-language/). arXiv preprint arXiv:2304.04675, 2023.

*Updated 2 November 2026*