---
title: "Do Multilingual Language Models Think Better in English? (2024)"
description: "Translate-test (machine-translating non-English inputs into English before querying a model) often… 19 references, cited by 73 papers here."
authors: ["Julen Etxaniz", "Gorka Azkune", "Aitor Soroa", "Oier Lopez de Lacalle", "Mikel Artetxe"]
year: 2024
venue: "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)"
source: "https://aclanthology.org/2024.naacl-short.46/"
doi: "10.18653/v1/2024.naacl-short.46"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/etxaniz-2024-do-multilingual-language-models-think/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/etxaniz-2024-do-multilingual-language-models-think/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Do Multilingual Language Models Think Better in English?

# Do Multilingual Language Models Think Better in English?

**Authors:** [Julen Etxaniz](https://md.research.elamur.ai/free/en/authors/julen-etxaniz/), [Gorka Azkune](https://md.research.elamur.ai/free/en/authors/gorka-azkune/), [Aitor Soroa](https://md.research.elamur.ai/free/en/authors/aitor-soroa/), [Oier Lopez de Lacalle](https://md.research.elamur.ai/free/en/authors/oier-lopez-de-lacalle/) and [Mikel Artetxe](https://md.research.elamur.ai/free/en/authors/mikel-artetxe/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2024](https://md.research.elamur.ai/free/en/venues/naacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.naacl-short.46/) · [PDF](https://aclanthology.org/2024.naacl-short.46.pdf) · [DOI 10.18653/v1/2024.naacl-short.46](https://doi.org/10.18653/v1/2024.naacl-short.46) · [OpenAlex](https://openalex.org/W4401042751)

**Citations:** 25 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/etxaniz-2024-do-multilingual-language-models-think/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Translate-test (machine-translating non-English inputs into English before querying a model) often helps multilingual language models. However, the external MT system is trained on large amounts of parallel data the model never saw, so the source of the gain is unclear. The authors isolate the effect of translation with **self-translate**: the same model first translates the input into English using a 4-shot prompt built from FLORES-200 examples, then solves the task on its own translation. They test XGLM (564M–7.5B) and LLaMA (7B–30B) on XCOPA, XStoryCloze, XNLI, PAWS-X and MGSM. They also compare against the external NLLB-200 3.3B translator. Self-translate beats direct inference in the original language on average for every model. The gains are larger for bigger models and high-resource languages. NLLB still does better, but the difference shrinks with scale. The results show that multilingual models have capabilities they cannot fully use when prompted in non-English languages, even though substantial cross-lingual transfer takes place.

## Key points
- **Setup:** 7 base models from two families, with extra results for BLOOM, LLaMA 2, OpenLLaMA, RedPajama and PolyLM in the appendix.
  - Translation uses greedy decoding, and each input field (e.g., the XNLI premise and hypothesis) is translated separately.
  - MGSM uses 8-shot chain-of-thought. The other tasks are scored zero-shot by picking the most probable candidate.
- **Main gains:** LLaMA 30B rises from 43.1 to 46.6 average accuracy (+3.5), with +6.5 on XNLI. XGLM 7.5B improves from 43.4 to 45.7. The smallest XGLM gains only +0.3 and drops on some tasks, which the authors attribute to its weak translation ability.
- **Cross-lingual transfer:** LLaMA outperforms XGLM on MGSM despite trailing it on the other tasks. This indicates that math skills carry over across languages, although the transfer is incomplete.
- **Comparison with NLLB:** NLLB-200 gives higher downstream accuracy and COMET scores than self-translate. The gap narrows as the models grow larger.
- **Limitations:**
  - Self-translate is slower because it adds a translation step.
  - Only base models were studied.
  - All benchmarks are human-translated, which may introduce evaluation artifacts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of the Basque Country](https://md.research.elamur.ai/free/en/universities/university-of-the-basque-country/) — University, Leioa, Spain · 40 works · 16,234 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works

## Cite

Cite as: Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle and Mikel Artetxe. 2024. Do Multilingual Language Models Think Better in English?. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{etxaniz2024do,
  title = {{Do Multilingual Language Models Think Better in English?}},
  author = {Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez {de Lacalle} and Mikel Artetxe},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-short.46},
  url = {https://doi.org/10.18653/v1/2024.naacl-short.46}
}
```

- [← previous Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://md.research.elamur.ai/free/en/papers/li-2024-eliciting-translation-ability-large-language/)
- [next → How do Large Language Models Handle Multilingualism?](https://md.research.elamur.ai/free/en/papers/zhao-2024-large-language-models-handle-multilingualism/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv. [find](https://scholar.google.com/scholar?q=Amr+Hendy%2C+Mohamed+Abdelrehim%2C+Amr+Sharaf%2C+Vikas+Raunak%2C+Mohamed+Gabr%2C+Hitokazu+Matsushita%2C+Young+Jin+Kim%2C+Mohamed+Afify%2C+and+Hany+Hassan+Awadalla%2E+2023%2E+How+good+are+gpt+models+at+machine+translation)
2. Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. [find](https://scholar.google.com/scholar?q=Kabir+Ahuja%2C+Rishav+Hada%2C+Millicent+Ochieng%2C+Prachi+Jain%2C+Harshita+Diddee%2C+Samuel+Maina%2C+Tanuja+Ganu%2C+Sameer+Segal%2C+Maxamed+Axmed%2C+Kalika+Bali%2C+et+al%2E)
3. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. [Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting](https://md.research.elamur.ai/free/en/papers/huang-2023-all-languages-created-equal-llms/). arXiv.
4. [Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano…](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
5. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
6. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. [Xcopa: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). arXiv.
7. Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulić, and Siva Reddy. 2021. Modelling latent translations for cross-lingual transfer. arXiv. [find](https://scholar.google.com/scholar?q=Edoardo+Maria+Ponti%2C+Julia+Kreutzer%2C+Ivan+Vulic%CC%81%2C+and+Siva+Reddy%2E+2021%2E+Modelling+latent+translations+for+cross%2Dlingual+transfer%2E+arXiv%2E)
8. Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv. [find](https://scholar.google.com/scholar?q=Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023b%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv%2E)
9. David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023. Prompting PaLM for translation: Assessing strategies and performance. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15406– 154… [find](https://scholar.google.com/scholar?q=David+Vilar%2C+Markus+Freitag%2C+Colin+Cherry%2C+Jiaming+Luo%2C+Viresh+Ratnakar%2C+and+George+Foster%2E+2023%2E+Prompting+PaLM+for+translation%3A+Assessing+strategies+and+performance%2E+In+Proceedings+of+the+61st+Annua)
10. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Transactions on Ma…
11. Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023. Polylm: An open source polyglot large language model. arXiv. [find](https://scholar.google.com/scholar?q=Xiangpeng+Wei%2C+Haoran+Wei%2C+Huan+Lin%2C+Tianhao+Li%2C+Pei+Zhang%2C+Xingzhang+Ren%2C+Mei+Li%2C+Yu+Wan%2C+Zhiwei+Cao%2C+Binbin+Xie%2C+et+al%2E+2023%2E+Polylm%3A+An+open+source+polyglot+large+language+model%2E+arXiv%2E)
12. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. [Paws-x: A cross-lingual adversarial dataset for paraphrase identification](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/). arXiv.
13. Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. 2022. [Comet-22: Unbabel-ist 2022 submission for the metrics shared task](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–58…
14. Machel Reid and Mikel Artetxe. 2023. On the role of parallel data in cross-lingual transfer learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5999–6006, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Machel+Reid+and+Mikel+Artetxe%2E+2023%2E+On+the+role+of+parallel+data+in+cross%2Dlingual+transfer+learning%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+5999%E2%80%936006%2C+Toronto%2C+)
15. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. arXiv. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2023%2E+Bloom%3A+A+176b%2Dparameter)
16. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). arXiv.
17. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv.
18. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. [Prompting large language model for machine translation: A case study](https://md.research.elamur.ai/free/en/papers/zhang-2023-prompting-large-language-model-machine/).
19. Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023b. Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7915–7927. [find](https://scholar.google.com/scholar?q=Xiang+Zhang%2C+Senyu+Li%2C+Bradley+Hauer%2C+Ning+Shi%2C+and+Grzegorz+Kondrak%2E+2023b%2E+Don%E2%80%99t+trust+chatgpt+when+your+question+is+not+in+english%3A+A+study+of+multilingual+abilities+and+types+of+llms%2E+In+Proceedin)

*Updated 24 October 2026*