---
title: "English is Not All You Need — paper by Dhaliwal et al. (2026)"
description: "Most LLM post-training is still done almost entirely in English. 23 references. Among them: 0."
authors: ["Mehak Dhaliwal", "Shashwat Chaurasia", "Yao Qin", "Dezhi Hong", "Thomas Butler"]
year: 2026
venue: "arXiv:2604.13286"
source: "https://arxiv.org/abs/2604.13286"
doi: "10.48550/arxiv.2604.13286"
arxiv: "2604.13286"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dhaliwal-2026-english-all-need-systematically-exploring/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dhaliwal-2026-english-all-need-systematically-exploring/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/28/) / English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training

# English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training

**Authors:** Mehak Dhaliwal, Shashwat Chaurasia, Yao Qin, [Dezhi Hong](https://md.research.elamur.ai/free/en/authors/dezhi-hong/) and [Thomas Butler](https://md.research.elamur.ai/free/en/authors/thomas-butler/) · **Year:** 2026 · **Venue:** arXiv:2604.13286 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2604.13286) · [PDF](https://arxiv.org/pdf/2604.13286) · [DOI 10.48550/arxiv.2604.13286](https://doi.org/10.48550/arxiv.2604.13286) · [arXiv 2604.13286](https://arxiv.org/abs/2604.13286) · [OpenAlex](https://openalex.org/W7154616075)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Most LLM post-training is still done almost entirely in English. This paper asks whether that choice costs performance, in English and in other languages, and how the answer changes with model size and task type. The authors ran 220 supervised fine-tuning runs on Qwen-3 (0.6B, 1.7B, 8B) and Gemma-3 (1B, 4B). They used 22 training mixtures of up to ten languages and covered two tasks: math reasoning and API calling. The mixtures are built from parallel translations, so each language contributes the same task content and the same amount of data. This keeps the effect of language diversity separate from the effect of more data.

The main findings:
- English-only tuning is usually worse than adding even one more language, and the extra language helps English too.
- Adding more languages tends to help low-resource languages, while high-resource ones level off rather than decline.
- Diverse mixtures transfer well zero-shot to languages not seen in training, except for distant, low-resource languages.

The work matters because it gives practitioners a cheap lever: translate existing task data.

## Key points
- **Experimental design.** There are 22 mixtures × 2 tasks × 5 models = 220 runs. Five core languages appear in exhaustive combinations, and five extra languages extend mixtures to ten. Evaluation covers six languages that differ in resource level, family, region and script.
- **Math task.**
  - Training uses mCoT-MATH: 10,000 parallel examples per language plus 200 for validation, with a step-by-step prompt in each language.
  - Testing uses MGSM, a set of 250 human-translated grade-school problems.
  - Scoring extracts answers with a regex and normalizes numbers with Python's Babel.
- **New dataset.** The authors release mAPICall-Bank, a multilingual API-calling dataset covering 11 languages and derived from API-Bank. Its training split has 3,174 queries.
- **Pooled regression.**
  - The model uses a square-root-transformed language-coverage term and reaches an out-of-sample R² of 80.5%.
  - Wider coverage still helps after controlling for whether the test language was included in training (β = 0.053, 95% CI [0.018, 0.089]).
- **Exception to "more languages helps."** Very small models (≤1B) on API calling, where limiting the number of languages is advised. Beyond that, the authors see hardly any sign of a "curse of multilinguality" in mixtures with as many as ten languages.

## Interesting ideas
- Keep the amount of data per source constant (parallel copies) while varying only how many sources are mixed, so that diversity is not confused with volume.
- Build mixtures from every combination of a small core set plus a fixed sequence of add-ons, so the result depends on more than one lucky ordering.
- Summarize many runs with a pooled regression that controls for an obvious confound (direct inclusion), and report held-out R² and bootstrap confidence intervals.
- Normalize locale-specific number formats with a dedicated parser before comparing the predicted answer with the gold answer.
- When covering every target is impossible, train on a diverse set, but still collect some direct data for distant, low-resource targets, because transfer to them stays weak.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): When testing whether adding more witness parsers or LLM teacher tiers improves silver data, fix the sentence count and vary only the number of witnesses, so any gain can be credited to diversity.
- **I2** (learn): Ablate self-induced repair rules or teacher sources this way to see which combinations, not just which single additions, actually help.
- **I3** (learn): Fit a similar model over evaluation runs, with factors such as treebank source, domain and parser type, to tell real effects from noise alongside the negative controls.
- **I4** (answer): In maths word problem tests, canonicalize numerals, separators and spelled-out numbers before scoring, so formatting differences are not counted as reasoning errors.
- **I5** (tree): Train gparse on a varied mix of UD English treebanks and genres, and add a small in-domain sample for genres far from that mix, such as fables and tales.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Mehak Dhaliwal, Shashwat Chaurasia, Yao Qin, Dezhi Hong and Thomas Butler. 2026. English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training. arXiv:2604.13286.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{dhaliwal2026english,
  title = {{English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training}},
  author = {Mehak Dhaliwal, Shashwat Chaurasia, Yao Qin, Dezhi Hong and Thomas Butler},
  year = {2026},
  doi = {10.48550/arxiv.2604.13286},
  eprint = {2604.13286},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2604.13286}
}
```

- [← previous English Grammar (Wikibooks)](https://md.research.elamur.ai/free/en/papers/contributors-2026-english-grammar/)
- [next → English to Central Kurdish Speech Translation: Corpus Creation, Evaluation, and Orthographic Standardization](https://md.research.elamur.ai/free/en/papers/mohammadamini-2026-english-central-kurdish-speech-translation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/28/)

## References

23 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tyler A Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. When is multilinguality a curse? language modeling for 250 high-and low-resource languages. In Proceedings of the 2024 [Conference on Empirical Methods in Natural Language Processing, pp](https://md.research.elamur.ai/free/en/papers/chang-2024-when-is-multilinguality-curse-language/). 4074–4096, 2024.
2. Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Andrey Kutuzov, Barry Haddow, and Kenneth Heafield. Monolingual or multilingual instruction tuning: Which makes a better alpaca. In Findings of the Association for Computational Linguistics: EACL 2024, pp. 1347–1356, 2024. [find](https://scholar.google.com/scholar?q=Pinzhen+Chen%2C+Shaoxiong+Ji%2C+Nikolay+Bogoychev%2C+Andrey+Kutuzov%2C+Barry+Haddow%2C+and+Kenneth+Heafield%2E+Monolingual+or+multilingual+instruction+tuning%3A+Which+makes+a+better+alpaca%2E+In+Findings+of+the+Assoc)
3. Nadezhda Chirkova and Vassilina Nikoulina. Zero-shot cross-lingual transfer in instruction tuning of large language models. In Proceedings of the 17th International Natural Language Generation Conference, pp. 695–708, 2024. [find](https://scholar.google.com/scholar?q=Nadezhda+Chirkova+and+Vassilina+Nikoulina%2E+Zero%2Dshot+cross%2Dlingual+transfer+in+instruction+tuning+of+large+language+models%2E+In+Proceedings+of+the+17th+International+Natural+Language+Generation+Confere)
4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 8440–8451, 2020. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Unsupervised+cross%2Dlingual+re)
5. Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. [Do multilingual language models think better in english?](https://md.research.elamur.ai/free/en/papers/etxaniz-2024-do-multilingual-language-models-think/) In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pp. 550–564, 2024. [↗](https://doi.org/10.48550/arxiv.2308.01223)
6. Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, and Fei Yuan. Benchmax: A comprehensive multilingual evaluation suite for large language models. arXiv preprint arXiv:2502.07346, 2025. [↗](https://arxiv.org/abs/2502.07346)
7. Shaoxiong Ji and Pinzhen Chen. How many languages make good multilingual instruction tuning? a case study on bloom. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 2575–2581, 2025. [↗](https://doi.org/10.48550/arxiv.2404.04850)
8. Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 4, 2025. [↗](https://arxiv.org/abs/2503.19786)
9. Tannon Kew, Florian Schottmann, and Rico Sennrich. Turning english-centric llms into polyglots: How much multilinguality is needed? In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 13097–13124, 2024. [find](https://scholar.google.com/scholar?q=Tannon+Kew%2C+Florian+Schottmann%2C+and+Rico+Sennrich%2E+Turning+english%2Dcentric+llms+into+polyglots%3A+How+much+multilinguality+is+needed%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+)
10. Simran Khanuja, Sebastian Ruder, and Partha Talukdar. Evaluating the diversity, equity, and inclusion of nlp technology: A case study for indian languages. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 1763–1777, 2023. [find](https://scholar.google.com/scholar?q=Simran+Khanuja%2C+Sebastian+Ruder%2C+and+Partha+Talukdar%2E+Evaluating+the+diversity%2C+equity%2C+and+inclusion+of+nlp+technology%3A+A+case+study+for+indian+languages%2E+In+Findings+of+the+Association+for+Computati)
11. Mayank Kulkarni, Vittorio Mazzia, Judith Gaspers, Christopher Hench, Jack FitzGerald, and AGI Amazon. Massive-agents: A benchmark for multilingual function-calling in 52 languages. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 20193–20215, 2025. [find](https://scholar.google.com/scholar?q=Mayank+Kulkarni%2C+Vittorio+Mazzia%2C+Judith+Gaspers%2C+Christopher+Hench%2C+Jack+FitzGerald%2C+and+AGI+Amazon%2E+Massive%2Dagents%3A+A+benchmark+for+multilingual+function%2Dcalling+in+52+languages%2E+In+Findings+of+the+)
12. Huiyuan Lai and Malvina Nissim. [mcot: Multilingual instruction tuning for reasoning consistency in language models](https://md.research.elamur.ai/free/en/papers/lai-2024-mcot-multilingual-instruction-tuning-reasoning/). arXiv preprint arXiv:2406.02301, 2024. [↗](https://arxiv.org/abs/2406.02301)
13. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. [Api-bank: A comprehensive benchmark for tool-augmented llms](https://md.research.elamur.ai/free/en/papers/li-2023-api-bank-comprehensive-benchmark-tool/). arXiv preprint arXiv:2304.08244, 2023. [↗](https://arxiv.org/abs/2304.08244)
14. Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, et al. [Choosing transfer languages for cross-lingual learning](https://md.research.elamur.ai/free/en/papers/lin-2019-choosing-transfer-languages-cross-lingual/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3125–3135, 2019.
15. Shayne Longpre, Sneha Kudugunta, Niklas Muennighoff, I Hsu, Isaac Caswell, Alex Pentland, Sercan Arik, Chen-Yu Lee, Sayna Ebrahimi, et al. Atlas: Adaptive transfer scaling laws for multilingual pretraining, finetuning, and decoding the curse of multilinguality. arXiv preprint arXiv:2510.22037, 2025. [↗](https://arxiv.org/abs/2510.22037)
16. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, et al. [Crosslingual generalization through multitask finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/). In Proceedings of the 61st annual meeting of the Association for Computational Linguistics (volume 1: Long papers), pp. 15991– 16111, 2023.
17. Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). Frontiers of Computer Science, 19(8):198343, 2025. [↗](https://doi.org/10.1007/s11704-024-40678-2)
18. Lisa Schut, Yarin Gal, and Sebastian Farquhar. Do multilingual llms think in english? arXiv preprint arXiv:2502.15603, 2025. [↗](https://arxiv.org/abs/2502.15603)
19. Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. [Multilingual instruction tuning with just a pinch of multilinguality](https://md.research.elamur.ai/free/en/papers/shaham-2024-multilingual-instruction-tuning-just-pinch/). arXiv preprint arXiv:2401.01854, 2024. [↗](https://arxiv.org/abs/2401.01854)
20. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. [Language models are multilingual chain-of-thought reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/). arXiv preprint arXiv:2210.03057, 2022. [↗](https://arxiv.org/abs/2210.03057)
21. Luisa Shimabucoro, Ahmet Ustun, Marzieh Fadaee, and Sebastian Ruder. A post-trainer’s guide to multilingual training data: Uncovering cross-lingual transfer dynamics. arXiv preprint arXiv:2504.16677, 2025. [↗](https://arxiv.org/abs/2504.16677)
22. Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, and Kristina Toutanova. Revisiting the primacy of english in zero-shot cross-lingual transfer. arXiv preprint arXiv:2106.16171, 2021. [↗](https://arxiv.org/abs/2106.16171)
23. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388, 2025. *Figure 5: Example showing the final turns of a parallel multi-turn API-calling interaction from mAPICall-Bank in three languages: (a) English, (b) Spanish, and (c) Bengali.* *Figure 6: Prompt used to create mAPICall-Bank* [↗](https://arxiv.org/abs/2505.09388)

*Updated 1 November 2026*