---
title: "On the Multilingual Ability of Decoder-based Pre-trained"
description: "Decoder-only pre-trained language models often perform well across languages, but it is unclear how… 33 references, cited by 62 papers here."
authors: ["Takeshi Kojima", "Itsuki Okimura", "Yusuke Iwasawa", "Hitomi Yanaka", "Yutaka Matsuo"]
year: 2024
venue: "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.naacl-long.384/"
doi: "10.18653/v1/2024.naacl-long.384"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kojima-2024-multilingual-ability-decoder-based-pre/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kojima-2024-multilingual-ability-decoder-based-pre/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons

# On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons

**Authors:** [Takeshi Kojima](https://md.research.elamur.ai/free/en/authors/takeshi-kojima/), Itsuki Okimura, [Yusuke Iwasawa](https://md.research.elamur.ai/free/en/authors/yusuke-iwasawa/), [Hitomi Yanaka](https://md.research.elamur.ai/free/en/authors/hitomi-yanaka/) and [Yutaka Matsuo](https://md.research.elamur.ai/free/en/authors/yutaka-matsuo/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2024](https://md.research.elamur.ai/free/en/venues/naacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.naacl-long.384/) · [PDF](https://aclanthology.org/2024.naacl-long.384.pdf) · [DOI 10.18653/v1/2024.naacl-long.384](https://doi.org/10.18653/v1/2024.naacl-long.384) · [OpenAlex](https://openalex.org/W4401042559)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/kojima-2024-multilingual-ability-decoder-based-pre/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Decoder-only pre-trained language models often perform well across languages, but it is unclear how they handle multilingualism internally. Earlier neuron-level work mostly studied encoder models such as M-BERT and looked for neurons shared across languages. This paper instead looks for neurons tied to one particular language in XGLM, BLOOM and Llama2, covering English, German, French, Spanish, Chinese and Japanese. The authors adapt the method of Cuadros et al. (2022). Texts in the target language are treated as positive examples and texts in the other languages as negative. Each neuron's activation is averaged over tokens and scored by average precision. Both the highest- and lowest-ranked neurons are kept as language-specific. These neurons overlap little across languages and are concentrated mainly in a handful of the earliest and final layers, a pattern that holds for every model and language. Fixing their outputs to the median value seen on target-language text strongly shifts which language the model generates. This also helps on translation prompts that do not name the target language. The findings support future work on language-aware model compression and fine-tuning.

## Key points
- **Models:** XGLM (564M, 1.7B, 2.9B), BLOOM (560M, 1.7B, 3B) and Llama2 (7B, 13B). Llama2 was trained mostly on English data.
- **Data:** The corpus combines PAWS-X and FLORES-200 in equal proportions, with 500 texts per language. The input and output embedding layers were excluded from the analysis.
- **Neuron selection:** For each language, the top-1000 and bottom-1000 neurons by average precision are selected. Together they make up under 1% of all neurons. Cross-language overlap stays below 5%; for example, the largest pairwise overlap in XGLM-564M is 93 of 2000 neurons.
- **Translation results:** With Llama2-7b and a translation prompt that leaves the target language unspecified, intervention raised target-language accuracy from roughly 0–5% to about 55–79%. BLEU scores rose as well.
- **Explicit prompts:** When the prompt named German or Japanese as the target, intervention changed little. The exception was Chinese neurons under the Japanese prompt, possibly because Llama2-7b shares more neurons between Chinese and Japanese.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 465 works

## Cite

Cite as: Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka and Yutaka Matsuo. 2024. On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kojima2024multilingual,
  title = {{On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons}},
  author = {Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka and Yutaka Matsuo},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.384},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.384}
}
```

- [← previous GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot](https://md.research.elamur.ai/free/en/papers/zeng-2024-glm-4-voice-intelligent/)
- [next → Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-agent-flan-designing-data-methods/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

33 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omer Antverg and Yonatan Belinkov. 2022. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Omer+Antverg+and+Yonatan+Belinkov%2E+2022%2E+On+the+pitfalls+of+analyzing+individual+neurons+in+language+models%2E+In+International+Conference+on+Learning+Representations%2E)
2. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Anthony+Bau%2C+Yonatan+Belinkov%2C+Hassan+Sajjad%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+and+James+Glass%2E+2019%2E+Identifying+and+controlling+important+neurons+in+neural+machine+translation%2E+In+International+Conferenc)
3. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.". [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Ewan+Klein%2C+and+Edward+Loper%2E+2009%2E+Natural+language+processing+with+Python%3A+analyzing+text+with+the+natural+language+toolkit%2E+%22+O%E2%80%99Reilly+Media%2C+Inc%2E%22%2E)
4. Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi,… — [Findings of the 2016 Conference on Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/)
5. Ond rej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. [Findings of the 2018 conference on machine translation (wmt18)](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/). In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 272–…
6. Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. 2014. [Findings of the 2014 workshop on statistical machine translation](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/). In Proceedings of the…
7. Eleftheria Briakou, Colin Cherry, and George Foster. 2023. Searching for needles in a haystack: On the role of incidental bilingualism in PaLM’s translation capability. arXiv preprint arXiv:2305.10266. [find](https://scholar.google.com/scholar?q=Eleftheria+Briakou%2C+Colin+Cherry%2C+and+George+Foster%2E+2023%2E+Searching+for+needles+in+a+haystack%3A+On+the+role+of+incidental+bilingualism+in+PaLM%E2%80%99s+translation+capability%2E+arXiv+preprint+arXiv%3A2305%2E10266)
8. Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker Sebastian, Sudoh Katsuitho, Yoshino Koichiro, and Federmann Christian. 2017. [Overview of the iwslt 2017 evaluation campaign](https://md.research.elamur.ai/free/en/papers/cettolo-2017-overview-iwslt-2017-evaluation-campaign/). In Proceedings of the 14th International Workshop on Spoken Language Translation, pages 2–14.
9. Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2023. Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons. arXiv preprint arXiv:2308.13198. — [arXiv:2308.13198](https://arxiv.org/abs/2308.13198) [find](https://scholar.google.com/scholar?q=Yuheng+Chen%2C+Pengfei+Cao%2C+Yubo+Chen%2C+Kang+Liu%2C+and+Jun+Zhao%2E+2023%2E+Journey+to+the+center+of+the+knowledge+neurons%3A+Discoveries+of+language%2Dindependent+knowledge+neurons+and+degenerate+knowledge+neuron)
10. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+2022%2E+No+language+left+behind%3A+Scaling+h)
11. Xavier Suau Cuadros, Luca Zappella, and Nicholas Apostoloff. 2022. Self-conditioning pre-trained language models. In International Conference on Machine Learning, pages 4455–4473. PMLR. [find](https://scholar.google.com/scholar?q=Xavier+Suau+Cuadros%2C+Luca+Zappella%2C+and+Nicholas+Apostoloff%2E+2022%2E+Self%2Dconditioning+pre%2Dtrained+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+4455%E2%80%934473%2E+PMLR%2E)
12. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493– 8502, Dublin, Ireland. Association for Computatio… [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Li+Dong%2C+Yaru+Hao%2C+Zhifang+Sui%2C+Baobao+Chang%2C+and+Furu+Wei%2E+2022%2E+Knowledge+neurons+in+pretrained+transformers%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computationa)
13. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jegou, and Tomas Mikolov. 2017a. [Fasttext.zip: Compressing text classification models](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/).
14. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017b. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Associati… [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2017b%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associat)
15. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. [Few-shot learning with multilingual language models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/). arXiv preprint arXiv:2112.10668.
16. Jesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153–17163. [find](https://scholar.google.com/scholar?q=Jesse+Mu+and+Jacob+Andreas%2E+2020%2E+Compositional+explanations+of+neurons%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A17153%E2%80%9317163%2E)
17. Aaron Mueller, Yu Xia, and Tal Linzen. 2022. Causal analysis of syntactic agreement neurons in multilingual language models. arXiv preprint arXiv:2210.14328. [find](https://scholar.google.com/scholar?q=Aaron+Mueller%2C+Yu+Xia%2C+and+Tal+Linzen%2E+2022%2E+Causal+analysis+of+syntactic+agreement+neurons+in+multilingual+language+models%2E+arXiv+preprint+arXiv%3A2210%2E14328%2E)
18. Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. 2021. [First align, then predict: Understanding the cross-lingual ability of multilingual BERT](https://md.research.elamur.ai/free/en/papers/muller-2021-first-align-then-predict-understanding/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2214–2231,…
19. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, pages 2303–08774.
20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
21. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. [How multilingual is multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
22. Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. 2022. [Neuron-level interpretation of deep nlp models: A survey](https://md.research.elamur.ai/free/en/papers/sajjad-2022-neuron-level-interpretation-deep-nlp/). Transactions of the Association for Computational Linguistics, 10:1285–1303.
23. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
24. Karolina Stańczak, Lucas Torroba Hennigen, Adina Williams, Ryan Cotterell, and Isabelle Augenstein. 2023. A latent-variable model for intrinsic probing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13591–13599. [find](https://scholar.google.com/scholar?q=Karolina+Stan%CC%81czak%2C+Lucas+Torroba+Hennigen%2C+Adina+Williams%2C+Ryan+Cotterell%2C+and+Isabelle+Augenstein%2E+2023%2E+A+latent%2Dvariable+model+for+intrinsic+probing%2E+In+Proceedings+of+the+AAAI+Conference+on+Artif)
25. Karolina Stańczak, Edoardo Ponti, Lucas Torroba Hennigen, Ryan Cotterell, and Isabelle Augenstein. 2022. Same neurons, different languages: Probing morphosyntax in multilingual pre-trained models. arXiv preprint arXiv:2205.02023. [find](https://scholar.google.com/scholar?q=Karolina+Stan%CC%81czak%2C+Edoardo+Ponti%2C+Lucas+Torroba+Hennigen%2C+Ryan+Cotterell%2C+and+Isabelle+Augenstein%2E+2022%2E+Same+neurons%2C+different+languages%3A+Probing+morphosyntax+in+multilingual+pre%2Dtrained+models%2E+ar)
26. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
27. Andrea Gregor de Varda and Marco Marelli. 2023. Data-driven cross-lingual syntax: An agreement study with massively multilingual models. Computational Linguistics, 49(2):261–299. [find](https://scholar.google.com/scholar?q=Andrea+Gregor+de+Varda+and+Marco+Marelli%2E+2023%2E+Data%2Ddriven+cross%2Dlingual+syntax%3A+An+agreement+study+with+massively+multilingual+models%2E+Computational+Linguistics%2C+49%282%29%3A261%E2%80%93299%2E)
28. Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. 2022. [Finding skill neurons in pre-trained transformer-based language models](https://md.research.elamur.ai/free/en/papers/wang-2022-finding-skill-neurons-pre-trained/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11132–11152, Abu Dhabi, United Arab Emirates.…
29. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. [Do Llamas work in English? On the latent language of multilingual transformers](https://md.research.elamur.ai/free/en/papers/wendler-2024-do-llamas-work-english-latent/). arXiv preprint arXiv:2402.10588.
30. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. [CCNet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003–4012, Mar…
31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. [Huggingface’s transformers: State-ofthe-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771.
32. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
33. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. [PAWS-X: A cross-lingual adversarial dataset for paraphrase identification](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin…

*Updated 24 October 2026*