---
title: "Beyond English-Centric LLMs — paper by Zhong et al. (2024)"
description: "This study asks which language non-English-centric LLMs rely on in their hidden layers. 21 references, cited by 22 papers here."
authors: ["Chengzhi Zhong", "Fei Cheng", "Qianying Liu", "Junfeng Jiang", "Zhen Wan", "Chenhui Chu", "Yugo Murawaki", "Sadao Kurohashi"]
year: 2024
venue: "arXiv:2408.10811"
source: "https://arxiv.org/abs/2408.10811"
doi: "10.48550/arxiv.2408.10811"
arxiv: "2408.10811"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhong-2024-english-centric-llms-language-multilingual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhong-2024-english-centric-llms-language-multilingual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?

# Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?

**Authors:** Chengzhi Zhong, [Fei Cheng](https://md.research.elamur.ai/free/en/authors/fei-cheng/), [Qianying Liu](https://md.research.elamur.ai/free/en/authors/qianying-liu/), Junfeng Jiang, [Zhen Wan](https://md.research.elamur.ai/free/en/authors/zhen-wan/), [Chenhui Chu](https://md.research.elamur.ai/free/en/authors/chenhui-chu/), [Yugo Murawaki](https://md.research.elamur.ai/free/en/authors/yugo-murawaki/) and [Sadao Kurohashi](https://md.research.elamur.ai/free/en/authors/sadao-kurohashi/) · **Year:** 2024 · **Venue:** arXiv:2408.10811 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2408.10811) · [PDF](https://arxiv.org/pdf/2408.10811) · [DOI 10.48550/arxiv.2408.10811](https://doi.org/10.48550/arxiv.2408.10811) · [arXiv 2408.10811](https://arxiv.org/abs/2408.10811) · [OpenAlex](https://openalex.org/W4403007390)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [22](https://md.research.elamur.ai/free/en/papers/zhong-2024-english-centric-llms-language-multilingual/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This study asks which language non-English-centric LLMs rely on in their hidden layers. Earlier work found that Llama-2 routes processing through English. The authors focus on Japanese and compare three model types. Llama-2 is English-centric. Swallow is Llama-2 with continued pre-training (CPT) on Japanese. LLM-jp was trained from scratch on a balanced English–Japanese corpus. Using the logit lens, they project each intermediate layer onto the vocabulary. They extend the single-token analysis of Wendler et al. (2024) to multi-token words, which suits Japanese script. Tasks include translation between French, Chinese, English and Japanese, plus culturally loaded questions. Llama-2 reasons via English alone. Swallow and LLM-jp use Japanese when producing Japanese. For other target languages, they mix English and Japanese, favouring whichever is closer to the target. Culture-specific answers can pass through the latent language's cultural frame before the final layers correct them. The switch to the output language appears confined to a few dimensions. These results help explain why language-specific models do well in their dominant language. They also matter for multilingual model design and knowledge editing.

## Key points
- Models compared: Llama-2, Swallow (Japanese CPT on Llama-2) and LLM-jp (bilingual pre-training from scratch), analysed with the logit lens.
- Multi-token words are scored by chaining token probabilities, e.g. P(x1)·P(x2|x1). Two-character phrases are used to avoid ambiguity between Chinese and Japanese characters.
- For French and Chinese targets, Swallow keeps both latent languages active with English generally stronger. LLM-jp tends to lean heavily toward a single one.
- On cultural questions, such as the month the school year starts, Llama-2's middle layers favour the American answer (September). The correct Japanese answer emerges only in late layers, and LLM-jp largely avoids this error.
- With 30 English–Japanese word pairs in Swallow (13B), the authors averaged the hidden-state difference between layers 26 and 40. Adding this shift turned English top tokens into Japanese ones, and the change sat in sparse, language-identity dimensions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Tokyo](https://md.research.elamur.ai/free/en/universities/the-university-of-tokyo/) — University, Tokyo, Japan · 96 works · 9,600 citations
- [Kyoto University](https://md.research.elamur.ai/free/en/universities/kyoto-university/) — University, Kyoto, Japan · 42 works · 8,235 citations
- [National Institute of Informatics](https://md.research.elamur.ai/free/en/labs/national-institute-of-informatics/) — Research institute, Tokyo, Japan · 38 works · 3,247 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,379 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 882 works

## Cite

Cite as: Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki and Sadao Kurohashi. 2024. Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?. arXiv:2408.10811.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhong2024beyond,
  title = {{Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?}},
  author = {Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki and Sadao Kurohashi},
  year = {2024},
  doi = {10.48550/arxiv.2408.10811},
  eprint = {2408.10811},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.10811}
}
```

- [← previous Assessing the Capabilities of Large Language Models in Coreference: An Evaluation](https://md.research.elamur.ai/free/en/papers/gan-2024-assessing-capabilities-large-language-models/)
- [next → Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models](https://md.research.elamur.ai/free/en/papers/kim-2024-developing-pragmatic-benchmark-assessing-korean/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazuki Fujii, Kensuke Fukumoto, Takuya Fukushima, Namgi Han, et al. 2024. [Llm-jp: A cross-organizational project Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman,…](https://md.research.elamur.ai/free/en/papers/belrose-2023-eliciting-latent-predictions-transformers-tuned/)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
4. Zoltan Csaki, Bo Li, Jonathan Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024. Sambalingo: Teaching large language models new languages. Preprint, arXiv:2404.05829. [find](https://scholar.google.com/scholar?q=Zoltan+Csaki%2C+Bo+Li%2C+Jonathan+Li%2C+Qiantong+Xu%2C+Pian+Pawakapan%2C+Leon+Zhang%2C+Yun+Du%2C+Hengyu+Zhao%2C+Changran+Hu%2C+and+Urmish+Thakker%2E+2024%2E+Sambalingo%3A+Teaching+large+language+models+new+languages%2E+Preprin)
5. Yiming Cui, Ziqing Yang, and Xin Yao. 2023. [Efficient and effective text encoding for chinese llama and alpaca](https://md.research.elamur.ai/free/en/papers/cui-2023-efficient-effective-text-encoding-chinese/). arXiv preprint arXiv:2304.08177.
6. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. Preprint, arXiv:2208.07339. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+2022%2E+Llm%2Eint8%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+Preprint%2C+arXiv%3A2208%2E07339%2E)
7. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Tristan+Hume%2C+Catherine+Olsson%2C+Nicholas+Schiefer%2C+Tom+Henighan%2C+Shauna+Kravec%2C+Zac+Hatfield%2DDodds%2C+Robert+Lasenby%2C+Dawn+Drain%2C+Carol+Chen%2C+et+al%2E+2022%2E+Toy+models+of+superposition%2E+arX)
8. Manuel Faysse, Patrick Fernandes, Nuno Guerreiro, António Loison, Duarte Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro Martins, et al. 2024. Croissantllm: A truly bilingual french-english language model. arXiv preprint arXiv:2402.00786. [find](https://scholar.google.com/scholar?q=Manuel+Faysse%2C+Patrick+Fernandes%2C+Nuno+Guerreiro%2C+Ant%C3%B3nio+Loison%2C+Duarte+Alves%2C+Caio+Corro%2C+Nicolas+Boizard%2C+Jo%C3%A3o+Alves%2C+Ricardo+Rei%2C+Pedro+Martins%2C+et+al%2E+2024%2E+Croissantllm%3A+A+truly+bilingual+french)
9. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. [Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities](https://md.research.elamur.ai/free/en/papers/fujii-2024-continual-pre-training-cross-lingual/). arXiv preprint arXiv:2404.17790.
10. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Robert+Huben%2C+Hoagy+Cunningham%2C+Logan+Riggs+Smith%2C+Aidan+Ewart%2C+and+Lee+Sharkey%2E+2023%2E+Sparse+autoencoders+find+highly+interpretable+features+in+language+models%2E+In+The+Twelfth+International+Conferenc)
11. Julie Hunter, Jérôme Louradour, Virgile Rennard, Is-maïl Harrando, Guokan Shang, and Jean-Pierre Lorré. 2023. The claire french dialogue dataset. arXiv preprint arXiv:2311.16840. [find](https://scholar.google.com/scholar?q=Julie+Hunter%2C+J%C3%A9r%C3%B4me+Louradour%2C+Virgile+Rennard%2C+Is%2Dma%C3%AFl+Harrando%2C+Guokan+Shang%2C+and+Jean%2DPierre+Lorr%C3%A9%2E+2023%2E+The+claire+french+dialogue+dataset%2E+arXiv+preprint+arXiv%3A2311%2E16840%2E)
12. Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. [WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models](https://md.research.elamur.ai/free/en/papers/minixhofer-2022-wechsel-effective-initialization-subword-embeddings/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
13. Nostalgebraist. 2024. Interpreting gpt: The logit lens. [https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/) interpreting-gpt-the-logit-lens. Accessed: 2024-07-28. [↗](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/)
14. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023. [Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/). arXiv preprint arXiv:2308.16149.
15. Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2020. Lamol: Language modeling for lifelong language learning. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Fan%2DKeng+Sun%2C+Cheng%2DHao+Ho%2C+and+Hung%2DYi+Lee%2E+2020%2E+Lamol%3A+Language+modeling+for+lifelong+language+learning%2E+In+International+Conference+on+Learning+Representations%2E)
16. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
17. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
18. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kevin+Ro+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+2022%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+gpt%2D2+small%2E+In+The+Elevent)
19. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. [Do llamas work in english? on the latent language of multilingual transformers](https://md.research.elamur.ai/free/en/papers/wendler-2024-llamas-work-english-latent-language/). arXiv preprint arXiv:2402.10588.
20. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+et+al%2E+2024%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2E)
21. 松下 達彦, 陳 夢夏, 王 雪竹, and 陳 林柯. 2020. 日 中対照漢字語データベースの開発と応用. 日本 語教育, 177:62–76. [find](https://scholar.google.com/scholar?q=%E6%9D%BE%E4%B8%8B+%E9%81%94%E5%BD%A6%2C+%E9%99%B3+%E5%A4%A2%E5%A4%8F%2C+%E7%8E%8B+%E9%9B%AA%E7%AB%B9%2C+and+%E9%99%B3+%E6%9E%97%E6%9F%AF%2E+2020%2E+%E6%97%A5+%E4%B8%AD%E5%AF%BE%E7%85%A7%E6%BC%A2%E5%AD%97%E8%AA%9E%E3%83%87%E3%83%BC%E3%82%BF%E3%83%99%E3%83%BC%E3%82%B9%E3%81%AE%E9%96%8B%E7%99%BA%E3%81%A8%E5%BF%9C%E7%94%A8%2E+%E6%97%A5%E6%9C%AC+%E8%AA%9E%E6%95%99%E8%82%B2%2C+177%3A62%E2%80%9376%2E)

*Updated 2 November 2026*