---
title: "Language Ranker — paper by Li et al. (2024)"
description: "The authors propose Language Ranker, an intrinsic metric that ranks languages using a model's… 29 references, cited by 53 papers here."
authors: ["Zihao Li", "Yucheng Shi", "Zirui Liu", "Fan Yang", "Ali Payani", "Ninghao Liu", "Mengnan Du"]
year: 2024
venue: "arXiv:2404.11553"
source: "https://arxiv.org/abs/2404.11553"
doi: "10.48550/arxiv.2404.11553"
arxiv: "2404.11553"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2024-language-ranker-metric-quantifying-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2024-language-ranker-metric-quantifying-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages

# Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages

**Authors:** Zihao Li, Yucheng Shi, [Zirui Liu](https://md.research.elamur.ai/free/en/authors/zirui-liu/), [Fan Yang](https://md.research.elamur.ai/free/en/authors/fan-yang/), Ali Payani, [Ninghao Liu](https://md.research.elamur.ai/free/en/authors/ninghao-liu/) and [Mengnan Du](https://md.research.elamur.ai/free/en/authors/mengnan-du/) · **Year:** 2024 · **Venue:** [arXiv:2404.11553](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/) ([Proceedings of the AAAI Conference on Artificial Intelligence 2024](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/2024/)) · **Publisher:** [Association for the Advancement of Artificial Intelligence](https://md.research.elamur.ai/free/en/publishers/association-for-the-advancement-of-artificial-intelligence/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.11553) · [PDF](https://arxiv.org/pdf/2404.11553) · [DOI 10.48550/arxiv.2404.11553](https://doi.org/10.48550/arxiv.2404.11553) · [arXiv 2404.11553](https://arxiv.org/abs/2404.11553) · [OpenAlex](https://openalex.org/W4409362017)

**Citations:** 42 citations (OpenAlex, 2026-10-14) · [53](https://md.research.elamur.ai/free/en/papers/li-2024-language-ranker-metric-quantifying-llm/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

LLMs are trained on corpora dominated by English. For example, about 92.65% of GPT-3 tokens and 89.7% of LlaMa2 pre-training data are English. As a result, low-resource languages get weaker support, yet there are few quantitative ways to measure this gap. The authors propose Language Ranker, an intrinsic metric that ranks languages using a model's hidden states instead of task outputs. They take parallel sentence pairs from OPUS-100 and feed each English sentence and its non-English counterpart into the model. They then compare the last-token representations with cosine similarity, averaged over several intermediate layers. Languages are ordered by this score. Agreement between models' rankings is measured with a longest-common-ordered-sublist ratio. Across LlaMa2, LlaMa3, Qwen, Mistral-v0.1 and Gemma, high-resource languages consistently land closer to English than low-resource ones. The scores also track each language's share of pre-training data. This gives a cheap, task-independent way to diagnose multilingual bias. It could also guide the design of more balanced training corpora.

## Key points

- **Probing data:** OPUS-100 (English-centric), filtered to 94 language pairs with 2000 samples each. The pairs span high-resource (German, French, Chinese) and low-resource languages (Oriya, Kannada, Kazakh, Igbo, Turkmen).
- **Score:** cosine similarity of last-token representations, averaged over layers 5, 10, 15, 20 and 25. Major European languages typically score above 0.6, while low-resource ones often fall below 0.4.
- **Link to pre-training share (LlaMa2):**
  - German (0.17% of the corpus) scores 0.723 and French scores 0.737.
  - Welsh, Persian, Urdu and Kannada each make up ≤0.01% and score between 0.236 and 0.396.
- **Consistency across models:** 7B models produce largely consistent rankings, and the pattern holds for 13B LlaMa2 and Qwen. Qwen's extra Chinese training lifts Chinese to the level of other high-resource languages.
- **"Double variance" check:** this measures the spread of the top covariance eigenvalues of a language's embeddings. It shows that low-resource languages occupy narrow, unbalanced subspaces, with high double variance paired with low similarity.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Minnesota](https://md.research.elamur.ai/free/en/universities/university-of-minnesota/) — University, Minneapolis, United States · 9 works · 1,833 citations
- [University of Georgia](https://md.research.elamur.ai/free/en/universities/university-of-georgia/) — University, Athens, United States · 9 works · 26,161 citations
- [New Jersey Institute of Technology](https://md.research.elamur.ai/free/en/universities/new-jersey-institute-of-technology/) — University, Newark, United States · 4 works · 1,234 citations
- [Wake Forest University](https://md.research.elamur.ai/free/en/universities/wake-forest-university/) — University, Winston-Salem, United States · 3 works · 884 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🌱 Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) · 166 works

## Cite

Cite as: Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu and Mengnan Du. 2024. Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages. arXiv:2404.11553.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2024language,
  title = {{Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages}},
  author = {Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu and Mengnan Du},
  year = {2024},
  doi = {10.48550/arxiv.2404.11553},
  eprint = {2404.11553},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.11553}
}
```

- [← previous Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification](https://md.research.elamur.ai/free/en/papers/fadeeva-2024-fact-checking-output-large-language/)
- [next → LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models](https://md.research.elamur.ai/free/en/papers/han-2024-lm-infinite-zero-shot-extreme/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

29 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Ahuja, S.; Aggarwal, D.; Gumma, V.; Watts, I.; Sathe, A.; Ochieng, M.; Hada, R.; Jain, P.; Axmed, M.; Bali, K.; and Sitaram, S. 2024. [MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks](https://md.research.elamur.ai/free/en/papers/ahuja-2024-megaverse-benchmarking-large-language-models/). arXiv:2311.07463.
3. Bai, J.; Bai, S.; Chu, Y.; Cui, Z.; Dang, K.; Deng, X.; Fan, Y.; Ge, W.; Han, Y.; Huang, F.; et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
4. Blasi, D.; Anastasopoulos, A.; and Neubig, G. 2021. [Systematic Inequalities in Language Technology Performance across the World’s Languages](https://md.research.elamur.ai/free/en/papers/blasi-2022-systematic-inequalities-language-technology-performance/). arXiv preprint arXiv:2110.06733.
5. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.… [find](https://scholar.google.com/scholar?q=Clark%2C+P%2E%3B+Cowhey%2C+I%2E%3B+Etzioni%2C+O%2E%3B+Khot%2C+T%2E%3B+Sabharwal%2C+A%2E%3B+Schoenick%2C+C%2E%3B+and+Tafjord%2C+O%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+arXiv+preprint+arXiv%3A1)
6. Gurnee, W.; and Tegmark, M. 2023. Language models represent space and time. arXiv preprint arXiv:2310.02207. [find](https://scholar.google.com/scholar?q=Gurnee%2C+W%2E%3B+and+Tegmark%2C+M%2E+2023%2E+Language+models+represent+space+and+time%2E+arXiv+preprint+arXiv%3A2310%2E02207%2E)
7. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
8. Huang, H.; Tang, T.; Zhang, D.; Zhao, W. X.; Song, T.; Xia, Y.; and Wei, F. 2023. [Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting](https://md.research.elamur.ai/free/en/papers/huang-2023-all-languages-created-equal-llms/). arXiv preprint arXiv:2305.07004. Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S…
9. Ju, T.; Sun, W.; Du, W.; Yuan, X.; Ren, Z.; and Liu, G. 2024. How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study. arXiv preprint arXiv:2402.16061. [find](https://scholar.google.com/scholar?q=Ju%2C+T%2E%3B+Sun%2C+W%2E%3B+Du%2C+W%2E%3B+Yuan%2C+X%2E%3B+Ren%2C+Z%2E%3B+and+Liu%2C+G%2E+2024%2E+How+Large+Language+Models+Encode+Context+Knowledge%3F+A+Layer%2DWise+Probing+Study%2E+arXiv+preprint+arXiv%3A2402%2E16061%2E)
10. Kalyan, K. S.; Rajasekharan, A.; and Sangeetha, S. 2021. Ammus: A survey of transformer-based pretrained models in natural language processing. arXiv preprint arXiv:2108.05542. — [arXiv:2108.05542](https://arxiv.org/abs/2108.05542) [find](https://scholar.google.com/scholar?q=Kalyan%2C+K%2E+S%2E%3B+Rajasekharan%2C+A%2E%3B+and+Sangeetha%2C+S%2E+2021%2E+Ammus%3A+A+survey+of+transformer%2Dbased+pretrained+models+in+natural+language+processing%2E+arXiv+preprint+arXiv%3A2108%2E05542%2E+%E2%80%94+%5BarXiv%3A2108%2E05542%5D%28ht)
11. Lankford, S.; Afli, H.; and Way, A. 2024. Transformers for Low-Resource Languages: Is F\\’eidir Linn! arXiv preprint arXiv:2403.01985. Li, K.; Patel, O.; Viégas, F.; Pfister, H.; and Wattenberg, M. 2024. [Inference-time intervention: Eliciting truthful answers from a language model](https://md.research.elamur.ai/free/en/papers/li-2023-inference-time-intervention-eliciting-truthful/). Advances in Neura…
12. Liu, C.; Zhang, W.; Zhao, Y.; Luu, A. T.; and Bing, L. 2024. Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models. arXiv preprint arXiv:2403.10258. [find](https://scholar.google.com/scholar?q=Liu%2C+C%2E%3B+Zhang%2C+W%2E%3B+Zhao%2C+Y%2E%3B+Luu%2C+A%2E+T%2E%3B+and+Bing%2C+L%2E+2024%2E+Is+Translation+All+You+Need%3F+A+Study+on+Solving+Multilingual+Tasks+with+Large+Language+Models%2E+arXiv+preprint+arXiv%3A2403%2E10258%2E)
13. Marks, S.; and Tegmark, M. 2023. [The geometry of truth: Emergent linear structure in large language model representations of true/false datasets](https://md.research.elamur.ai/free/en/papers/marks-2023-geometry-truth-emergent-linear-structure/). arXiv preprint arXiv:2310.06824.
14. Meta-AI. 2024. LlaMa3. [https://github.com/meta-llama/](https://github.com/meta-llama/) llama3. Accessed: 2024-06-14. [↗](https://github.com/meta-llama/)
15. OpenAI. 2023. GPT-3 Dataset Statistics. [https://github.com/](https://github.com/) openai/gpt-3/tree/master/dataset statistics. [↗](https://github.com/)
16. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35: 27730–27744.
17. Qin, L.; Chen, Q.; Wei, F.; Huang, S.; and Che, W. 2023. [Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages](https://md.research.elamur.ai/free/en/papers/qin-2023-cross-lingual-prompting-improving-zero/). arXiv preprint arXiv:2310.14799.
18. Qin, L.; Chen, Q.; Zhou, Y.; Chen, Z.; Li, Y.; Liao, L.; Li, M.; Che, W.; and Yu, P. S. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925. Schäfer, A.; Ravfogel, S.; Hofmann, T.; Pimentel, T.; and Schlag, I. 2024. Language Imbala… [find](https://scholar.google.com/scholar?q=Qin%2C+L%2E%3B+Chen%2C+Q%2E%3B+Zhou%2C+Y%2E%3B+Chen%2C+Z%2E%3B+Li%2C+Y%2E%3B+Liao%2C+L%2E%3B+Li%2C+M%2E%3B+Che%2C+W%2E%3B+and+Yu%2C+P%2E+S%2E+2024%2E+Multilingual+large+language+model%3A+A+survey+of+resources%2C+taxonomy+and+frontiers%2E+arXiv+preprint+arXiv%3A240)
19. Shen, L.; Tan, W.; Chen, S.; Chen, Y.; Zhang, J.; Xu, H.; Zheng, B.; Koehn, P.; and Khashabi, D. 2024. [The language barrier: Dissecting safety challenges of llms in multilingual contexts](https://md.research.elamur.ai/free/en/papers/shen-2024-language-barrier-dissecting-safety-challenges/). arXiv preprint arXiv:2401.13136.
20. Steck, H.; Ekanadham, C.; and Kallus, N. 2024. Is Cosine-Similarity of Embeddings Really About Similarity? In Companion Proceedings of the ACM on Web Conference 2024, WWW ’24. ACM. Team, G.; Mesnard, T.; Hardin, C.; Dadashi, R.; Bhupatiraju, S.; Pathak, S.; Sifre, L.; Rivière, M.; Kale, M. S.; Love… [find](https://scholar.google.com/scholar?q=Steck%2C+H%2E%3B+Ekanadham%2C+C%2E%3B+and+Kallus%2C+N%2E+2024%2E+Is+Cosine%2DSimilarity+of+Embeddings+Really+About+Similarity%3F+In+Companion+Proceedings+of+the+ACM+on+Web+Conference+2024%2C+WWW+%E2%80%9924%2E+ACM%2E+Team%2C+G%2E%3B+Mesnard%2C+)
21. Tiedemann, J. 2020. [The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT](https://md.research.elamur.ai/free/en/papers/tiedemann-2020-tatoeba-translation-challenge-realistic-data/). In Proceedings of the Fifth Conference on Machine Translation, 1174–1182. Online: Association for Computational Linguistics.
22. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%3B+Martin%2C+L%2E%3B+Stone%2C+K%2E%3B+Albert%2C+P%2E%3B+Almahairi%2C+A%2E%3B+Babaei%2C+Y%2E%3B+Bashlykov%2C+N%2E%3B+Batra%2C+S%2E%3B+Bhargava%2C+P%2E%3B+Bhosale%2C+S%2E%3B+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+)
23. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
24. Wendler, C.; Veselovsky, V.; Monea, G.; and West, R. 2024. [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://md.research.elamur.ai/free/en/papers/wendler-2024-do-llamas-work-english-latent/). arXiv:2402.10588.
25. Xie, Y.; Chen, X.; Zhan, H.; Shivakumara, P.; Yin, B.; Liu, C.; and Lu, Y. 2024. Weakly supervised scene text generation for low-resource languages. Expert Systems with Applications, 237: 121622. [find](https://scholar.google.com/scholar?q=Xie%2C+Y%2E%3B+Chen%2C+X%2E%3B+Zhan%2C+H%2E%3B+Shivakumara%2C+P%2E%3B+Yin%2C+B%2E%3B+Liu%2C+C%2E%3B+and+Lu%2C+Y%2E+2024%2E+Weakly+supervised+scene+text+generation+for+low%2Dresource+languages%2E+Expert+Systems+with+Applications%2C+237%3A+121622%2E)
26. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%3B+Holtzman%2C+A%2E%3B+Bisk%2C+Y%2E%3B+Farhadi%2C+A%2E%3B+and+Choi%2C+Y%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
27. Zhang, B.; Williams, P.; Titov, I.; and Sennrich, R. 2020. [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://md.research.elamur.ai/free/en/papers/zhang-2020-improving-massively-multilingual-neural-machine/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1628–1639. Online: Association for Computational Linguistics.
28. Zhang, X.; Li, S.; Hauer, B.; Shi, N.; and Kondrak, G. 2023. Don’t trust ChatGPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 7915–7927. [find](https://scholar.google.com/scholar?q=Zhang%2C+X%2E%3B+Li%2C+S%2E%3B+Hauer%2C+B%2E%3B+Shi%2C+N%2E%3B+and+Kondrak%2C+G%2E+2023%2E+Don%E2%80%99t+trust+ChatGPT+when+your+question+is+not+in+English%3A+A+study+of+multilingual+abilities+and+types+of+LLMs%2E+In+Proceedings+of+the+2023+C)
29. Zou, A.; Phan, L.; Chen, S.; Campbell, J.; Guo, P.; Ren, R.; Pan, A.; Yin, X.; Mazeika, M.; Dombrowski, A.-K.; et al. 2023. [Representation engineering: A top-down approach to ai transparency](https://md.research.elamur.ai/free/en/papers/zou-2023-representation-engineering-top-down-approach/). arXiv preprint arXiv:2310.01405.

*Updated 24 October 2026*