---
title: "Evaluating Multilingual Long-Context Models for Retrieval and"
description: "Long-context LLMs score almost perfectly on synthetic \"needle in a haystack\" tests, but these tests… 35 references, cited by 8 papers here."
authors: ["Ameeta Agrawal", "Andy Dang", "Sina Bagheri Nezhad", "Rhitabrat Pokharel", "Russell Scheinberg"]
year: 2024
venue: "arXiv:2409.18006"
source: "https://arxiv.org/abs/2409.18006"
doi: "10.48550/arxiv.2409.18006"
arxiv: "2409.18006"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/agrawal-2024-evaluating-multilingual-long-context-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agrawal-2024-evaluating-multilingual-long-context-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / Evaluating Multilingual Long-Context Models for Retrieval and Reasoning

# Evaluating Multilingual Long-Context Models for Retrieval and Reasoning

**Authors:** [Ameeta Agrawal](https://md.research.elamur.ai/free/en/authors/ameeta-agrawal/), Andy Dang, [Sina Bagheri Nezhad](https://md.research.elamur.ai/free/en/authors/sina-bagheri-nezhad/), Rhitabrat Pokharel and Russell Scheinberg · **Year:** 2024 · **Venue:** [arXiv:2409.18006](https://md.research.elamur.ai/free/en/venues/mrl/) ([Workshop on Multilingual Representation Learning 2024](https://md.research.elamur.ai/free/en/venues/mrl/2024/)) · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2409.18006) · [PDF](https://arxiv.org/pdf/2409.18006) · [DOI 10.48550/arxiv.2409.18006](https://doi.org/10.48550/arxiv.2409.18006) · [arXiv 2409.18006](https://arxiv.org/abs/2409.18006) · [OpenAlex](https://openalex.org/W4404781182)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/agrawal-2024-evaluating-multilingual-long-context-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Long-context LLMs score almost perfectly on synthetic "needle in a haystack" tests, but these tests mostly use English and hide only one target fact. The authors ask whether such results hold in other languages and when several facts must be found and compared. They built mLongRR, a benchmark that inserts one to three "magic number" sentences (a city paired with a 7-digit number) into naturally occurring text. The data covers English, Vietnamese, Indonesian, Swahili and Somali. These five languages all use Latin script but come from different families and range from high to very low resource. The authors tested six models at context lengths from 2k to 64k tokens and at several insertion depths. Tasks included plain retrieval and comparison-based reasoning, such as finding the largest number. Accuracy dropped sharply for lower-resource languages, longer inputs and more needles. Reasoning was consistently harder than retrieval. Models whose tokenizers split text into fewer pieces did better. The study shows that even simple synthetic probes reveal large multilingual gaps, which matters for building LLMs for under-served languages.

## Key points
- **mLongRR dataset and code** (github.com/PortNLP/mLongRR):
  - 69 city names were machine-translated and then post-edited into all five languages.
  - Needles were placed at 0/25/50/75/100% depth.
  - Human translators handled the non-English languages.
- **Models tested:** GPT-4 (gpt-4-0125-preview), GPT-4o, Gemini-1.5 Pro, Claude-3 Sonnet, YaRN-Llama-2-7b (128K) and Llama-3-8B-Instruct (8k).
- **Best-model accuracy** (Gemini-1.5, GPT-4o):
  - With one needle: roughly 96% in English versus about 36% in Somali.
  - With three needles: about 40% in English and 0% in Somali.
- **Tokenization (fertility):**
  - English needs about 1.1–1.3 tokens per word.
  - Swahili and Somali need up to about 2.5–2.7 tokens per word.
  - GPT-4o and Gemini-1.5 have the lowest average (about 1.5) and also the best results.
- **Number versus city questions:** Models generally answered "largest number" questions better than "which city" questions, but Swahili and Somali showed the opposite pattern.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Portland State University](https://md.research.elamur.ai/free/en/universities/portland-state-university/) — University, Portland, United States · 6 works · 53 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (4): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

## Cite

Cite as: Ameeta Agrawal, Andy Dang, Sina Bagheri Nezhad, Rhitabrat Pokharel and Russell Scheinberg. 2024. Evaluating Multilingual Long-Context Models for Retrieval and Reasoning. arXiv:2409.18006.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{agrawal2024evaluating,
  title = {{Evaluating Multilingual Long-Context Models for Retrieval and Reasoning}},
  author = {Ameeta Agrawal, Andy Dang, Sina Bagheri Nezhad, Rhitabrat Pokharel and Russell Scheinberg},
  year = {2024},
  doi = {10.48550/arxiv.2409.18006},
  eprint = {2409.18006},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.18006}
}
```

- [← previous Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context](https://md.research.elamur.ai/free/en/papers/ruan-2024-evaluating-llms-divergent-thinking-capabilities/)
- [next → From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks](https://md.research.elamur.ai/free/en/papers/stephan-2024-calculation-adjudication-examining-llm-judges/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

35 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. [Do all languages cost the same? tokenization in the era of commercial language models](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904…
3. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. [MEGA: Multilingual evaluation of generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/). In Proceedings of the 2023 Conference on Empirical Methods in N…
4. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
5. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. [Online; accessed 20-March-2024]. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+%5BOnline%3B+accessed+20%2DMarch%2D2024%5D%2E)
6. Sina Bagheri Nezhad and Ameeta Agrawal. 2024a. Exploring the maze of multilingual modeling. Preprint, arXiv:2310.05404. — [arXiv:2310.05404](https://arxiv.org/abs/2310.05404) [find](https://scholar.google.com/scholar?q=Sina+Bagheri+Nezhad+and+Ameeta+Agrawal%2E+2024a%2E+Exploring+the+maze+of+multilingual+modeling%2E+Preprint%2C+arXiv%3A2310%2E05404%2E+%E2%80%94+%5BarXiv%3A2310%2E05404%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E05404%29)
7. Sina Bagheri Nezhad and Ameeta Agrawal. 2024b. [What drives performance in multilingual language models?](https://md.research.elamur.ai/free/en/papers/nezhad-2024-drives-performance-multilingual-language-models/) In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 16–27, Mexico City, Mexico. Association for Computational Linguistics.
8. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508.
9. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. [A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). In Proceedings of…
10. Patrick Bareiß, Roman Klinger, and Jeremy Barnes. 2024. [English prompts are better for nli-based zero-shot emotion classification than target-language prompts](https://md.research.elamur.ai/free/en/papers/barei-ss-2024-english-prompts-better-nli-based/). In Companion Proceedings of the ACM Web Conference 2024, WWW ’24, page 1318–1326, New York, NY, USA. Association for Computing Machinery.
11. Ethan C. Chau and Noah A. Smith. 2021. Specializing multilingual language models: An empirical study. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 51–61, Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ethan+C%2E+Chau+and+Noah+A%2E+Smith%2E+2021%2E+Specializing+multilingual+language+models%3A+An+empirical+study%2E+In+Proceedings+of+the+1st+Workshop+on+Multilingual+Representation+Learning%2C+pages+51%E2%80%9361%2C+Punta+Can)
12. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+2023%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2E)
13. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Assoc… [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annu)
14. Aparna Dhinakaran. 2024. Llmtest_needleinahaystack. [Online; accessed 20-March-2024]. [find](https://scholar.google.com/scholar?q=Aparna+Dhinakaran%2E+2024%2E+Llmtest%5Fneedleinahaystack%2E+%5BOnline%3B+accessed+20%2DMarch%2D2024%5D%2E)
15. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. [Longrope: Extending llm context window beyond 2 million tokens](https://md.research.elamur.ai/free/en/papers/ding-2024-longrope-extending-llm-context/). arXiv preprint arXiv:2402.13753.
16. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
17. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. [find](https://scholar.google.com/scholar?q=Quinn+McNemar%2E+1947%2E+Note+on+the+sampling+error+of+the+difference+between+correlated+proportions+or+percentages%2E+Psychometrika%2C+12%282%29%3A153%E2%80%93157%2E)
18. Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lacalle, and Mikel Artetxe. 2024. [Do multilingual language models think better in English?](https://md.research.elamur.ai/free/en/papers/etxaniz-2024-do-multilingual-language-models-think/) In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short…
19. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+2024%2E+YaRN%3A+Efficient+context+window+extension+of+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representat)
20. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. [Llm maybe longlm: Self-extend llm context window without tuning](https://md.research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/). arXiv preprint arXiv:2401.01325.
21. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. [The state and fate of linguistic diversity and inclusion in the NLP world](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
22. Greg Kamradt. 2023. Llmtest needleinahaystack: Doing simple retrieval from llm models. [https://github.com/gkamradt/](https://github.com/gkamradt/) LLMTest-NeedleInAHaystack/tree/main. [Online; accessed 20-March-2024]. [↗](https://github.com/gkamradt/)
23. Viet Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen. 2023a. [ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages…
24. Viet Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen. 2023b. [ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages…
25. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
26. Amirkeivan Mohtashami and Martin Jaggi. 2023. Random-access infinite context length for transformers. In Advances in Neural Information Processing Systems, volume 36, pages 54567–54585. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Amirkeivan+Mohtashami+and+Martin+Jaggi%2E+2023%2E+Random%2Daccess+infinite+context+length+for+transformers%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+36%2C+pages+54567%E2%80%9354585%2E+Curran+Associa)
27. Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. 2023. Giraffe: Adventures in expanding context lengths in llms. arXiv preprint arXiv:2308.10882. — [arXiv:2308.10882](https://arxiv.org/abs/2308.10882) [find](https://scholar.google.com/scholar?q=Arka+Pal%2C+Deep+Karkhanis%2C+Manley+Roberts%2C+Samuel+Dooley%2C+Arvind+Sundararajan%2C+and+Siddartha+Naidu%2E+2023%2E+Giraffe%3A+Adventures+in+expanding+context+lengths+in+llms%2E+arXiv+preprint+arXiv%3A2308%2E10882%2E+%E2%80%94+%5Ba)
28. Zexuan Qiu, Jingjing Li, Shijue Huang, Wanjun Zhong, and Irwin King. 2024. Clongeval: A chinese benchmark for evaluating long-context large language models. arXiv preprint arXiv:2403.03514. [find](https://scholar.google.com/scholar?q=Zexuan+Qiu%2C+Jingjing+Li%2C+Shijue+Huang%2C+Wanjun+Zhong%2C+and+Irwin+King%2E+2024%2E+Clongeval%3A+A+chinese+benchmark+for+evaluating+long%2Dcontext+large+language+models%2E+arXiv+preprint+arXiv%3A2403%2E03514%2E)
29. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.…
30. Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. 2024. [The language barrier: Dissecting safety challenges of LLMs in multilingual contexts](https://md.research.elamur.ai/free/en/papers/shen-2024-language-barrier-dissecting-safety-challenges/). In Findings of the Association for Computational Linguistics ACL 2024, pages 2668–…
31. Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. 2021. [Do long-range language models actually use long-range context?](https://md.research.elamur.ai/free/en/papers/sun-2021-do-long-range-language-models/) In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 807– 822, Online and Punta Cana, Dominican Republic. Associatio…
32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
33. Eline Visser. 2020. Kalamang dictionary. [find](https://scholar.google.com/scholar?q=Eline+Visser%2E+2020%2E+Kalamang+dictionary%2E)
34. Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. 2024. [Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k](https://md.research.elamur.ai/free/en/papers/yuan-2024-lv-eval-balanced-long-context/). Preprint, arXiv:2402.05136.
35. Judit Ács. 2019. Exploring bert’s vocabulary. [find](https://scholar.google.com/scholar?q=Judit+%C3%81cs%2E+2019%2E+Exploring+bert%E2%80%99s+vocabulary%2E)

*Updated 31 October 2026*