---
title: "The Overlooked Role of Graded Relevance Thresholds in"
description: "The authors build a synthetic training set from the MIRACL corpus. 36 references. Among them: 0."
authors: ["Tomer Wullach", "Ori Shapira", "Amir David Nissan Cohen"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.382/"
doi: "10.18653/v1/2026.findings-acl.382"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wullach-2026-overlooked-role-graded-relevance-thresholds/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wullach-2026-overlooked-role-graded-relevance-thresholds/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/52/) / The Overlooked Role of Graded Relevance Thresholds in Multilingual Dense Retrieval

# The Overlooked Role of Graded Relevance Thresholds in Multilingual Dense Retrieval

**Authors:** Tomer Wullach, [Ori Shapira](https://md.research.elamur.ai/free/en/authors/ori-shapira/) and [Amir David Nissan Cohen](https://md.research.elamur.ai/free/en/authors/amir-david-nissan-cohen/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.382/) · [PDF](https://aclanthology.org/2026.findings-acl.382.pdf) · [DOI 10.18653/v1/2026.findings-acl.382](https://doi.org/10.18653/v1/2026.findings-acl.382) · [OpenAlex](https://openalex.org/W7166889103)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Contrastive fine-tuning of dense retrievers needs binary positive/negative pairs, but relevance is a matter of degree. The paper asks how the cutoff used to turn graded relevance scores into binary labels affects multilingual retrieval quality. The authors build a synthetic training set from the MIRACL corpus. An LLM writes queries for sampled passages, a baseline dense retriever fetches top-k candidates, and an LLM scores each query–passage pair on a multi-level scale. They then fine-tune with different thresholds in three settings: monolingual, mixed-language, and cross-lingual retrieval.

The best threshold is not universal. It shifts with language and task, roughly tracking how well-resourced a language is. Lower-resource languages gain from permissive cutoffs that keep loosely related positives, while high-resource languages tend to prefer strict ones. A good cutoff can raise effectiveness, reduce the amount of fine-tuning data needed, and dampen LLM annotation noise. A bad one can leave the model worse than no fine-tuning at all. The authors argue the threshold should be tuned on validation data, like any hyperparameter.

## Key points
- **Data:** The training data is drawn from MIRACL (18 languages). The analyses use Finnish, Arabic, Japanese, Russian, Spanish, and English, which span different resource levels.
- **Cross-lingual evaluation:** This uses CLIRMatrix with nDCG@10. Finnish is excluded because it is missing from that test set.
- **Threshold comparison:** The paper reports how much performance rises or falls when the cutoff is raised from 1 to 3.
  - If either the query or the corpus language is lower-resource, the lower cutoff usually wins.
  - If both are high-resource, the higher cutoff wins.
  - English as a corpus language is the main exception, favoring stricter thresholds.
- **Annotator disagreement:** LLM annotators disagree noticeably, especially for lower-resource languages. Choosing the threshold is presented as a simple way to buffer this noise.
- **Resources and limitations:** Code and data are at github.com/tomerwull/graded-relevance-thresholds. Stated limitations are that changing the threshold shifts the positive/negative balance, and that retrieving candidates with a baseline model may bias the data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (2): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 882 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works

## Cite

Cite as: Tomer Wullach, Ori Shapira and Amir David Nissan Cohen. 2026. The Overlooked Role of Graded Relevance Thresholds in Multilingual Dense Retrieval. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wullach2026overlooked,
  title = {{The Overlooked Role of Graded Relevance Thresholds in Multilingual Dense Retrieval}},
  author = {Tomer Wullach, Ori Shapira and Amir David Nissan Cohen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.382},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.382}
}
```

- [← previous The No-Meaning Falsity: The Structural Impossibility of the Arbitrary Sign in Classical Arabic](https://md.research.elamur.ai/free/en/papers/abdelwahab-2026-no-meaning-falsity-structural-impossibility/)
- [next → The past and future of Fairslator](https://md.research.elamur.ai/free/en/papers/mechura-2026-past-future-fairslator/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/52/)

## References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Negar Arabzadeh and Charles L. A. Clarke. 2025. Benchmarking llm-based relevance judgment methods. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’25, page 3194–3204, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Negar+Arabzadeh+and+Charles+L%2E+A%2E+Clarke%2E+2025%2E+Benchmarking+llm%2Dbased+relevance+judgment+methods%2E+In+Proceedings+of+the+48th+International+ACM+SIGIR+Conference+on+Research+and+Development+in+Informat)
2. Akari Asai, Xinyan Yu, Jungo Kasai, and Hanna Hajishirzi. 2021. One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval. In Advances in Neural Information Processing Systems, volume 34, pages 7547–7560. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Akari+Asai%2C+Xinyan+Yu%2C+Jungo+Kasai%2C+and+Hanna+Hajishirzi%2E+2021%2E+One+Question+Answering+Model+for+Many+Languages+with+Cross%2Dlingual+Dense+Passage+Retrieval%2E+In+Advances+in+Neural+Information+Processing)
3. Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pre-training Tasks for Embedding-based Large-scale Retrieval. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Wei%2DCheng+Chang%2C+Felix+X%2E+Yu%2C+Yin%2DWen+Chang%2C+Yiming+Yang%2C+and+Sanjiv+Kumar%2E+2020%2E+Pre%2Dtraining+Tasks+for+Embedding%2Dbased+Large%2Dscale+Retrieval%2E+In+International+Conference+on+Learning+Representations%2E)
4. Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. 2009. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM ’09, page 621–630, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Olivier+Chapelle%2C+Donald+Metlzer%2C+Ya+Zhang%2C+and+Pierre+Grinspan%2E+2009%2E+Expected+reciprocal+rank+for+graded+relevance%2E+In+Proceedings+of+the+18th+ACM+Conference+on+Information+and+Knowledge+Management%2C)
5. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. [BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). CoRR, abs/2402.03216.
6. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Marton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzeminski, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Diganta Misra, Shreeya Dhakal, Jonathan Rystrom, Roman Solomatin, Omer Veysel C… — [MMTEB: Massive Multilingual Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/enevoldsen-2025-mmteb-massive-multilingual-text-embedding/)
7. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistic…
8. Robert Litschko, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2018. Unsupervised Cross-Lingual Information Retrieval Using Monolingual Data Only. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 1253–1256, New York, NY, USA.… [find](https://scholar.google.com/scholar?q=Robert+Litschko%2C+Goran+Glava%C5%A1%2C+Simone+Paolo+Ponzetto%2C+and+Ivan+Vulic%CC%81%2E+2018%2E+Unsupervised+Cross%2DLingual+Information+Retrieval+Using+Monolingual+Data+Only%2E+In+The+41st+International+ACM+SIGIR+Conferenc)
9. Team Gemma. 2025. Gemma 3 Technical Report. Preprint, arXiv:2503.19786. [find](https://scholar.google.com/scholar?q=Team+Gemma%2E+2025%2E+Gemma+3+Technical+Report%2E+Preprint%2C+arXiv%3A2503%2E19786%2E)
10. Peiyang Liu, Sen Wang, Xi Wang, Wei Ye, and Shikun Zhang. 2021a. QuadrupletBERT: An Efficient Model For Embedding-Based Large-Scale Retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3734… [find](https://scholar.google.com/scholar?q=Peiyang+Liu%2C+Sen+Wang%2C+Xi+Wang%2C+Wei+Ye%2C+and+Shikun+Zhang%2E+2021a%2E+QuadrupletBERT%3A+An+Efficient+Model+For+Embedding%2DBased+Large%2DScale+Retrieval%2E+In+Proceedings+of+the+2021+Conference+of+the+North+Americ)
11. Kailash A. Hambarde and Hugo Proença. 2023. Information Retrieval: Recent Advances and Beyond. IEEE Access, 11:76581–76604. [find](https://scholar.google.com/scholar?q=Kailash+A%2E+Hambarde+and+Hugo+Proen%C3%A7a%2E+2023%2E+Information+Retrieval%3A+Recent+Advances+and+Beyond%2E+IEEE+Access%2C+11%3A76581%E2%80%9376604%2E)
12. Chao Huang, Fengran Mo, Yufeng Chen, Changhao Guan, Zhenrui Yue, Xinyu Wang, Jinan Xu, and Kaiyu Huang. 2025. Boosting Data Utilization for Multilingual Dense Retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12362–12378, Suzhou, China. Assoc… [find](https://scholar.google.com/scholar?q=Chao+Huang%2C+Fengran+Mo%2C+Yufeng+Chen%2C+Changhao+Guan%2C+Zhenrui+Yue%2C+Xinyu+Wang%2C+Jinan+Xu%2C+and+Kaiyu+Huang%2E+2025%2E+Boosting+Data+Utilization+for+Multilingual+Dense+Retrieval%2E+In+Proceedings+of+the+2025+Con)
13. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. [Unsupervised Dense Information Retrieval with Contrastive Learning](https://md.research.elamur.ai/free/en/papers/izacard-2021-unsupervised-dense-information-retrieval-contrastive/). Transactions on Machine Learning Research.
14. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422–446. [find](https://scholar.google.com/scholar?q=Kalervo+J%C3%A4rvelin+and+Jaana+Kek%C3%A4l%C3%A4inen%2E+2002%2E+Cumulated+gain%2Dbased+evaluation+of+IR+techniques%2E+ACM+Trans%2E+Inf%2E+Syst%2E%2C+20%284%29%3A422%E2%80%93446%2E)
15. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769– 6781, Onl…
16. Omar Khattab and Matei Zaharia. 2020. [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/). In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 39–48, New York, NY, USA. Associati…
17. Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, and Ift… — [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://md.research.elamur.ai/free/en/papers/lee-2024-gecko-versatile-text-embeddings-distilled/)
18. Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou. 2025. From Matching to Generation: A Survey on Generative Information Retrieval. ACM Trans. Inf. Syst., 43(3). [find](https://scholar.google.com/scholar?q=Xiaoxi+Li%2C+Jiajie+Jin%2C+Yujia+Zhou%2C+Yuyao+Zhang%2C+Peitian+Zhang%2C+Yutao+Zhu%2C+and+Zhicheng+Dou%2E+2025%2E+From+Matching+to+Generation%3A+A+Survey+on+Generative+Information+Retrieval%2E+ACM+Trans%2E+Inf%2E+Syst%2E%2C+43%283)
19. Peiyang Liu, Xi Wang, Sen Wang, Wei Ye, Xiangyu Xi, and Shikun Zhang. 2021b. Improving Embedding-based Large-scale Retrieval via Label Enhancement. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 133–142, Punta Cana, Dominican Republic. Association for Computational L… [find](https://scholar.google.com/scholar?q=Peiyang+Liu%2C+Xi+Wang%2C+Sen+Wang%2C+Wei+Ye%2C+Xiangyu+Xi%2C+and+Shikun+Zhang%2E+2021b%2E+Improving+Embedding%2Dbased+Large%2Dscale+Retrieval+via+Label+Enhancement%2E+In+Findings+of+the+Association+for+Computational+Lin)
20. Peiyang Liu, Xiangyu Xi, Wei Ye, and Shikun Zhang. 2022. Label Smoothing for Text Mining. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2210–2219, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peiyang+Liu%2C+Xiangyu+Xi%2C+Wei+Ye%2C+and+Shikun+Zhang%2E+2022%2E+Label+Smoothing+for+Text+Mining%2E+In+Proceedings+of+the+29th+International+Conference+on+Computational+Linguistics%2C+pages+2210%E2%80%932219%2C+Gyeongju%2C+R)
21. Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 4(1):139– 148. [find](https://scholar.google.com/scholar?q=Tyler+McDonnell%2C+Matthew+Lease%2C+Mucahid+Kutlu%2C+and+Tamer+Elsayed%2E+2016%2E+Why+Is+That+Relevant%3F+Collecting+Annotator+Rationales+for+Relevance+Judgments%2E+Proceedings+of+the+AAAI+Conference+on+Human+Compu)
22. Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, and Maarten de Rijke. 2025. Query Performance Prediction Using Relevance Judgments Generated by Large Language Models. ACM Trans. Inf. Syst., 43(4). [find](https://scholar.google.com/scholar?q=Chuan+Meng%2C+Negar+Arabzadeh%2C+Arian+Askari%2C+Mohammad+Aliannejadi%2C+and+Maarten+de+Rijke%2E+2025%2E+Query+Performance+Prediction+Using+Relevance+Judgments+Generated+by+Large+Language+Models%2E+ACM+Trans%2E+Inf%2E+)
23. Suraj Nair, Eugene Yang, Dawn Lawrie, Kevin Duh, Paul McNamee, Kenton Murray, James Mayfield, and Douglas W. Oard. 2022. Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models. In Advances in Information Retrieval, pages 382–396, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=Suraj+Nair%2C+Eugene+Yang%2C+Dawn+Lawrie%2C+Kevin+Duh%2C+Paul+McNamee%2C+Kenton+Murray%2C+James+Mayfield%2C+and+Douglas+W%2E+Oard%2E+2022%2E+Transfer+Learning+Approaches+for+Building+Cross%2DLanguage+Dense+Retrieval+Models)
24. Team OpenAI. 2024. [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). Preprint, arXiv:2410.21276.
25. Filip Radlinski and Nick Craswell. 2010. Comparing the sensitivity of information retrieval metrics. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’10, page 667–674, New York, NY, USA. Association for Computing Machinery. Bi… [find](https://scholar.google.com/scholar?q=Filip+Radlinski+and+Nick+Craswell%2E+2010%2E+Comparing+the+sensitivity+of+information+retrieval+metrics%2E+In+Proceedings+of+the+33rd+International+ACM+SIGIR+Conference+on+Research+and+Development+in+Inform)
26. Kevin Roitero, Eddy Maddalena, Gianluca Demartini, and Stefano Mizzaro. 2018. On Fine-Grained Relevance Scales. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 675–684, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Kevin+Roitero%2C+Eddy+Maddalena%2C+Gianluca+Demartini%2C+and+Stefano+Mizzaro%2E+2018%2E+On+Fine%2DGrained+Relevance+Scales%2E+In+The+41st+International+ACM+SIGIR+Conference+on+Research+%26+Development+in+Information+)
27. Tetsuya Sakai. 2021. Graded Relevance, pages 1–20. Springer Singapore, Singapore. [find](https://scholar.google.com/scholar?q=Tetsuya+Sakai%2E+2021%2E+Graded+Relevance%2C+pages+1%E2%80%9320%2E+Springer+Singapore%2C+Singapore%2E)
28. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024b. [Improving Text Embeddings with Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-improving-text-embeddings-large-language/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, Bangkok, Thailand. Ass…
29. Tefko Saracevic. 2007. Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance. Journal of the American Society for Information Science and Technology, 58(13):1915– 1933. [find](https://scholar.google.com/scholar?q=Tefko+Saracevic%2E+2007%2E+Relevance%3A+A+review+of+the+literature+and+a+framework+for+thinking+on+the+notion+in+information+science%2E+Part+II%3A+nature+and+manifestations+of+relevance%2E+Journal+of+the+American)
30. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024c. [Multilingual E5 Text Embeddings: A Technical Report](https://md.research.elamur.ai/free/en/papers/wang-2024-multilingual-e5-text-embeddings-technical/). Preprint, arXiv:2402.05672.
31. Falk Scholer and Andrew Turpin. 2009. Metric and Relevance Mismatch in Retrieval Evaluation. In Information Retrieval Technology, pages 50–62, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Falk+Scholer+and+Andrew+Turpin%2E+2009%2E+Metric+and+Relevance+Mismatch+in+Retrieval+Evaluation%2E+In+Information+Retrieval+Technology%2C+pages+50%E2%80%9362%2C+Berlin%2C+Heidelberg%2E+Springer+Berlin+Heidelberg%2E)
32. Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. [MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages](https://md.research.elamur.ai/free/en/papers/zhang-2023-miracl-multilingual-retrieval-dataset-covering/). Transactions of the Association for Computational Linguistics, 11:1114–1131.
33. Tianhao Shen, Mingtong Liu, Ming Zhou, and Deyi Xiong. 2022. Recovering Gold from Black Sand: Multilingual Dense Passage Retrieval with Hard and False Negative Samples. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10659–10670, Abu Dhabi, United Ara… [find](https://scholar.google.com/scholar?q=Tianhao+Shen%2C+Mingtong+Liu%2C+Ming+Zhou%2C+and+Deyi+Xiong%2E+2022%2E+Recovering+Gold+from+Black+Sand%3A+Multilingual+Dense+Passage+Retrieval+with+Hard+and+False+Negative+Samples%2E+In+Proceedings+of+the+2022+Conf)
34. Shuo Sun and Kevin Duh. 2020. CLIRMatrix: A massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4160–4170, Online. Association for Computati… [find](https://scholar.google.com/scholar?q=Shuo+Sun+and+Kevin+Duh%2E+2020%2E+CLIRMatrix%3A+A+massively+large+collection+of+bilingual+and+multilingual+datasets+for+Cross%2DLingual+Information+Retrieval%2E+In+Proceedings+of+the+2020+Conference+on+Empirica)
35. Nandan Thakur, Jianmo Ni, Gustavo Hernandez Abrego, John Wieting, Jimmy Lin, and Daniel Cer. 2024. Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computat… [find](https://scholar.google.com/scholar?q=Nandan+Thakur%2C+Jianmo+Ni%2C+Gustavo+Hernandez+Abrego%2C+John+Wieting%2C+Jimmy+Lin%2C+and+Daniel+Cer%2E+2024%2E+Leveraging+LLMs+for+Synthesizing+Training+Data+Across+Many+Languages+in+Multilingual+Dense+Retrieval%2E)
36. Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024. Dense Text Retrieval Based on Pretrained Language Models: A Survey. ACM Trans. Inf. Syst., 42(4). [find](https://scholar.google.com/scholar?q=Wayne+Xin+Zhao%2C+Jing+Liu%2C+Ruiyang+Ren%2C+and+Ji%2DRong+Wen%2E+2024%2E+Dense+Text+Retrieval+Based+on+Pretrained+Language+Models%3A+A+Survey%2E+ACM+Trans%2E+Inf%2E+Syst%2E%2C+42%284%29%2E)

*Updated 2 November 2026*