---
title: "Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval (2021)"
description: "The authors build Mr. TyDi, an open-retrieval version of TyDi. 24 references, cited by 78 papers here."
authors: ["Xinyu Zhang", "Xueguang Ma", "Peng Shi", "Jimmy Lin"]
year: 2021
venue: "Proceedings of the 1st Workshop on Multilingual Representation Learning"
source: "https://aclanthology.org/2021.mrl-1.12/"
doi: "10.18653/v1/2021.mrl-1.12"
arxiv: "2108.08787"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2021-mr-tydi-multi-lingual-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2021-mr-tydi-multi-lingual-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval

# Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval

**Authors:** [Xinyu Zhang](https://md.research.elamur.ai/free/en/authors/xinyu-zhang/), [Xueguang Ma](https://md.research.elamur.ai/free/en/authors/xueguang-ma/), [Peng Shi](https://md.research.elamur.ai/free/en/authors/peng-shi/) and [Jimmy Lin](https://md.research.elamur.ai/free/en/authors/jimmy-lin/) · **Year:** 2021 · **Venue:** [Proceedings of the 1st Workshop on Multilingual Representation Learning](https://md.research.elamur.ai/free/en/venues/mrl/) ([Workshop on Multilingual Representation Learning 2021](https://md.research.elamur.ai/free/en/venues/mrl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.mrl-1.12/) · [PDF](https://aclanthology.org/2021.mrl-1.12.pdf) · [DOI 10.18653/v1/2021.mrl-1.12](https://doi.org/10.18653/v1/2021.mrl-1.12) · [arXiv 2108.08787](https://arxiv.org/abs/2108.08787) · [OpenAlex](https://openalex.org/W3214151748)

**Citations:** 51 citations (OpenAlex, 2026-10-14) · [78](https://md.research.elamur.ai/free/en/papers/zhang-2021-mr-tydi-multi-lingual-benchmark/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Dense retrieval models are usually trained on English data. They are known to degrade on out-of-distribution input, but how they behave in other languages has been poorly studied. Existing resources do not fill this gap. TyDi is essentially a reading-comprehension task over pre-selected articles, XOR-TyDi targets English Wikipedia (cross-lingual), and MKQA relies on translated, entity-centred questions. The authors build Mr. TyDi, an open-retrieval version of TyDi. Questions in eleven typologically diverse languages must be answered by searching the full Wikipedia of the same language, using the original TyDi dumps. As zero-shot baselines they compare BM25, a multi-lingual DPR variant called mDPR, and a sparse–dense hybrid. On its own, mDPR is far weaker than BM25. Still, combining it with BM25 improves results, which shows that even weak dense representations carry useful relevance signals. The benchmark gives a foundation for studying the robustness of dense retrieval beyond English and for expanding information access in non-English languages.

## Key points
- **Coverage:** Mr. TyDi handles mono-lingual retrieval only, with queries and passages in the same language. The benchmark spans eleven languages: English, Finnish, Russian, Arabic, Swahili, Bengali, Telugu, Thai, Indonesian, Japanese and Korean.
- **Training data size:** The number of training questions varies widely by language, from 12,377 for Arabic down to 1,295 for Korean.
- **Baseline results:** mDPR was evaluated without language-specific fine-tuning and lags BM25. The BM25+mDPR hybrid outperforms BM25 alone.
- **Where hybrid gains come from:**
  - Bengali, Indonesian, Swahili and Telugu gain mostly through better recall.
  - Thai gains mainly through better ranking.
  - The remaining languages improve on both recall and ranking.
- **Availability and future work:** The dataset is published at github.com/castorini/mr.tydi. Planned next steps include multi-step fine-tuning strategies and using the benchmark to examine multi-lingual transformer models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Waterloo](https://md.research.elamur.ai/free/en/universities/university-of-waterloo/) — University, Waterloo, Canada · 80 works · 15,030 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works

## Cite

Cite as: Xinyu Zhang, Xueguang Ma, Peng Shi and Jimmy Lin. 2021. Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval. Proceedings of the 1st Workshop on Multilingual Representation Learning.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2021mr,
  title = {{Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval}},
  author = {Xinyu Zhang, Xueguang Ma, Peng Shi and Jimmy Lin},
  booktitle = {Proceedings of the 1st Workshop on Multilingual Representation Learning},
  year = {2021},
  doi = {10.18653/v1/2021.mrl-1.12},
  eprint = {2108.08787},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.mrl-1.12}
}
```

- [← previous Improving Factual Consistency of Abstractive Summarization via Question Answering](https://md.research.elamur.ai/free/en/papers/nan-2021-improving-factual-consistency-abstractive-summarization/)
- [next → Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://md.research.elamur.ai/free/en/papers/niklaus-2021-swiss-judgment-prediction-multilingual-legal/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

24 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021. [XOR QA: Cross-lingual open-retrieval question answering](https://md.research.elamur.ai/free/en/papers/asai-2021-xor-qa-cross-lingual-open/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). arXiv:1611.09268…
3. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading Wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870– 1879, Vancouver, Canada.
4. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. [TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics, 8:454– 47…
5. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 deep learning track. arXiv:2102.07662. — [arXiv:2102.07662](https://arxiv.org/abs/2102.07662) [find](https://scholar.google.com/scholar?q=Nick+Craswell%2C+Bhaskar+Mitra%2C+Emine+Yilmaz%2C+and+Daniel+Campos%2E+2021%2E+Overview+of+the+TREC+2020+deep+learning+track%2E+arXiv%3A2102%2E07662%2E+%E2%80%94+%5BarXiv%3A2102%2E07662%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2102%2E07662%29)
6. Omar Khattab and Matei Zaharia. 2020. [ColBERT: Efficient and effective passage search via contextualized late interaction over BERT](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/). In Proceedings of the 43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020), pages 39–48.
7. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
8. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021a. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and… [find](https://scholar.google.com/scholar?q=Jimmy+Lin%2C+Xueguang+Ma%2C+Sheng%2DChieh+Lin%2C+Jheng%2DHong+Yang%2C+Ronak+Pradeep%2C+and+Rodrigo+Nogueira%2E+2021a%2E+Pyserini%3A+A+Python+toolkit+for+reproducible+information+retrieval+research+with+sparse+and+dense+r)
9. Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020. Pretrained transformers for text ranking: BERT and beyond. arXiv:2010.06467. — [arXiv:2010.06467](https://arxiv.org/abs/2010.06467) [find](https://scholar.google.com/scholar?q=Jimmy+Lin%2C+Rodrigo+Nogueira%2C+and+Andrew+Yates%2E+2020%2E+Pretrained+transformers+for+text+ranking%3A+BERT+and+beyond%2E+arXiv%3A2010%2E06467%2E+%E2%80%94+%5BarXiv%3A2010%2E06467%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2010%2E06467%29)
10. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. [Overview of the TREC 2019 deep learning track](https://md.research.elamur.ai/free/en/papers/craswell-2020-overview-trec-2019-deep-learning/). arXiv:2003.07820.
11. Luyu Gao, Zhuyun Dai, Zhen Fan, and Jamie Callan. 2020. Complementing lexical retrieval with semantic residual embedding. arXiv:2004.13969. — [arXiv:2004.13969](https://arxiv.org/abs/2004.13969) [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Zhuyun+Dai%2C+Zhen+Fan%2C+and+Jamie+Callan%2E+2020%2E+Complementing+lexical+retrieval+with+semantic+residual+embedding%2E+arXiv%3A2004%2E13969%2E+%E2%80%94+%5BarXiv%3A2004%2E13969%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2004%2E13969%29)
12. Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving efficient neural ranking models with cross-architecture knowledge distillation. arXiv:2010.02666. — [arXiv:2010.02666](https://arxiv.org/abs/2010.02666) [find](https://scholar.google.com/scholar?q=Sebastian+Hofst%C3%A4tter%2C+Sophia+Althammer%2C+Michael+Schr%C3%B6der%2C+Mete+Sertkan%2C+and+Allan+Hanbury%2E+2020%2E+Improving+efficient+neural+ranking+models+with+cross%2Darchitecture+knowledge+distillation%2E+arXiv%3A2010%2E02)
13. Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. [Efficiently teaching an effective dense retriever with balanced topic aware sampling](https://md.research.elamur.ai/free/en/papers/hofstatter-2021-efficiently-teaching-effective-dense-retriever/). In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval…
14. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. [Billion-scale similarity search with GPUs](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/). arXiv:1702.08734.
15. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769– 6781.
16. Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021b. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 163–173. [find](https://scholar.google.com/scholar?q=Sheng%2DChieh+Lin%2C+Jheng%2DHong+Yang%2C+and+Jimmy+Lin%2E+2021b%2E+In%2Dbatch+negatives+for+knowledge+distillation+with+tightly%2Dcoupled+teachers+for+dense+retrieval%2E+In+Proceedings+of+the+6th+Workshop+on+Represent)
17. Shayne Longpre, Yi Lu, and Joachim Daiber. 2020. [MKQA: A linguistically diverse benchmark for multilingual open domain question answering](https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/). arXiv:2007.15207.
18. Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. 2021. A replication study of dense passage retriever. arXiv:2104.05740. — [arXiv:2104.05740](https://arxiv.org/abs/2104.05740) [find](https://scholar.google.com/scholar?q=Xueguang+Ma%2C+Kai+Sun%2C+Ronak+Pradeep%2C+and+Jimmy+Lin%2E+2021%2E+A+replication+study+of+dense+passage+retriever%2E+arXiv%3A2104%2E05740%2E+%E2%80%94+%5BarXiv%3A2104%2E05740%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2104%2E05740%29)
19. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas.
20. Peng Shi, He Bai, and Jimmy Lin. 2020. Cross-lingual training of neural models for document ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2768–2773. [find](https://scholar.google.com/scholar?q=Peng+Shi%2C+He+Bai%2C+and+Jimmy+Lin%2E+2020%2E+Cross%2Dlingual+training+of+neural+models+for+document+ranking%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+2768%E2%80%932773%2E)
21. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. [BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models](https://md.research.elamur.ai/free/en/papers/thakur-2021-beir-heterogenous-benchmark-zero/). arXiv:2104.08663.
22. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
23. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. [Approximate nearest neighbor negative contrastive learning for dense text retrieval](https://md.research.elamur.ai/free/en/papers/xiong-2020-approximate-nearest-neighbor-negative-contrastive/). In Proceedings of the 9th International Conference on Learning Representations (ICLR 2021).
24. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of Lucene for information retrieval research. In Proceedings of the 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017), pages 1253–1256, Tokyo, Japan. [find](https://scholar.google.com/scholar?q=Peilin+Yang%2C+Hui+Fang%2C+and+Jimmy+Lin%2E+2017%2E+Anserini%3A+Enabling+the+use+of+Lucene+for+information+retrieval+research%2E+In+Proceedings+of+the+40th+Annual+International+ACM+SIGIR+Conference+on+Research+an)

*Updated 2 November 2026*