---
title: "DRAMA — paper by Ma et al. (2025) — summary and ideas"
description: "Billion-parameter LLMs fine-tuned as dense retrievers generalize well but are expensive at… 1.5 points. 48 references, cited by 8 papers here."
authors: ["Xueguang Ma", "Xi Victoria Lin", "Barlas Oguz", "Jimmy Lin", "Wen-tau Yih", "Xilun Chen"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.1457/"
doi: "10.18653/v1/2025.acl-long.1457"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ma-2025-drama-diverse-augmentation-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ma-2025-drama-diverse-augmentation-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) / DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers

# DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers

**Authors:** [Xueguang Ma](https://md.research.elamur.ai/free/en/authors/xueguang-ma/), [Xi Victoria Lin](https://md.research.elamur.ai/free/en/authors/xi-victoria-lin/), [Barlas Oguz](https://md.research.elamur.ai/free/en/authors/barlas-oguz/), [Jimmy Lin](https://md.research.elamur.ai/free/en/authors/jimmy-lin/), [Wen-tau Yih](https://md.research.elamur.ai/free/en/authors/wen-tau-yih/) and [Xilun Chen](https://md.research.elamur.ai/free/en/authors/xilun-chen/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.1457/) · [PDF](https://aclanthology.org/2025.acl-long.1457.pdf) · [DOI 10.18653/v1/2025.acl-long.1457](https://doi.org/10.18653/v1/2025.acl-long.1457) · [OpenAlex](https://openalex.org/W4412889736)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/ma-2025-drama-diverse-augmentation-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Billion-parameter LLMs fine-tuned as dense retrievers generalize well but are expensive at inference. For example, an 8B Llama backbone costs roughly 40× more than a BERT-based retriever. Small encoder models are cheap but transfer poorly when supervised data is scarce. DRAMA is a framework that uses LLMs to build compact retrievers in two ways. First, LLMs act as data generators. A Llama3.1 8B retriever and Llama3.3 70B-Instruct produce training triplets from a large multilingual corpus, and the paper compares these augmentation strategies under controlled conditions using only open models and open data. Second, LLMs supply the backbone. Llama3.2 1B is pruned further into 0.1B and 0.3B decoder models, and bi-directional attention is enabled during training. Everything is trained in a single contrastive (InfoNCE) stage, with no multi-stage pre-training. The resulting models perform strongly on BEIR, MIRACL and multilingual MTEB retrieval tasks, beating encoder-based retrievers that rely on multi-stage training. They also handle long contexts and multiple languages well. Because the backbone comes from LLMs, small retrievers can keep improving as LLMs advance.

## Key points
- The paper tests three augmentation routes, ordered from cheapest to most expensive:
  - **Cropped sentences:** these serve as pseudo-queries, with positives and hard negatives mined by the 8B retriever.
  - **Synthetic queries:** these come from the 70B Instruct model.
  - **Listwise reranking:** the 70B model reranks the 8B retriever's candidates.
- **Pruned backbones vs. encoders:** pruned decoder backbones beat their encoder counterparts.
  - The 0.1B model beats BERT by about 1 point and slightly exceeds ModernBERT, while staying multilingual.
  - The 0.3B model beats XLM-RoBERTa-Large by about 1.5 points.
- **Attention and pooling:** bi-directional attention clearly outperforms causal attention, and this choice matters more than pooling. Mean pooling still beats last-token pooling.
- **Cross-lingual transfer:** zero-shot performance depends on whether a language was seen during pruning.
  - Yoruba was included in pruning, and there the 0.3B model even beats the 1B one.
  - Polish was unseen in both pruning and fine-tuning, and it lags noticeably.
- **Limitations:**
  - The 0.1B/0.3B variants cover 20 languages.
  - Training context is limited to 256 tokens.
  - Fine-tuning follows the E5 setup, which excludes domain-specific data such as finance and medicine.
  - The models target retrieval only, not broader embedding tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Waterloo](https://md.research.elamur.ai/free/en/universities/university-of-waterloo/) — University, Waterloo, Canada · 67 works · 13,592 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

## Cite

Cite as: Xueguang Ma, Xi Victoria Lin, Barlas Oguz, Jimmy Lin, Wen-tau Yih and Xilun Chen. 2025. DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ma2025drama,
  title = {{DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers}},
  author = {Xueguang Ma, Xi Victoria Lin, Barlas Oguz, Jimmy Lin, Wen-tau Yih and Xilun Chen},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1457},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1457}
}
```

- [← previous Do Construction Distributions Shape Formal Language Learning In German BabyLMs?](https://md.research.elamur.ai/free/en/papers/bunzeck-2025-do-construction-distributions-shape-formal/)
- [next → Dynamic Parallel Tree Search for Efficient LLM Reasoning](https://md.research.elamur.ai/free/en/papers/ding-2025-dynamic-parallel-tree-search-efficient/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/)

## References

48 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2023. [Task-aware retrieval with instructions](https://md.research.elamur.ai/free/en/papers/asai-2023-task-aware-retrieval-instructions/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 3650– 3675, Toronto, Canada. Association for Comput…
2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. [MS MARCO: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). arXiv:1611.09268…
3. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. [LLM2Vec: Large language models are secretly powerful text encoders](https://md.research.elamur.ai/free/en/papers/behnamghader-2024-llm2vec-large-language-models-secretly/). In First Conference on Language Modeling.
4. Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. InPars: Data augmentation for information retrieval using large language models. arXiv:2202.05144. [find](https://scholar.google.com/scholar?q=Luiz+Bonifacio%2C+Hugo+Abonizio%2C+Marzieh+Fadaee%2C+and+Rodrigo+Nogueira%2E+2022%2E+InPars%3A+Data+augmentation+for+information+retrieval+using+large+language+models%2E+arXiv%3A2202%2E05144%2E)
5. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. [M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bang…
6. Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. 2022. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? In Findings of the Association for Computational Linguistics: EMNLP 2022, page… [find](https://scholar.google.com/scholar?q=Xilun+Chen%2C+Kushal+Lakhotia%2C+Barlas+Oguz%2C+Anchit+Gupta%2C+Patrick+Lewis%2C+Stan+Peshterliev%2C+Yashar+Mehdad%2C+Sonal+Gupta%2C+and+Wen%2Dtau+Yih%2E+2022%2E+Salient+phrase+aware+dense+retrieval%3A+Can+a+dense+retriever+)
7. Mathieu Ciancone, Imene Kerboua, Marion Schaeffer, and Wissam Siblini. 2024. MTEB-French: Resources for french sentence embedding evaluation and analysis. arXiv:2405.20468. [find](https://scholar.google.com/scholar?q=Mathieu+Ciancone%2C+Imene+Kerboua%2C+Marion+Schaeffer%2C+and+Wissam+Siblini%2E+2024%2E+MTEB%2DFrench%3A+Resources+for+french+sentence+embedding+evaluation+and+analysis%2E+arXiv%3A2405%2E20468%2E)
8. Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2023. [Promptagator: Few-shot dense retrieval from 8 examples](https://md.research.elamur.ai/free/en/papers/dai-2022-promptagator-few-shot-dense-retrieval/). In ICLR.
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
10. Luyu Gao and Jamie Callan. 2021. [Condenser: a pre-training architecture for dense retrieval](https://md.research.elamur.ai/free/en/papers/gao-2021-condenser-pre-training-architecture-dense/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 981–993, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
12. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. [Unsupervised dense information retrieval with contrastive learning](https://md.research.elamur.ai/free/en/papers/izacard-2021-unsupervised-dense-information-retrieval-contrastive/). arXiv:2112.09118.
13. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
14. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2022. Matryoshka representation learning. In Advances in Neural Information Processing Systems, volume 35, pages 30233–30… [find](https://scholar.google.com/scholar?q=Aditya+Kusupati%2C+Gantavya+Bhatt%2C+Aniket+Rege%2C+Matthew+Wallingford%2C+Aditya+Sinha%2C+Vivek+Ramanujan%2C+William+Howard%2DSnyder%2C+Kaifeng+Chen%2C+Sham+Kakade%2C+Prateek+Jain%2C+and+Ali+Farhadi%2E+2022%2E+Matryoshka+repr)
15. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. [NV-Embed: Improved techniques for training LLMs as generalist embedding models](https://md.research.elamur.ai/free/en/papers/lee-2024-nv-embed-improved-techniques-training/). In The Thirteenth International Conference on Learning Representations.
16. Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, and Ift… — [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://md.research.elamur.ai/free/en/papers/lee-2024-gecko-versatile-text-embeddings-distilled/)
17. Chaofan Li, Minghao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Defu Lian, Yingxia Shao, and Zheng Liu. 2025. Making text embedders few-shot learners. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Chaofan+Li%2C+Minghao+Qin%2C+Shitao+Xiao%2C+Jianlyu+Chen%2C+Kun+Luo%2C+Defu+Lian%2C+Yingxia+Shao%2C+and+Zheng+Liu%2E+2025%2E+Making+text+embedders+few%2Dshot+learners%2E+In+The+Thirteenth+International+Conference+on+Learni)
18. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. [Towards general text embeddings with multi-stage contrastive learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/). arXiv:2308.03281.
19. Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023. [How to train your dragon: Diverse augmentation towards generalizable dense retrieval](https://md.research.elamur.ai/free/en/papers/lin-2023-how-train-your-dragon-diverse/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6385–6400, Singapor…
20. Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through l 0 regularization. arXiv:1712.01312. [find](https://scholar.google.com/scholar?q=Christos+Louizos%2C+Max+Welling%2C+and+Diederik+P%2E+Kingma%2E+2018%2E+Learning+sparse+neural+networks+through+l+0+regularization%2E+arXiv%3A1712%2E01312%2E)
21. Kun Luo, Minghao Qin, Zheng Liu, Shitao Xiao, Jun Zhao, and Kang Liu. 2024. Large language models as foundations for next-gen dense retrieval: A comprehensive empirical assessment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1354–1365, Miami, Flor… [find](https://scholar.google.com/scholar?q=Kun+Luo%2C+Minghao+Qin%2C+Zheng+Liu%2C+Shitao+Xiao%2C+Jun+Zhao%2C+and+Kang+Liu%2E+2024%2E+Large+language+models+as+foundations+for+next%2Dgen+dense+retrieval%3A+A+comprehensive+empirical+assessment%2E+In+Proceedings+of+t)
22. Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. [Fine-tuning llama for multi-stage text retrieval](https://md.research.elamur.ai/free/en/papers/ma-2023-fine-tuning-llama-multi-stage/). In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 2421–2425, New York, NY, USA. Association for Comput…
23. Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-shot listwise document reranking with a large language model. arXiv:2305.02156. [find](https://scholar.google.com/scholar?q=Xueguang+Ma%2C+Xinyu+Zhang%2C+Ronak+Pradeep%2C+and+Jimmy+Lin%2E+2023%2E+Zero%2Dshot+listwise+document+reranking+with+a+large+language+model%2E+arXiv%3A2305%2E02156%2E)
24. John Xavier Morris and Alexander M Rush. 2025. Contextual document embeddings. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=John+Xavier+Morris+and+Alexander+M+Rush%2E+2025%2E+Contextual+document+embeddings%2E+In+The+Thirteenth+International+Conference+on+Learning+Representations%2E)
25. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. [Generative representational instruction tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/). arXiv:2402.09906.
26. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022. [MTEB: Massive text embedding benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/). arXiv:2210.07316.
27. Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. 2024. [Nomic Embed: Training a reproducible long context text embedder](https://md.research.elamur.ai/free/en/papers/nussbaum-2024-nomic-embed-training-reproducible-long/). arXiv:2402.01613.
28. Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. [find](https://scholar.google.com/scholar?q=Stephen+E%2E+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+BM25+and+beyond%2E+Foundations+and+Trends+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2E)
29. Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen tau Yih, Pang Wei Koh, and Luke Zettlemoyer. 2025. ReasonIR: Training retrievers for reasoning tasks. arXiv:2504.20595. [find](https://scholar.google.com/scholar?q=Rulin+Shao%2C+Rui+Qiao%2C+Varsha+Kishore%2C+Niklas+Muennighoff%2C+Xi+Victoria+Lin%2C+Daniela+Rus%2C+Bryan+Kian+Hsiang+Low%2C+Sewon+Min%2C+Wen+tau+Yih%2C+Pang+Wei+Koh%2C+and+Luke+Zettlemoyer%2E+2025%2E+ReasonIR%3A+Training+retr)
30. Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. 2024. Repetition improves language model embeddings. arXiv:2402.15449. [find](https://scholar.google.com/scholar?q=Jacob+Mitchell+Springer%2C+Suhas+Kotha%2C+Daniel+Fried%2C+Graham+Neubig%2C+and+Aditi+Raghunathan%2E+2024%2E+Repetition+improves+language+model+embeddings%2E+arXiv%3A2402%2E15449%2E)
31. Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. [One embedder, any task: Instruction-finetuned text embeddings](https://md.research.elamur.ai/free/en/papers/su-2022-one-embedder-any-task-instruction/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 1102–1121, Toronto…
32. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. [Is ChatGPT good at search? investigating large language models as re-ranking agents](https://md.research.elamur.ai/free/en/papers/sun-2023-chatgpt-good-search-investigating-large/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1491…
33. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). [find](https://scholar.google.com/scholar?q=Nandan+Thakur%2C+Nils+Reimers%2C+Andreas+R%C3%BCckl%C3%A9%2C+Abhishek+Srivastava%2C+and+Iryna+Gurevych%2E+2021%2E+BEIR%3A+A+heterogeneous+benchmark+for+zero%2Dshot+evaluation+of+information+retrieval+models%2E+In+Thirty%2Dfifth+Co)
34. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. [Representation learning with contrastive predictive coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). arXiv:1807.03748.
35. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2023. [SimLM: Pre-training with representation bottleneck for dense passage retrieval](https://md.research.elamur.ai/free/en/papers/wang-2023-simlm-pre-training-representation-bottleneck/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon…
36. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024a. [Text embeddings by weakly-supervised contrastive pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/). arXiv:2212.03533.
37. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024b. [Improving text embeddings with large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-improving-text-embeddings-large-language/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, Bangkok, Thailand. Ass…
38. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024c. [Multilingual E5 text embeddings: A technical report](https://md.research.elamur.ai/free/en/papers/wang-2024-multilingual-e5-text-embeddings-technical/). arXiv:2402.05672.
39. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. [Smarter, better, faster, longer: A modern bidirectional encoder for fast,…](https://md.research.elamur.ai/free/en/papers/warner-2024-smarter-better-faster-longer-modern/)
40. Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. 2025. [FollowIR: Evaluating and teaching information retrieval models to follow instructions](https://md.research.elamur.ai/free/en/papers/weller-2025-followir-evaluating-teaching-information-retrieval/). In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Ass…
41. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Mengzhou+Xia%2C+Tianyu+Gao%2C+Zhiyuan+Zeng%2C+and+Danqi+Chen%2E+2024%2E+Sheared+LLaMA%3A+Accelerating+language+model+pre%2Dtraining+via+structured+pruning%2E+In+The+Twelfth+International+Conference+on+Learning+Repres)
42. Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022. [RetroMAE: Pre-training retrieval-oriented language models via masked auto-encoder](https://md.research.elamur.ai/free/en/papers/xiao-2022-retromae-pre-training-retrieval-oriented/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 538–548, Abu Dhabi, United Arab Emirates. Association for Comp…
43. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. [C-Pack: Packed resources for general chinese embeddings](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/). arXiv:2309.07597.
44. Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. 2024. Arctic-Embed 2.0: Multilingual retrieval without compromise. arXiv:2412.04506. — [arXiv:2412.04506](https://arxiv.org/abs/2412.04506) [find](https://scholar.google.com/scholar?q=Puxuan+Yu%2C+Luke+Merrick%2C+Gaurav+Nuti%2C+and+Daniel+Campos%2E+2024%2E+Arctic%2DEmbed+2%2E0%3A+Multilingual+retrieval+without+compromise%2E+arXiv%3A2412%2E04506%2E+%E2%80%94+%5BarXiv%3A2412%2E04506%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2412%2E04506%29)
45. Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2025. Jasper and Stella: distillation of SOTA embedding models. arXiv:2412.19048. — [arXiv:2412.19048](https://arxiv.org/abs/2412.19048) [find](https://scholar.google.com/scholar?q=Dun+Zhang%2C+Jiacheng+Li%2C+Ziyang+Zeng%2C+and+Fulong+Wang%2E+2025%2E+Jasper+and+Stella%3A+distillation+of+SOTA+embedding+models%2E+arXiv%3A2412%2E19048%2E+%E2%80%94+%5BarXiv%3A2412%2E19048%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2412%2E19048%29)
46. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. [mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval](https://md.research.elamur.ai/free/en/papers/zhang-2024-mgte-generalized-long-context-text/). arXiv:2407.19669.
47. Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. [MIRACL: A multilingual retrieval dataset covering 18 diverse languages](https://md.research.elamur.ai/free/en/papers/zhang-2023-miracl-multilingual-retrieval-dataset-covering/). Transactions of the Association for Computational Linguistics, 11:1114–1131.
48. Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2024. [LongEmbed: Extending embedding models for long context retrieval](https://md.research.elamur.ai/free/en/papers/zhu-2024-longembed-extending-embedding-models-long/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 802–816, Miami, Florida, USA. Association for Comp…

*Updated 31 October 2026*