---
title: "SimLM — paper by Wang et al. (2023) — summary and ideas"
description: "Dense retrievers usually start from general-purpose language models. 49 references, cited by 64 papers here."
authors: ["Liang Wang", "Nan Yang", "Xiaolong Huang", "Binxing Jiao", "Linjun Yang", "Daxin Jiang", "Rangan Majumder", "Furu Wei"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.125/"
doi: "10.18653/v1/2023.acl-long.125"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2023-simlm-pre-training-representation-bottleneck/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2023-simlm-pre-training-representation-bottleneck/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

# SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

**Authors:** [Liang Wang](https://md.research.elamur.ai/free/en/authors/liang-wang/), [Nan Yang](https://md.research.elamur.ai/free/en/authors/nan-yang/), [Xiaolong Huang](https://md.research.elamur.ai/free/en/authors/xiaolong-huang/), [Binxing Jiao](https://md.research.elamur.ai/free/en/authors/binxing-jiao/), [Linjun Yang](https://md.research.elamur.ai/free/en/authors/linjun-yang/), [Daxin Jiang](https://md.research.elamur.ai/free/en/authors/daxin-jiang/), [Rangan Majumder](https://md.research.elamur.ai/free/en/authors/rangan-majumder/) and [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) · **Year:** 2023 · **Venue:** [Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2023](https://md.research.elamur.ai/free/en/venues/acl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.125/) · [PDF](https://aclanthology.org/2023.acl-long.125.pdf) · [DOI 10.18653/v1/2023.acl-long.125](https://doi.org/10.18653/v1/2023.acl-long.125) · [OpenAlex](https://openalex.org/W4385571915)

**Citations:** 42 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/wang-2023-simlm-pre-training-representation-bottleneck/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Dense retrievers usually start from general-purpose language models. However, models that score better on GLUE do not necessarily retrieve better. On MS-MARCO, BERT outperforms RoBERTa and ELECTRA. The authors attribute this to the [CLS] vector not being trained to capture a whole passage. SimLM is a self-supervised pre-training scheme that needs only unlabeled text. A deep Transformer encoder, initialized from BERT, feeds its [CLS] vector as a bottleneck into a weak 2-layer bidirectional decoder. A frozen ELECTRA-style generator corrupts the input. Both encoder and decoder must then recover the original tokens at every position. Because the decoder is shallow and has no skip connections, the encoder is pushed to pack passage meaning into the bottleneck. Removing [MASK] tokens also narrows the gap between pre-training and fine-tuning. After pre-training on the target corpus, the encoder initializes a multi-stage bi-encoder pipeline that uses hard-negative mining and re-ranker distillation. On MS-MARCO, the TREC DL tracks and NQ, SimLM beats strong baselines. It even surpasses the multi-vector ColBERTv2 while needing far less storage.

## Key points
- **Motivation (Table 1):** BERT, RoBERTa and ELECTRA reach MRR@10 of 33.7, 33.1 and 31.9 on MS-MARCO, yet their GLUE scores rise in the opposite order (80.5, 88.1, 89.4).
- **Training objective:**
  - The pre-training loss is the sum of the encoder and decoder replaced-language-modeling cross-entropies.
  - Fine-tuning uses a contrastive loss with temperature-scaled cosine similarity (τ = 0.02).
- **Efficiency:** SimLM is competitive after only 10k pre-training steps and converges by about 60k, while MLM keeps improving slowly.
- **Effect of pre-training corpus (Table 10):**
  - With BM25 negatives, MS-MARCO MRR@10 rises from 33.7 (BERT base) to 38.0 when pre-training on MS-MARCO.
  - Pre-training on Wikipedia gives 36.3, so the non-target corpus still helps.
  - On NQ with mined negatives, Wikipedia pre-training lifts R@20 from 82.9 to 84.3 and R@100 from 88.0 to 89.3.
- **Limitations and availability:**
  - SimLM cannot act as a zero-shot retriever because pre-training has no contrastive objective, so labeled fine-tuning is required.
  - Code and checkpoints are released at github.com/microsoft/unilm (simlm).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 240 works · 123,537 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 145 works

## Cite

Cite as: Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder and Furu Wei. 2023. SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2023simlm,
  title = {{SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval}},
  author = {Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder and Furu Wei},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.125},
  url = {https://doi.org/10.18653/v1/2023.acl-long.125}
}
```

- [← previous Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://md.research.elamur.ai/free/en/papers/allen-zhu-2023-physics-language-models-part-3/)
- [next → TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models](https://md.research.elamur.ai/free/en/papers/gekhman-2023-trueteacher-learning-factual-consistency-evaluation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

49 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dr. Hiteshwar Kumar Azad and Akshay Deepak. 2019. Query expansion techniques for information retrieval: a survey. Inf. Process. Manag., 56:1698– 1735. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Dr%2E+Hiteshwar+Kumar+Azad+and+Akshay+Deepak%2E+2019%2E+Query+expansion+techniques+for+information+retrieval%3A+a+survey%2E+Inf%2E+Process%2E+Manag%2E%2C+56%3A1698%E2%80%93+1735%2E+8th+International+Conference+on+Learning+Represen)
2. Xilun Chen, Kushal Lakhotia, Barlas Oğuz, Anchit Gupta, Patrick Lewis, Stanislav Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen tau Yih. 2021. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? ArXiv, abs/2110.06918. [find](https://scholar.google.com/scholar?q=Xilun+Chen%2C+Kushal+Lakhotia%2C+Barlas+Og%CC%86uz%2C+Anchit+Gupta%2C+Patrick+Lewis%2C+Stanislav+Peshterliev%2C+Yashar+Mehdad%2C+Sonal+Gupta%2C+and+Wen+tau+Yih%2E+2021%2E+Salient+phrase+aware+dense+retrieval%3A+Can+a+dense+retr)
3. Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. [DiffCSE: Difference-based contrastive learning for sentence embeddings](https://md.research.elamur.ai/free/en/papers/chuang-2022-diffcse-difference-based-contrastive-learning/). In Proceedings of the 2022 Conference of the North American Chapter of the Associ…
4. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+ELECTRA%3A+pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+8th+International+Conference+on+Learning+R)
5. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020a. [Overview of the trec 2019 deep learning track](https://md.research.elamur.ai/free/en/papers/craswell-2020-overview-trec-2019-deep-learning/). ArXiv preprint, abs/2003.07820.
6. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and Ellen M. Voorhees. 2020b. [Overview of the trec 2020 deep learning track](https://md.research.elamur.ai/free/en/papers/craswell-2020-overview-trec-2019-deep-learning/). ArXiv, abs/2003.07820.
7. Zhuyun Dai and Jamie Callan. 2019. Context-aware sentence/passage term importance estimation for first stage retrieval. ArXiv, abs/1910.10687. — [arXiv:1910.10687](https://arxiv.org/abs/1910.10687) [find](https://scholar.google.com/scholar?q=Zhuyun+Dai+and+Jamie+Callan%2E+2019%2E+Context%2Daware+sentence%2Fpassage+term+importance+estimation+for+first+stage+retrieval%2E+ArXiv%2C+abs%2F1910%2E10687%2E+%E2%80%94+%5BarXiv%3A1910%2E10687%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E10687%29)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
9. Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. [Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). ArXiv, abs/1611.09268.
10. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. [Unified language model pre-training for natural language understanding and generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information…
11. Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-based large-scale retrieval. In Luyu Gao and Jamie Callan. 2021. [Condenser: a pre-training architecture for dense retrieval](https://md.research.elamur.ai/free/en/papers/gao-2021-condenser-pre-training-architecture-dense/). In Proceedings of the 2021 Conference on Empirical Methods i…
12. [Natural questions: A benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7:452–466.
13. Luyu Gao and Jamie Callan. 2022. [Unsupervised corpus aware language model pre-training for dense passage retrieval](https://md.research.elamur.ai/free/en/papers/gao-2022-unsupervised-corpus-aware-language-model/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2843–2853, Dublin, Ireland. Association for Computational Ling…
14. Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Informatio… [find](https://scholar.google.com/scholar?q=Patrick+S%2E+H%2E+Lewis%2C+Ethan+Perez%2C+Aleksandra+Piktus%2C+Fabio+Petroni%2C+Vladimir+Karpukhin%2C+Naman+Goyal%2C+Heinrich+K%C3%BCttler%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Tim+Rockt%C3%A4schel%2C+Sebastian+Riedel%2C+and+Douwe+Kiela%2E+2020)
15. Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. [COIL: Revisit exact lexical match in information retrieval with contextualized inverted list](https://md.research.elamur.ai/free/en/papers/gao-2021-coil-revisit-exact-lexical-match/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3030–30…
16. Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy J. Lin, and Allan Hanbury. 2021. [Efficiently teaching an effective dense retriever with balanced topic aware sampling](https://md.research.elamur.ai/free/en/papers/hofstatter-2021-efficiently-teaching-effective-dense-retriever/). Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval.
17. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In 22nd ACM International Conference on Information and Knowledge Management, CIKM’13, San Francisco, CA, USA, October 27 - November… [find](https://scholar.google.com/scholar?q=Po%2DSen+Huang%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+Li+Deng%2C+Alex+Acero%2C+and+Larry+P%2E+Heck%2E+2013%2E+Learning+deep+structured+semantic+models+for+web+search+using+clickthrough+data%2E+In+22nd+ACM+International+Confer)
18. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. [Unsupervised dense information retrieval with contrastive learning](https://md.research.elamur.ai/free/en/papers/izacard-2021-unsupervised-dense-information-retrieval-contrastive/).
19. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769– 6781, Onl…
20. Omar Khattab and Matei Zaharia. 2020. [Colbert: Efficient and effective passage search via contextualized late interaction over BERT](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/). In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SI-GIR 2020, Virtual Event, China, July 25-30, 2020…
21. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Redfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+Toutanova%2C+Llion+Jones%2C+Matt)
22. Jimmy J. Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, Rodrigo Nogueira, and David R. Cheriton. 2021. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. Proceedings of the 44th International ACM SIGIR Conference on R… [find](https://scholar.google.com/scholar?q=Jimmy+J%2E+Lin%2C+Xueguang+Ma%2C+Sheng%2DChieh+Lin%2C+Jheng%2DHong+Yang%2C+Ronak+Pradeep%2C+Rodrigo+Nogueira%2C+and+David+R%2E+Cheriton%2E+2021%2E+Pyserini%3A+A+python+toolkit+for+reproducible+information+retrieval+research+wi)
23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+A)
24. Zheng Liu and Yingxia Shao. 2022. Retromae: Pre-training retrieval-oriented transformers via masked auto-encoder. ArXiv, abs/2205.12035. — [arXiv:2205.12035](https://arxiv.org/abs/2205.12035) [find](https://scholar.google.com/scholar?q=Zheng+Liu+and+Yingxia+Shao%2E+2022%2E+Retromae%3A+Pre%2Dtraining+retrieval%2Doriented+transformers+via+masked+auto%2Dencoder%2E+ArXiv%2C+abs%2F2205%2E12035%2E+%E2%80%94+%5BarXiv%3A2205%2E12035%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2205%2E12035%29)
25. Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021. Less is more: Pre-train a strong Siamese encoder for dense text retrieval using a weak decoder. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language… [find](https://scholar.google.com/scholar?q=Shuqi+Lu%2C+Di+He%2C+Chenyan+Xiong%2C+Guolin+Ke%2C+Waleed+Malik%2C+Zhicheng+Dou%2C+Paul+Bennett%2C+Tie%2DYan+Liu%2C+and+Arnold+Overwijk%2E+2021%2E+Less+is+more%3A+Pre%2Dtrain+a+strong+Siamese+encoder+for+dense+text+retrieval+u)
26. Xinyu Ma, J. Guo, Ruqing Zhang, Yixing Fan, and Xueqi Cheng. 2022. Pre-train a discriminative text encoder for dense retrieval via contrastive span prediction. ArXiv, abs/2204.10641. — [arXiv:2204.10641](https://arxiv.org/abs/2204.10641) [find](https://scholar.google.com/scholar?q=Xinyu+Ma%2C+J%2E+Guo%2C+Ruqing+Zhang%2C+Yixing+Fan%2C+and+Xueqi+Cheng%2E+2022%2E+Pre%2Dtrain+a+discriminative+text+encoder+for+dense+retrieval+via+contrastive+span+prediction%2E+ArXiv%2C+abs%2F2204%2E10641%2E+%E2%80%94+%5BarXiv%3A2204%2E106)
27. Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xiang Ji, and Xueqi Cheng. 2021. Prop: Pre-training with representative words prediction for ad-hoc retrieval. Proceedings of the 14th ACM International Conference on Web Search and Data Mining. [find](https://scholar.google.com/scholar?q=Xinyu+Ma%2C+Jiafeng+Guo%2C+Ruqing+Zhang%2C+Yixing+Fan%2C+Xiang+Ji%2C+and+Xueqi+Cheng%2E+2021%2E+Prop%3A+Pre%2Dtraining+with+representative+words+prediction+for+ad%2Dhoc+retrieval%2E+Proceedings+of+the+14th+ACM+Internationa)
28. Yu A. Malkov and D. A. Yashunin. 2020. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:824–836. [find](https://scholar.google.com/scholar?q=Yu+A%2E+Malkov+and+D%2E+A%2E+Yashunin%2E+2020%2E+Efficient+and+robust+approximate+nearest+neighbor+search+using+hierarchical+navigable+small+world+graphs%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intel)
29. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2005. Introduction to information retrieval. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Prabhakar+Raghavan%2C+and+Hinrich+Sch%C3%BCtze%2E+2005%2E+Introduction+to+information+retrieval%2E)
30. [Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas A. Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David P. Schnurr, Felipe Petroski Such, Kenny Sai-Ki…](https://md.research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/)
31. Rodrigo Nogueira and Jimmy Lin. From doc2query to doctttttquery. [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira+and+Jimmy+Lin%2E+From+doc2query+to+doctttttquery%2E)
32. Rodrigo Nogueira, Wei Yang, Jimmy J. Lin, and Kyunghyun Cho. 2019. [Document expansion by query prediction](https://md.research.elamur.ai/free/en/papers/nogueira-2019-document-expansion-query-prediction/). ArXiv, abs/1904.08375.
33. Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, et al. 2022. Domain-matched pre-training tasks for dense retrieval. In Findings of the Association for Computational Linguistics: NAACL 2022, pages… [find](https://scholar.google.com/scholar?q=Barlas+Oguz%2C+Kushal+Lakhotia%2C+Anchit+Gupta%2C+Patrick+Lewis%2C+Vladimir+Karpukhin%2C+Aleksandra+Piktus%2C+Xilun+Chen%2C+Sebastian+Riedel%2C+Scott+Yih%2C+Sonal+Gupta%2C+et+al%2E+2022%2E+Domain%2Dmatched+pre%2Dtraining+tasks+f)
34. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. [RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/). In Proceedings of the 2021 Conference of the North American Chapter of the Ass…
35. Nils Reimers and Iryna Gurevych. 2021. The curse of dense low-dimensional information retrieval for large index sizes. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Shor… [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2021%2E+The+curse+of+dense+low%2Ddimensional+information+retrieval+for+large+index+sizes%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Li)
36. Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021a. PAIR: Leveraging passage-centric similarity relation for improving dense passage retrieval. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages… [find](https://scholar.google.com/scholar?q=Ruiyang+Ren%2C+Shangwen+Lv%2C+Yingqi+Qu%2C+Jing+Liu%2C+Wayne+Xin+Zhao%2C+QiaoQiao+She%2C+Hua+Wu%2C+Haifeng+Wang%2C+and+Ji%2DRong+Wen%2E+2021a%2E+PAIR%3A+Leveraging+passage%2Dcentric+similarity+relation+for+improving+dense+pass)
37. Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021b. [RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking](https://md.research.elamur.ai/free/en/papers/ren-2021-rocketqav2-joint-training-method-dense/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2825–…
38. Keshav Santhanam, O. Khattab, Jon Saad-Falcon, Christopher Potts, and Matei A. Zaharia. 2021. [find](https://scholar.google.com/scholar?q=Keshav+Santhanam%2C+O%2E+Khattab%2C+Jon+Saad%2DFalcon%2C+Christopher+Potts%2C+and+Matei+A%2E+Zaharia%2E+2021%2E)
39. [Colbertv2: Effective and efficient retrieval via lightweight late interaction](https://md.research.elamur.ai/free/en/papers/santhanam-2022-colbertv2-effective-efficient-retrieval-lightweight/). ArXiv, abs/2112.01488.
40. Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. [Simple entity-centric questions challenge dense retrievers](https://md.research.elamur.ai/free/en/papers/sciavolino-2021-simple-entity-centric-questions-challenge/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6138–6148, Online and Punta Cana, Dominican Republic. Association for…
41. Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. Learning semantic representations using convolutional neural networks for web search. Proceedings of the 23rd International Conference on World Wide Web. [find](https://scholar.google.com/scholar?q=Yelong+Shen%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+Li+Deng%2C+and+Gr%C3%A9goire+Mesnil%2E+2014%2E+Learning+semantic+representations+using+convolutional+neural+networks+for+web+search%2E+Proceedings+of+the+23rd+International+)
42. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. [The fact extraction and VERification (FEVER) shared task](https://md.research.elamur.ai/free/en/papers/thorne-2018-fact-extraction-verification-fever-shared/). In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 1–9, Brussels, Belgium. Association for Computational…
43. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevie…
44. Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. [Gpl: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval](https://md.research.elamur.ai/free/en/papers/wang-2022-gpl-generative-pseudo-labeling-unsupervised/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
45. Liang Wang, Nan Yang, and Furu Wei. 2023. [Query2doc: Query expansion with large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-query2doc-query-expansion-large-language/). ArXiv, abs/2303.07678.
46. Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2022. Should you mask 15% in masked language modeling? ArXiv, abs/2202.08005. [find](https://scholar.google.com/scholar?q=Alexander+Wettig%2C+Tianyu+Gao%2C+Zexuan+Zhong%2C+and+Danqi+Chen%2E+2022%2E+Should+you+mask+15%25+in+masked+language+modeling%3F+ArXiv%2C+abs%2F2202%2E08005%2E)
47. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. [Approximate nearest neighbor negative contrastive learning for dense text retrieval](https://md.research.elamur.ai/free/en/papers/xiong-2020-approximate-nearest-neighbor-negative-contrastive/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria,…
48. Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. Pretrained transformers for text ranking: BERT and beyond. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials, pages 1–4, Online. Association… [find](https://scholar.google.com/scholar?q=Andrew+Yates%2C+Rodrigo+Nogueira%2C+and+Jimmy+Lin%2E+2021%2E+Pretrained+transformers+for+text+ranking%3A+BERT+and+beyond%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+f)
49. Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021. Adversarial retriever-ranker for dense text retrieval. ArXiv preprint, abs/2110.03611. — [arXiv:2110.03611](https://arxiv.org/abs/2110.03611) [find](https://scholar.google.com/scholar?q=Hang+Zhang%2C+Yeyun+Gong%2C+Yelong+Shen%2C+Jiancheng+Lv%2C+Nan+Duan%2C+and+Weizhu+Chen%2E+2021%2E+Adversarial+retriever%2Dranker+for+dense+text+retrieval%2E+ArXiv+preprint%2C+abs%2F2110%2E03611%2E+%E2%80%94+%5BarXiv%3A2110%2E03611%5D%28https%3A%2F%2F)

*Updated 24 October 2026*