---
title: "RetroMAE — paper by Xiao et al. (2022) — summary and ideas"
description: "These objectives do little to build strong whole-sentence embeddings, which limits the models as… 45 references, cited by 67 papers here."
authors: ["Shitao Xiao", "Zheng Liu", "Yingxia Shao", "Zhao Cao"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.35/"
doi: "10.18653/v1/2022.emnlp-main.35"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiao-2022-retromae-pre-training-retrieval-oriented/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiao-2022-retromae-pre-training-retrieval-oriented/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder

# RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder

**Authors:** [Shitao Xiao](https://md.research.elamur.ai/free/en/authors/shitao-xiao/), [Zheng Liu](https://md.research.elamur.ai/free/en/authors/zheng-liu/), Yingxia Shao and Zhao Cao · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.35/) · [PDF](https://aclanthology.org/2022.emnlp-main.35.pdf) · [DOI 10.18653/v1/2022.emnlp-main.35](https://doi.org/10.18653/v1/2022.emnlp-main.35) · [OpenAlex](https://openalex.org/W4385572770)

**Citations:** 78 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/xiao-2022-retromae-pre-training-retrieval-oriented/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Standard language models such as BERT learn mainly through token-level objectives. These objectives do little to build strong whole-sentence embeddings, which limits the models as encoders for dense retrieval. Contrastive pre-training could help, but it depends on good data augmentation and on many negatives. RetroMAE instead uses an auto-encoding scheme built around a masked auto-encoder.

Each sentence is corrupted twice with independent masks:
- **Encoder side:** a full BERT-base encoder sees a lightly masked copy (15–30%) and produces a [CLS] embedding.
- **Decoder side:** a single-layer transformer receives a heavily masked copy (50–70%) plus that embedding, and must restore the original tokens via MLM.

Because the decoder is so weak and sees so little, reconstruction depends heavily on the embedding. An enhanced decoding variant turns every token into a training target.

Pre-training used only moderate data: Wikipedia, BookCorpus and the MS MARCO corpus. The resulting retriever set new records for its model size, in both zero-shot and fine-tuned settings. This shows that cheap, augmentation-free pre-training can substantially strengthen dense retrievers.

## Key points
- **Architecture:** asymmetric, pairing a 12-layer, 768-dimensional BERT-like encoder with a one-layer decoder. Enhanced decoding draws on two-stream attention and position-specific attention masks, giving each token its own sampled context.
- **Results:** the model reaches an average of 45.2 on BEIR zero-shot. With standard knowledge distillation, it scores 41.6 MRR@10 on the MS MARCO passage-retrieval task. Natural Questions was also evaluated.
- **Decoder ablations:**
  - Enhanced decoding clearly beats basic decoding.
  - Deeper decoders (2–3 layers) give no benefit.
  - The best decoder masking ratio is 0.5 with enhanced decoding and 0.7 without it.
- **Encoder ablations:** masking 0.3 of encoder input works better than the usual 0.15, while 0.9 badly hurts quality.
- **Availability and limitations:** code and checkpoints are released at github.com/staoxiao/RetroMAE. Larger models and more pre-training data remain unexplored.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 29 works · 9,991 citations
- [Beijing University of Posts and Telecommunications](https://md.research.elamur.ai/free/en/universities/beijing-university-of-posts-and-telecommunications/) — University, Beijing, China · 22 works · 32,364 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 145 works

## Cite

Cite as: Shitao Xiao, Zheng Liu, Yingxia Shao and Zhao Cao. 2022. RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xiao2022retromae,
  title = {{RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder}},
  author = {Shitao Xiao, Zheng Liu, Yingxia Shao and Zhao Cao},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.35},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.35}
}
```

- [← previous Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks](https://md.research.elamur.ai/free/en/papers/mireshghallah-2022-quantifying-privacy-risks-masked-language/)
- [next → In-Context Learning for Few-Shot Dialogue State Tracking](https://md.research.elamur.ai/free/en/papers/hu-2022-context-learning-few-shot-dialogue/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

45 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hangbo Bao, Li Dong, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254. — [arXiv:2106.08254](https://arxiv.org/abs/2106.08254) [find](https://scholar.google.com/scholar?q=Hangbo+Bao%2C+Li+Dong%2C+and+Furu+Wei%2E+2021%2E+Beit%3A+Bert+pre%2Dtraining+of+image+transformers%2E+arXiv+preprint+arXiv%3A2106%2E08254%2E+%E2%80%94+%5BarXiv%3A2106%2E08254%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2106%2E08254%29)
2. Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-based large-scale retrieval. arXiv preprint arXiv:2002.03932. — [arXiv:2002.03932](https://arxiv.org/abs/2002.03932) [find](https://scholar.google.com/scholar?q=Wei%2DCheng+Chang%2C+Felix+X+Yu%2C+Yin%2DWen+Chang%2C+Yiming+Yang%2C+and+Sanjiv+Kumar%2E+2020%2E+Pre%2Dtraining+tasks+for+embedding%2Dbased+large%2Dscale+retrieval%2E+arXiv+preprint+arXiv%3A2002%2E03932%2E+%E2%80%94+%5BarXiv%3A2002%2E03932%5D%28htt)
3. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In International conference on machine learning, pages 1597–1607. PMLR.
4. Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Wen-tau Yih, Yoon Kim, and James R. Glass. 2022. [Diffcse: Difference-based contrastive learning for sentence embeddings](https://md.research.elamur.ai/free/en/papers/chuang-2022-diffcse-difference-based-contrastive-learning/). CoRR, abs/2204.10298.
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
6. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. [Unified language model pre-training for natural language understanding and generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). Advances in Neural Information Processing Systems, 32.
7. Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. 2021. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112.10740. — [arXiv:2112.10740](https://arxiv.org/abs/2112.10740) [find](https://scholar.google.com/scholar?q=Alaaeldin+El%2DNouby%2C+Gautier+Izacard%2C+Hugo+Touvron%2C+Ivan+Laptev%2C+Herve%CC%81+Jegou%2C+and+Edouard+Grave%2E+2021%2E+Are+large%2Dscale+datasets+necessary+for+self%2Dsupervised+pre%2Dtraining%3F+arXiv+preprint+arXiv%3A2112%2E10)
8. Luyu Gao and Jamie Callan. 2021. [Condenser: a pre-training architecture for dense retrieval](https://md.research.elamur.ai/free/en/papers/gao-2021-condenser-pre-training-architecture-dense/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 981–993.
9. Luyu Gao and Jamie Callan. 2022. [Unsupervised corpus aware language model pre-training for dense passage retrieval](https://md.research.elamur.ai/free/en/papers/gao-2022-unsupervised-corpus-aware-language-model/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2843–2853, Dublin, Ireland.
10. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. [Simcse: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). arXiv preprint arXiv:2104.08821.
11. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909. [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Kenton+Lee%2C+Zora+Tung%2C+Panupong+Pasupat%2C+and+Ming%2DWei+Chang%2E+2020%2E+Realm%3A+Retrieval%2Daugmented+language+model+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2002%2E08909%2E)
12. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020a. [Momentum contrast for unsupervised visual representation learning](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738.
13. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020b. [Deberta: Decoding-enhanced bert with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). arXiv preprint arXiv:2006.03654.
14. Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117–128. [find](https://scholar.google.com/scholar?q=Herve+Jegou%2C+Matthijs+Douze%2C+and+Cordelia+Schmid%2E+2010%2E+Product+quantization+for+nearest+neighbor+search%2E+IEEE+transactions+on+pattern+analysis+and+machine+intelligence%2C+33%281%29%3A117%E2%80%93128%2E)
15. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. [Spanbert: Improving pre-training by representing and predicting spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/). Transactions of the Association for Computational Linguistics, 8:64–77.
16. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6769–6781.
17. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistic…
18. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+CoRR%2C+abs%2F1907%2E11692%2E)
19. Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021. Less is more: Pretrain a strong Siamese encoder for dense text retrieval using a weak decoder. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… [find](https://scholar.google.com/scholar?q=Shuqi+Lu%2C+Di+He%2C+Chenyan+Xiong%2C+Guolin+Ke%2C+Waleed+Malik%2C+Zhicheng+Dou%2C+Paul+Bennett%2C+Tie%2DYan+Liu%2C+and+Arnold+Overwijk%2E+2021%2E+Less+is+more%3A+Pretrain+a+strong+Siamese+encoder+for+dense+text+retrieval+us)
20. Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020. Twinbert: Distilling knowledge to twin-structured bert models for efficient retrieval. arXiv preprint arXiv:2002.06275. — [arXiv:2002.06275](https://arxiv.org/abs/2002.06275) [find](https://scholar.google.com/scholar?q=Wenhao+Lu%2C+Jian+Jiao%2C+and+Ruofei+Zhang%2E+2020%2E+Twinbert%3A+Distilling+knowledge+to+twin%2Dstructured+bert+models+for+efficient+retrieval%2E+arXiv+preprint+arXiv%3A2002%2E06275%2E+%E2%80%94+%5BarXiv%3A2002%2E06275%5D%28https%3A%2F%2Farxiv)
21. Yuxiang Lu, Yiding Liu, Jiaxiang Liu, Yunsheng Shi, Zhengjie Huang, Shikun Feng Yu Sun, Hao Tian, Hua Wu, Shuaiqiang Wang, Dawei Yin, et al. 2022. Ernie-search: Bridging cross-encoder with dual-encoder via self on-the-fly distillation for dense passage retrieval. arXiv preprint arXiv:2205.09153. — [arXiv:2205.09153](https://arxiv.org/abs/2205.09153) [find](https://scholar.google.com/scholar?q=Yuxiang+Lu%2C+Yiding+Liu%2C+Jiaxiang+Liu%2C+Yunsheng+Shi%2C+Zhengjie+Huang%2C+Shikun+Feng+Yu+Sun%2C+Hao+Tian%2C+Hua+Wu%2C+Shuaiqiang+Wang%2C+Dawei+Yin%2C+et+al%2E+2022%2E+Ernie%2Dsearch%3A+Bridging+cross%2Dencoder+with+dual%2Dencode)
22. Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. [Sparse, dense, and attentional representations for text retrieval](https://md.research.elamur.ai/free/en/papers/luan-2021-sparse-dense-attentional-representations-text/). Transactions of the Association for Computational Linguistics, 9:329–345.
23. Yu A Malkov and Dmitry A Yashunin. 2018. [Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs](https://md.research.elamur.ai/free/en/papers/malkov-2016-efficient-robust-approximate-nearest-neighbor/). IEEE transactions on pattern analysis and machine intelligence, 42(4):824–836.
24. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. [Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). In CoCo@ NIPS.
25. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-Wei Chang, et al. 2021. [Large dual encoders are generalizable retrievers](https://md.research.elamur.ai/free/en/papers/ni-2021-large-dual-encoders-generalizable-retrievers/). arXiv preprint arXiv:2112.07899.
26. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461.
27. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2020. [Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/). arXiv preprint arXiv:2010.08191.
28. Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. 2020. [Optimus: Organizing sentences via pre-trained modeling of a latent space](https://md.research.elamur.ai/free/en/papers/li-2020-optimus-organizing-sentences-pre-trained/). arXiv preprint arXiv:2004.04092.
29. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
30. Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021. Pretrained transformers for text ranking: Bert and beyond. Synthesis Lectures on Human Language Technologies, 14(4):1–325. [find](https://scholar.google.com/scholar?q=Jimmy+Lin%2C+Rodrigo+Nogueira%2C+and+Andrew+Yates%2E+2021%2E+Pretrained+transformers+for+text+ranking%3A+Bert+and+beyond%2E+Synthesis+Lectures+on+Human+Language+Technologies%2C+14%284%29%3A1%E2%80%93325%2E)
31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683.
32. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E)
33. Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. [Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking](https://md.research.elamur.ai/free/en/papers/ren-2021-rocketqav2-joint-training-method-dense/). arXiv preprint arXiv:2110.07367.
34. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2021. [Colbertv2: Effective and efficient retrieval via lightweight late interaction](https://md.research.elamur.ai/free/en/papers/santhanam-2022-colbertv2-effective-efficient-retrieval-lightweight/). arXiv preprint arXiv:2112.01488.
35. Jianjin Zhang, Zheng Liu, Weihao Han, Shitao Xiao, Ruicheng Zheng, Yingxia Shao, Hao Sun, Hanqing Zhu, Premkumar Srinivasan, Weiwei Deng, et al. 2022. Uni-retriever: Towards learning the unified embedding based retriever in bing sponsored search. In Proceedings of the 28th ACM SIGKDD Conference on K… [find](https://scholar.google.com/scholar?q=Jianjin+Zhang%2C+Zheng+Liu%2C+Weihao+Han%2C+Shitao+Xiao%2C+Ruicheng+Zheng%2C+Yingxia+Shao%2C+Hao+Sun%2C+Hanqing+Zhu%2C+Premkumar+Srinivasan%2C+Weiwei+Deng%2C+et+al%2E+2022%2E+Uni%2Dretriever%3A+Towards+learning+the+unified+embed)
36. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. [Ernie: Enhanced representation through knowledge integration](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/). arXiv preprint arXiv:1904.09223.
37. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. [Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models](https://md.research.elamur.ai/free/en/papers/thakur-2021-beir-heterogenous-benchmark-zero/). arXiv preprint arXiv:2104.08663.
38. Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. Tsdae: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning. arXiv preprint arXiv:2104.06979. — [arXiv:2104.06979](https://arxiv.org/abs/2104.06979) [find](https://scholar.google.com/scholar?q=Kexin+Wang%2C+Nils+Reimers%2C+and+Iryna+Gurevych%2E+2021%2E+Tsdae%3A+Using+transformer%2Dbased+sequential+denoising+auto%2Dencoder+for+unsupervised+sentence+embedding+learning%2E+arXiv+preprint+arXiv%3A2104%2E06979%2E+%E2%80%94+%5Ba)
39. Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, et al. 2022a. Distill-vq: Learning retrieval oriented vector quantization by distilling knowledge from dense embeddings. arXiv preprint arXiv:2204.00185. — [arXiv:2204.00185](https://arxiv.org/abs/2204.00185) [find](https://scholar.google.com/scholar?q=Shitao+Xiao%2C+Zheng+Liu%2C+Weihao+Han%2C+Jianjin+Zhang%2C+Defu+Lian%2C+Yeyun+Gong%2C+Qi+Chen%2C+Fan+Yang%2C+Hao+Sun%2C+Yingxia+Shao%2C+et+al%2E+2022a%2E+Distill%2Dvq%3A+Learning+retrieval+oriented+vector+quantization+by+distill)
40. Shitao Xiao, Zheng Liu, Yingxia Shao, Tao Di, Bhuvan Middha, Fangzhao Wu, and Xing Xie. 2022b. Training large-scale news recommenders with pretrained language models in the loop. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4215–4225. [find](https://scholar.google.com/scholar?q=Shitao+Xiao%2C+Zheng+Liu%2C+Yingxia+Shao%2C+Tao+Di%2C+Bhuvan+Middha%2C+Fangzhao+Wu%2C+and+Xing+Xie%2E+2022b%2E+Training+large%2Dscale+news+recommenders+with+pretrained+language+models+in+the+loop%2E+In+Proceedings+of+the)
41. Shitao Xiao, Zheng Liu, Yingxia Shao, Defu Lian, and Xing Xie. 2021. Matching-oriented product quantization for ad-hoc retrieval. arXiv preprint arXiv:2104.07858. — [arXiv:2104.07858](https://arxiv.org/abs/2104.07858) [find](https://scholar.google.com/scholar?q=Shitao+Xiao%2C+Zheng+Liu%2C+Yingxia+Shao%2C+Defu+Lian%2C+and+Xing+Xie%2E+2021%2E+Matching%2Doriented+product+quantization+for+ad%2Dhoc+retrieval%2E+arXiv+preprint+arXiv%3A2104%2E07858%2E+%E2%80%94+%5BarXiv%3A2104%2E07858%5D%28https%3A%2F%2Farxiv%2Eor)
42. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. [Approximate nearest neighbor negative contrastive learning for dense text retrieval](https://md.research.elamur.ai/free/en/papers/xiong-2020-approximate-nearest-neighbor-negative-contrastive/). arXiv preprint arXiv:2007.00808.
43. Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2022. Laprador: Unsupervised pretrained dense retriever for zero-shot text retrieval. arXiv preprint arXiv:2203.06169. [find](https://scholar.google.com/scholar?q=Canwen+Xu%2C+Daya+Guo%2C+Nan+Duan%2C+and+Julian+McAuley%2E+2022%2E+Laprador%3A+Unsupervised+pretrained+dense+retriever+for+zero%2Dshot+text+retrieval%2E+arXiv+preprint+arXiv%3A2203%2E06169%2E)
44. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). Advances in neural information processing systems, 32.
45. Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021. Adversarial retriever-ranker for dense text retrieval. arXiv preprint arXiv:2110.03611. — [arXiv:2110.03611](https://arxiv.org/abs/2110.03611) [find](https://scholar.google.com/scholar?q=Hang+Zhang%2C+Yeyun+Gong%2C+Yelong+Shen%2C+Jiancheng+Lv%2C+Nan+Duan%2C+and+Weizhu+Chen%2E+2021%2E+Adversarial+retriever%2Dranker+for+dense+text+retrieval%2E+arXiv+preprint+arXiv%3A2110%2E03611%2E+%E2%80%94+%5BarXiv%3A2110%2E03611%5D%28https%3A%2F)

*Updated 24 October 2026*