---
title: "End-to-End Training of Neural Retrievers for Open-Domain"
description: "The paper examines how unsupervised and supervised signals can best be combined to train dense… 30 references, cited by 53 papers here."
authors: ["Devendra Sachan", "Mostofa Patwary", "Mohammad Shoeybi", "Neel Kant", "Wei Ping", "William L. Hamilton", "Bryan Catanzaro"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.519/"
doi: "10.18653/v1/2021.acl-long.519"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sachan-2021-end-end-training-neural-retrievers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sachan-2021-end-end-training-neural-retrievers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / End-to-End Training of Neural Retrievers for Open-Domain Question Answering

# End-to-End Training of Neural Retrievers for Open-Domain Question Answering

**Authors:** [Devendra Sachan](https://md.research.elamur.ai/free/en/authors/devendra-sachan/), [Mostofa Patwary](https://md.research.elamur.ai/free/en/authors/mostofa-patwary/), [Mohammad Shoeybi](https://md.research.elamur.ai/free/en/authors/mohammad-shoeybi/), [Neel Kant](https://md.research.elamur.ai/free/en/authors/neel-kant/), [Wei Ping](https://md.research.elamur.ai/free/en/authors/wei-ping/), [William L. Hamilton](https://md.research.elamur.ai/free/en/authors/william-l-hamilton/) and [Bryan Catanzaro](https://md.research.elamur.ai/free/en/authors/bryan-catanzaro/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.519/) · [PDF](https://aclanthology.org/2021.acl-long.519.pdf) · [DOI 10.18653/v1/2021.acl-long.519](https://doi.org/10.18653/v1/2021.acl-long.519) · [OpenAlex](https://openalex.org/W3119437245)

**Citations:** 62 citations (OpenAlex, 2026-10-14) · [53](https://md.research.elamur.ai/free/en/papers/sachan-2021-end-end-training-neural-retrievers/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper examines how unsupervised and supervised signals can best be combined to train dense dual-encoder retrievers for open-domain question answering (OpenQA), where the evidence is a Wikipedia collection of about 21M documents. The authors first pre-train BERT-initialized question and context encoders without labels, using the Inverse Cloze Task (ICT) or masked salient spans. They then finetune them on question–context pairs obtained via BM25 distant supervision, including hard negatives. A small but useful change is scaling relevance scores by √d. Next, they train the retriever and a T5-based generative reader jointly. In one variant the reader handles each retrieved passage on its own (Individual Top-k). In the other it reads all passages together (Joint Top-k). Evidence embeddings are refreshed asynchronously during training. The pipeline sets new state-of-the-art retrieval results on Natural Questions and TriviaQA and also improves answer extraction. The study clarifies when unsupervised pre-training adds value on top of strong supervised retrievers, which is useful for building QA systems on large corpora.

## Key points
- Pre-training with ICT and then supervised finetuning raises top-20 retrieval accuracy by more than 2 points over the previous best on Natural Questions and TriviaQA.
- Masked salient spans pre-training helps most when little supervised data is available.
- End-to-end training reaches 84% top-20 accuracy on Natural Questions, about 5 points above DPR.
- Answer extraction beats REALM and RAG by more than 3 points.
- The two end-to-end variants complement each other:
  - Individual Top-k mainly boosts retrieval.
  - Joint Top-k mainly boosts answer extraction, and similarity-enriched attention adds roughly 0.8–1 EM for top-k of 5–25.
  - Code is released in NVIDIA's Megatron-LM repository.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [McGill University](https://md.research.elamur.ai/free/en/universities/mcgill-university/) — University, Montreal, Canada · 51 works · 10,476 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 31 works · 13,061 citations
- [Nvidia (United Kingdom)](https://md.research.elamur.ai/free/en/companies/nvidia-united-kingdom/) — Company, Reading, United Kingdom · 13 works · 4,550 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (2): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 242 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 145 works

## Cite

Cite as: Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton and Bryan Catanzaro. 2021. End-to-End Training of Neural Retrievers for Open-Domain Question Answering. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sachan2021end,
  title = {{End-to-End Training of Neural Retrievers for Open-Domain Question Answering}},
  author = {Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton and Bryan Catanzaro},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.519},
  url = {https://doi.org/10.18653/v1/2021.acl-long.519}
}
```

- [← previous A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods](https://md.research.elamur.ai/free/en/papers/deutsch-2021-statistical-analysis-summarization-evaluation-metrics/)
- [next → Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features](https://md.research.elamur.ai/free/en/papers/lee-2021-pushing-text-readability-assessment-transformer/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

30 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexandr Andoni, Piotr Indyk, Thijs Laarhoven, Ilya Razenshteyn, and Ludwig Schmidt. 2015. Practical and optimal lsh for angular distance. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Alexandr+Andoni%2C+Piotr+Indyk%2C+Thijs+Laarhoven%2C+Ilya+Razenshteyn%2C+and+Ludwig+Schmidt%2E+2015%2E+Practical+and+optimal+lsh+for+angular+distance%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
2. Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1994. Signature verification using a ”siamese” time delay neural network. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Jane+Bromley%2C+Isabelle+Guyon%2C+Yann+LeCun%2C+Eduard+Sa%CC%88ckinger%2C+and+Roopak+Shah%2E+1994%2E+Signature+verification+using+a+%E2%80%9Dsiamese%E2%80%9D+time+delay+neural+network%2E+In+Advances+in+Neural+Information+Processing+Sys)
3. Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-based large-scale retrieval. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Wei%2DCheng+Chang%2C+Felix+X%2E+Yu%2C+Yin%2DWen+Chang%2C+Yiming+Yang%2C+and+Sanjiv+Kumar%2E+2020%2E+Pre%2Dtraining+tasks+for+embedding%2Dbased+large%2Dscale+retrieval%2E+In+International+Conference+on+Learning+Representations%2E)
4. Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Ph.D. thesis, Stanford University. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2E+2018%2E+Neural+Reading+Comprehension+and+Beyond%2E+Ph%2ED%2E+thesis%2C+Stanford+University%2E)
5. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading Wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
6. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In Proceedings of the 37th International Conference on Machine Learning.
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
8. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Kenton+Lee%2C+Zora+Tung%2C+Panupong+Pasupat%2C+and+Mingwei+Chang%2E+2020%2E+Retrieval+augmented+language+model+pre%2Dtraining%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+Learning%2E)
9. Gautier Izacard and Edouard Grave. 2020. [Leveraging passage retrieval with generative models for open domain question answering](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/). arXiv preprint arXiv:2007.01282.
10. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
11. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
12. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In The 2015 International Conference for Learning Representations. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+The+2015+International+Conference+for+Learning+Representations%2E)
13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
14. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Confe)
15. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. [Latent retrieval for weakly supervised open domain question answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
16. Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020a. [Pre-training via paraphrasing](https://md.research.elamur.ai/free/en/papers/lewis-2020-pre-training-paraphrasing/). In Advances in Neural Information Processing Systems.
17. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of th…
18. Patrick Lewis, Ethan Perez, Aleksandara Piktus, F. Petroni, V. Karpukhin, Naman Goyal, Heinrich Kuttler, M. Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020c. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing S… [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ethan+Perez%2C+Aleksandara+Piktus%2C+F%2E+Petroni%2C+V%2E+Karpukhin%2C+Naman+Goyal%2C+Heinrich+Kuttler%2C+M%2E+Lewis%2C+Wen+tau+Yih%2C+Tim+Rockta%CC%88schel%2C+Sebastian+Riedel%2C+and+Douwe+Kiela%2E+2020c%2E+Retrieval%2Dau)
19. Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. 2020. [Pytorch distributed: Experiences on accelerating data parallel training](https://md.research.elamur.ai/free/en/papers/li-2020-pytorch-distributed-experiences-accelerating-data/). Proc. VLDB Endow.
20. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Sharan+Narang%2C+Jonah+Alben%2C+Gregory+Diamos%2C+Erich+Elsen%2C+David+Garcia%2C+Boris+Ginsburg%2C+Michael+Houston%2C+Oleksii+Kuchaiev%2C+Ganesh+Venkatesh%2C+and+Hao+Wu%2E+2018%2E+Mixed+precision+trai)
21. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. [find](https://scholar.google.com/scholar?q=Mike+Mintz%2C+Steven+Bills%2C+Rion+Snow%2C+and+Daniel+Jurafsky%2E+2009%2E+Distant+supervision+for+relation+extraction+without+labeled+data%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+t)
22. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. [CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/). In Joint Conference on EMNLP and CoNLL - Shared Task.
23. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. [Stanza: A Python natural language processing toolkit for many human languages](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
25. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval. [find](https://scholar.google.com/scholar?q=Stephen+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+Bm25+and+beyond%2E+Foundations+and+Trends+in+Information+Retrieval%2E)
26. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+2019%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+gpu+model+parallelism%2E+arX)
27. Anshumali Shrivastava and Ping Li. 2014. Asymmetric lsh (alsh) for sublinear time maximum inner product search (mips). In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Anshumali+Shrivastava+and+Ping+Li%2E+2014%2E+Asymmetric+lsh+%28alsh%29+for+sublinear+time+maximum+inner+product+search+%28mips%29%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+27%2E+Curran+Associate)
28. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems.
30. Wen-tau Yih, Kristina Toutanova, John C. Platt, and Christopher Meek. 2011. [Learning discriminative projections for text similarity measures](https://md.research.elamur.ai/free/en/papers/yih-2011-learning-discriminative-projections-text-similarity/). In Proceedings of the Fifteenth Conference on Computational Natural Language Learning.

*Updated 24 October 2026*