---
title: "PAQ: 65 Million Probably-Asked Questions and — Lewis 2021"
description: "Open-domain QA systems that work directly with question-answer pairs, such as closed-book (CBQA) models and QA-pair retrievers, are fast, compact and…"
authors: ["Patrick Lewis", "Yuxiang Wu", "Linqing Liu", "Pasquale Minervini", "Heinrich Küttler", "Aleksandra Piktus", "Pontus Stenetorp", "Sebastian Riedel"]
year: 2021
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2021.tacl-1.65/"
doi: "10.1162/tacl_a_00415"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lewis-2021-paq-65-million-probably-asked/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lewis-2021-paq-65-million-probably-asked/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them

# PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them

**Authors:** [Patrick Lewis](https://md.research.elamur.ai/free/en/authors/patrick-lewis/), [Yuxiang Wu](https://md.research.elamur.ai/free/en/authors/yuxiang-wu/), Linqing Liu, [Pasquale Minervini](https://md.research.elamur.ai/free/en/authors/pasquale-minervini/), Heinrich Küttler, [Aleksandra Piktus](https://md.research.elamur.ai/free/en/authors/aleksandra-piktus/), [Pontus Stenetorp](https://md.research.elamur.ai/free/en/authors/pontus-stenetorp/), [Sebastian Riedel](https://md.research.elamur.ai/free/en/authors/sebastian-riedel/) · **Institutions:** [Meta](https://md.research.elamur.ai/free/en/companies/meta/) · [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) · [Meta (United Kingdom)](https://md.research.elamur.ai/free/en/companies/meta-united-kingdom/) · **Year:** 2021 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.tacl-1.65/) · [PDF](https://aclanthology.org/2021.tacl-1.65.pdf) · [DOI 10.1162/tacl\_a\_00415](https://doi.org/10.1162/tacl_a_00415) · [OpenAlex](https://openalex.org/W3132730484)

**Citations:** 146 citations (OpenAlex, 2026-10-14) · [140](https://md.research.elamur.ai/free/en/papers/lewis-2021-paq-65-million-probably-asked/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Open-domain QA systems that work directly with question-answer pairs, such as closed-book (CBQA) models and QA-pair retrievers, are fast, compact and easy to inspect and update. They are less accurate than retrieve-and-read systems because existing QA collections hold far less knowledge than Wikipedia. The authors address this gap by automatically building PAQ, a collection of about 65 million questions likely to be asked, each paired with its answer. The generation pipeline has four stages: it picks passages that people tend to ask about, extracts candidate answer spans, generates questions with BART, and keeps a pair only if a full open-domain QA system recovers the same answer. To exploit this resource they propose RePAQ, which answers a new question by finding the most similar stored questions through dense retrieval, with optional re-ranking. RePAQ reaches accuracy comparable to retrieve-and-read models like RAG while running much faster. Its confidence scores reliably signal likely errors. This lets RePAQ hand uncertain questions to a stronger model, producing a hybrid that is both more accurate and quicker.

## Key points
- PAQ contains 65M QA-pairs generated from Wikipedia. Passage selection uses RoBERTa trained on Natural Questions, answers come from spaCy NER or a BERT span extractor, and questions come from BART-base fine-tuned on NQ, TriviaQA and SQuAD.
- Global filtering checks each pair with an open-domain QA model rather than only the source passage. This removes many ambiguous or wrong questions and is key to downstream accuracy.
- RePAQ matches RAG-level results on NaturalQuestions and TriviaQA. It can be shrunk below 500MB or tuned to answer over 1,000 questions per second.
- BART CBQA models trained on PAQ beat standard-data baselines by about 5%, yet remain more than 15% behind RePAQ. This suggests that explicit retrieval works better than parametric memorization.
- Routing low-confidence questions to a stronger model such as FiD gives a combined system that is more accurate and roughly 2x faster than that model alone. Data, models and code are available at github.com/facebookresearch/PAQ.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{lewis2021paq,
  title = {{PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them}},
  author = {Lewis, Patrick and Wu, Yuxiang and Liu, Linqing and Minervini, Pasquale and Küttler, Heinrich and Piktus, Aleksandra and Stenetorp, Pontus and Riedel, Sebastian},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00415},
  url = {https://doi.org/10.1162/tacl_a_00415}
}
```

- [← previous Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://md.research.elamur.ai/free/en/papers/schuster-2021-get-your-vitamin-c-robust/)
- [next → Contextualized Perturbation for Textual Adversarial Attack](https://md.research.elamur.ai/free/en/papers/li-2021-contextualized-perturbation-textual-adversarial-attack/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

66 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6168–6173, Florence, Italy. Association for Computational… [find](https://scholar.google.com/scholar?q=Chris+Alberti%2C+Daniel+Andor%2C+Emily+Pitler%2C+Jacob+Devlin%2C+and+Michael+Collins%2E+2019%2E+Synthetic+QA+corpora+generation+with+roundtrip+consistency%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associa) [our card](https://md.research.elamur.ai/free/en/papers/alberti-2019-synthetic-qa-corpora-generation-roundtrip/)
2. Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lang… [find](https://scholar.google.com/scholar?q=Gabor+Angeli%2C+Melvin+Jose+Johnson+Premkumar%2C+and+Christopher+D%2E+Manning%2E+2015%2E+Leveraging+linguistic+structure+for+open+domain+information+extraction%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+) [our card](https://md.research.elamur.ai/free/en/papers/angeli-2015-leveraging-linguistic-structure-open-domain/)
3. Akari Asai and Eunsol Choi. 2020. Challenges in information seeking QA: Unanswerable questions and paragraph retrieval. arXiv:2010 .11915 [cs]. ArXiv: 2010.11915. [find](https://scholar.google.com/scholar?q=Akari+Asai+and+Eunsol+Choi%2E+2020%2E+Challenges+in+information+seeking+QA%3A+Unanswerable+questions+and+paragraph+retrieval%2E+arXiv%3A2010+%2E11915+%5Bcs%5D%2E+ArXiv%3A+2010%2E11915%2E)
4. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jonathan+Berant%2C+Andrew+Chou%2C+Roy+Frostig%2C+and+Percy+Liang%2E+2013%2E+Semantic+parsing+on+freebase+from+question%2Danswer+pairs%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+Languag) [our card](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/)
5. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, SIGMOD ’08, pages 1247–1250, Vancouver, Canada.… [find](https://scholar.google.com/scholar?q=Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge%2C+and+Jamie+Taylor%2E+2008%2E+Freebase%3A+A+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+Proceedings+of+the+2008+ACM+SI)
6. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Nicola+De+Cao%2C+Gautier+Izacard%2C+Sebastian+Riedel%2C+and+Fabio+Petroni%2E+2021%2E+Autoregressive+entity+retrieval%2E+In+International+Conference+on+Learning+Representations%2E)
7. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879, Vancouver, Canada. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Adam+Fisch%2C+Jason+Weston%2C+and+Antoine+Bordes%2E+2017%2E+Reading+Wikipedia+to+answer+open%2Ddomain+questions%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Lingui) [our card](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/)
8. Danqi Chen and Wen-tau Yih. 2020. Open-domain question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 34–37, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Danqi+Chen+and+Wen%2Dtau+Yih%2E+2020%2E+Open%2Ddomain+question+answering%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Tutorial+Abstracts%2C+pages+34%E2%80%9337%2C+Online%2E+A)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conferenceof the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conferenceof+the+No) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
10. Pedro Domingos. 2020. Every Model Learned by Gradient Descent Is Approximately a Kernel Machine. arXiv:2012.00152 [cs, stat]. ArXiv: 2012.00152. — [arXiv:2012.00152](https://arxiv.org/abs/2012.00152) [find](https://scholar.google.com/scholar?q=Pedro+Domingos%2E+2020%2E+Every+Model+Learned+by+Gradient+Descent+Is+Approximately+a+Kernel+Machine%2E+arXiv%3A2012%2E00152+%5Bcs%2C+stat%5D%2E+ArXiv%3A+2012%2E00152%2E+%E2%80%94+%5BarXiv%3A2012%2E00152%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2012%2E00152%29)
11. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1342–1352, Vancouver, Canada. Association for Computational… [find](https://scholar.google.com/scholar?q=Xinya+Du%2C+Junru+Shao%2C+and+Claire+Cardie%2E+2017%2E+Learning+to+ask%3A+Neural+question+generation+for+reading+comprehension%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Lin) [our card](https://md.research.elamur.ai/free/en/papers/du-2017-learning-ask-neural-question-generation/)
12. Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. 2020. Accelerating real-time question answering via question generation. arXiv:2009.05167 [cs]. ArXiv: 2009.05167. — [arXiv:2009.05167](https://arxiv.org/abs/2009.05167) [find](https://scholar.google.com/scholar?q=Yuwei+Fang%2C+Shuohang+Wang%2C+Zhe+Gan%2C+Siqi+Sun%2C+and+Jingjing+Liu%2E+2020%2E+Accelerating+real%2Dtime+question+answering+via+question+generation%2E+arXiv%3A2009%2E05167+%5Bcs%5D%2E+ArXiv%3A+2009%2E05167%2E+%E2%80%94+%5BarXiv%3A2009%2E05167%5D%28)
13. Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051–2060, Melbourne, Australia. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Nicholas+FitzGerald%2C+Julian+Michael%2C+Luheng+He%2C+and+Luke+Zettlemoyer%2E+2018%2E+Large%2Dscale+QA%2DSRL+parsing%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volu) [our card](https://md.research.elamur.ai/free/en/papers/fitzgerald-2018-large-scale-qa-srl-parsing/)
14. Jerome H. Friedman. 2001. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29(5):1189–1232. [find](https://scholar.google.com/scholar?q=Jerome+H%2E+Friedman%2E+2001%2E+Greedy+function+approximation%3A+A+gradient+boosting+machine%2E+Annals+of+Statistics%2C+29%285%29%3A1189%E2%80%931232%2E)
15. Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020. Entities as experts: Sparse memory access with entity supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4937–4951, Online. Associati… [find](https://scholar.google.com/scholar?q=Thibault+Fe%CC%81vry%2C+Livio+Baldini+Soares%2C+Nicholas+FitzGerald%2C+Eunsol+Choi%2C+and+Tom+Kwiatkowski%2E+2020%2E+Entities+as+experts%3A+Sparse+memory+access+with+entity+supervision%2E+In+Proceedings+of+the+2020+Confer) [our card](https://md.research.elamur.ai/free/en/papers/fevry-2020-entities-as-experts-sparse-memory/)
16. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938, PMLR. [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Kenton+Lee%2C+Zora+Tung%2C+Panupong+Pasupat%2C+and+Mingwei+Chang%2E+2020%2E+Retrieval+augmented+language+model+pre%2Dtraining%2E+In+Proceedings+of+the+37th+International+Conference+on+Machine+Learning%2C+)
17. Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength natural language processing in python. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal%2C+Ines+Montani%2C+Sofie+Van+Landeghem%2C+and+Adriane+Boyd%2E+2020%2E+spaCy%3A+Industrial%2Dstrength+natural+language+processing+in+python%2E)
18. Tom Hosking and Sebastian Riedel. 2019. Evaluating rewards for question generation models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2278–2283, Minneapolis… [find](https://scholar.google.com/scholar?q=Tom+Hosking+and+Sebastian+Riedel%2E+2019%2E+Evaluating+rewards+for+question+generation+models%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Ling)
19. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, NAACL-Short ’06, pages 57–60, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eduard+Hovy%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Lance+Ramshaw%2C+and+Ralph+Weischedel%2E+2006%2E+OntoNotes%3A+The+90%25+solution%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+NAACL%2C+Companion+)
20. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880, Online. Association for Computa… [find](https://scholar.google.com/scholar?q=Gautier+Izacard+and+Edouard+Grave%2E+2021%2E+Leveraging+passage+retrieval+with+generative+models+for+open+domain+question+answering%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+As) [our card](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/)
21. Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave. 2020. A memory efficient baseline for open domain question answering. arXiv: 2012.15156 [cs]. ArXiv: 2012.15156. — [arXiv:2012.15156](https://arxiv.org/abs/2012.15156) [find](https://scholar.google.com/scholar?q=Gautier+Izacard%2C+Fabio+Petroni%2C+Lucas+Hosseini%2C+Nicola+De+Cao%2C+Sebastian+Riedel%2C+and+Edouard+Grave%2E+2020%2E+A+memory+efficient+baseline+for+open+domain+question+answering%2E+arXiv%3A+2012%2E15156+%5Bcs%5D%2E+ArXiv%3A)
22. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. How can we know when language models know? arXiv:2012 .00955 [cs]. ArXiv: 2012.00955. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Jun+Araki%2C+Haibo+Ding%2C+and+Graham+Neubig%2E+2020a%2E+How+can+we+know+when+language+models+know%3F+arXiv%3A2012+%2E00955+%5Bcs%5D%2E+ArXiv%3A+2012%2E00955%2E)
23. Zhengbao Jiang, Wei Xu, Jun Araki, and Graham Neubig. 2020b. Generalizing natural language analysis through span-relation representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2120–2133, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Wei+Xu%2C+Jun+Araki%2C+and+Graham+Neubig%2E+2020b%2E+Generalizing+natural+language+analysis+through+span%2Drelation+representations%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+)
24. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, pages 1–1. [find](https://scholar.google.com/scholar?q=Jeff+Johnson%2C+Matthijs+Douze%2C+and+Herve%CC%81+Je%CC%81gou%2E+2019%2E+Billion%2Dscale+similarity+search+with+GPUs%2E+IEEE+Transactions+on+Big+Data%2C+pages+1%E2%80%931%2E) [our card](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/)
25. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver… [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+Proceedings+of+the+55th+Annual+Meeting+) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
26. H. Jégou, M. Douze, and C. Schmid. 2011. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128. https://doi.org/10.1109/TPAMI.2010.57 — [doi:10.1109/tpami.2010.57](https://doi.org/10.1109/tpami.2010.57) [↗](https://doi.org/10.1109/TPAMI.2010.57)
27. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli… [find](https://scholar.google.com/scholar?q=Vladimir+Karpukhin%2C+Barlas+Oguz%2C+Sewon+Min%2C+Patrick+Lewis%2C+Ledell+Wu%2C+Sergey+Edunov%2C+Danqi+Chen%2C+and+Wen%2Dtau+Yih%2E+2020%2E+Dense+Passage+retrieval+for+open%2Ddomain+question+answering%2E+In+Proceedings+of+th) [our card](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)
28. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Redfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Matthew+Kelcey%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+N%2E+Toutanova) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
29. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Confe)
30. Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021. Learning dense representations of phrases at scale. arXiv:2012.12624 [cs]. ArXiv: 2012.12624. [find](https://scholar.google.com/scholar?q=Jinhyuk+Lee%2C+Mujeen+Sung%2C+Jaewoo+Kang%2C+and+Danqi+Chen%2E+2021%2E+Learning+dense+representations+of+phrases+at+scale%2E+arXiv%3A2012%2E12624+%5Bcs%5D%2E+ArXiv%3A+2012%2E12624%2E)
31. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Ming%2DWei+Chang%2C+and+Kristina+Toutanova%2E+2019%2E+Latent+retrieval+for+weakly+supervised+open+domain+question+answering%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Comput) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)
32. Mike Lewis and Angela Fan. 2018. Generative question answering: Learning to answer the whole question. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Mike+Lewis+and+Angela+Fan%2E+2018%2E+Generative+question+answering%3A+Learning+to+answer+the+whole+question%2E+In+International+Conference+on+Learning+Representations%2E)
33. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of th… [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020a%2E+BART%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
34. Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019. Unsupervised question answering by cloze translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4896–4910, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ludovic+Denoyer%2C+and+Sebastian+Riedel%2E+2019%2E+Unsupervised+question+answering+by+cloze+translation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguis)
35. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information P… [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ethan+Perez%2C+Aleksandra+Piktus%2C+Fabio+Petroni%2C+Vladimir+Karpukhin%2C+Naman+Goyal%2C+Heinrich+Ku%CC%88ttler%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Tim+Rockta%CC%88schel%2C+Sebastian+Riedel%2C+and+Douwe+Kiela%2E+2020b%2E+R)
36. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000–1008, Online. Associ… [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2021%2E+Question+and+answer+test%2Dtrain+overlap+in+open%2Ddomain+question+answering+datasets%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Ch)
37. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv: 1907.11692 [cs]. ArXiv: 1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
38. Yu A. Malkov and D. A. Yashunin. 2020. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836. https://doi.org/10 .1109/TPAMI.2018.2889473 [find](https://scholar.google.com/scholar?q=Yu+A%2E+Malkov+and+D%2E+A%2E+Yashunin%2E+2020%2E+Efficient+and+robust+approximate+nearest+neighbor+search+using+hierarchical+navigable+small+world+graphs%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intel)
39. Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick Lewis, Yuxiang Wu, Heinrich Küttler, Linqing Liu, Pasquale Minervini, Pontus Stenetorp, Seba… [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Jordan+Boyd%2DGraber%2C+Chris+Alberti%2C+Danqi+Chen%2C+Eunsol+Choi%2C+Michael+Collins%2C+Kelvin+Guu%2C+Hannaneh+Hajishirzi%2C+Kenton+Lee%2C+Jennimaria+Palomaki%2C+Colin+Raffel%2C+Adam+Roberts%2C+Tom+Kwiatkowski%2C+P)
40. NeurIPS 2020 EfficientQA Competition: Systems, analyses and lessons learned. arXiv:2101.00133 [cs]. ArXiv: 2101.00133. — [arXiv:2101.00133](https://arxiv.org/abs/2101.00133) [find](https://scholar.google.com/scholar?q=NeurIPS+2020+EfficientQA+Competition%3A+Systems%2C+analyses+and+lessons+learned%2E+arXiv%3A2101%2E00133+%5Bcs%5D%2E+ArXiv%3A+2101%2E00133%2E+%E2%80%94+%5BarXiv%3A2101%2E00133%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2101%2E00133%29)
41. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020b. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783–5797, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2020b%2E+AmbigQA%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+L)
42. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. arXiv:1904.08375 [cs]. ArXiv: 1904.08375. [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira%2C+Wei+Yang%2C+Jimmy+Lin%2C+and+Kyunghyun+Cho%2E+2019%2E+Document+expansion+by+query+prediction%2E+arXiv%3A1904%2E08375+%5Bcs%5D%2E+ArXiv%3A+1904%2E08375%2E) [our card](https://md.research.elamur.ai/free/en/papers/nogueira-2019-document-expansion-query-prediction/)
43. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Sergey+Edunov%2C+Alexei+Baevski%2C+Angela+Fan%2C+Sam+Gross%2C+Nathan+Ng%2C+David+Grangier%2C+and+Michael+Auli%2E+2019%2E+fairseq%3A+A+fast%2C+extensible+toolkit+for+sequence+modeling%2E+In+Proceedings+of+the+2019) [our card](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)
44. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and S… [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Francisco+Massa%2C+Adam+Lerer%2C+James+Bradbury%2C+Gregory+Chanan%2C+Trevor+Killeen%2C+Zeming+Lin%2C+Natalia+Gimelshein%2C+Luca+Antiga%2C+Alban+Desmaison%2C+Andreas+Kopf%2C+Edward+Yang%2C+Zachary+De) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
45. Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models’ factual predictions. In Automated Knowledge Base Construction. [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Patrick+Lewis%2C+Aleksandra+Piktus%2C+Tim+Rockta%CC%88schel%2C+Yuxiang+Wu%2C+Alexander+H%2E+Miller%2C+and+Sebastian+Riedel%2E+2020%2E+How+context+affects+language+models%E2%80%99+factual+predictions%2E+In+Automated+K) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2020-context-affects-language-models-factual/)
46. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: A benchmark for knowledge intensive language tasks. In Proceedings of t… [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Aleksandra+Piktus%2C+Angela+Fan%2C+Patrick+Lewis%2C+Majid+Yazdani%2C+Nicola+De+Cao%2C+James+Thorne%2C+Yacine+Jernite%2C+Vladimir+Karpukhin%2C+Jean+Maillard%2C+Vassilis+Plachouras%2C+Tim+Rockta%CC%88schel%2C+and+S) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2020-kilt-benchmark-knowledge-intensive-language/)
47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex) [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
48. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+SQuAD%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
49. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Langu)
50. Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, and Jordan Boyd-Graber. 2019. Quizbowl: The case for incremental question answering. arXiv:1904.04792 [cs]. ArXiv: 1904.04792. — [arXiv:1904.04792](https://arxiv.org/abs/1904.04792) [find](https://scholar.google.com/scholar?q=Pedro+Rodriguez%2C+Shi+Feng%2C+Mohit+Iyyer%2C+He+He%2C+and+Jordan+Boyd%2DGraber%2E+2019%2E+Quizbowl%3A+The+case+for+incremental+question+answering%2E+arXiv%3A1904%2E04792+%5Bcs%5D%2E+ArXiv%3A+1904%2E04792%2E+%E2%80%94+%5BarXiv%3A1904%2E04792%5D%28https)
51. Minjoon Seo, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2018. Phrase-indexed question answering: A new challenge for scalable document comprehension. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 559–564, Brussels, Belgium.… [find](https://scholar.google.com/scholar?q=Minjoon+Seo%2C+Tom+Kwiatkowski%2C+Ankur+Parikh%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2018%2E+Phrase%2Dindexed+question+answering%3A+A+new+challenge+for+scalable+document+comprehension%2E+In+Proceedings+of+the+20)
52. Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019. Real-time open-domain question answering with dense-sparse phrase index. [find](https://scholar.google.com/scholar?q=Minjoon+Seo%2C+Jinhyuk+Lee%2C+Tom+Kwiatkowski%2C+Ankur+Parikh%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2019%2E+Real%2Dtime+open%2Ddomain+question+answering+with+dense%2Dsparse+phrase+index%2E) [our card](https://md.research.elamur.ai/free/en/papers/seo-2019-real-time-open-domain-question/)
53. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4430–4441, Florence, Italy. Association for Computational Linguistics. Iulian Vlad Serban, Alberto Garcı́a-Durán, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, and Yoshua Bengio. 2016.… [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4430%E2%80%934441%2C+Florence%2C+Italy%2E+Association+for+Computational+Linguistics%2E+Iulian+Vlad+Serban%2C+Alberto+Gar)
54. R. F. Simmons. 1965. Answering English questions by computer: A survey. Communications of the Association for Computing Machinery, 8(1):53–70. https://doi.org/10.1145/363707.363732 [↗](https://doi.org/10.1145/363707.363732)
55. M. Surdeanu. 2013. Overview of the TAC2013 KnowledgeBasePopulationEvaluation:English Slot Filling and Temporal Slot Filling. TAC. [find](https://scholar.google.com/scholar?q=M%2E+Surdeanu%2E+2013%2E+Overview+of+the+TAC2013+KnowledgeBasePopulationEvaluation%3AEnglish+Slot+Filling+and+Temporal+Slot+Filling%2E+TAC%2E)
56. Pat Verga, Haitian Sun, Livio Baldini Soares, and William W. Cohen. 2020. Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge. arXiv:2007.00849 [cs]. ArXiv: 2007.00849. — [arXiv:2007.00849](https://arxiv.org/abs/2007.00849) [find](https://scholar.google.com/scholar?q=Pat+Verga%2C+Haitian+Sun%2C+Livio+Baldini+Soares%2C+and+William+W%2E+Cohen%2E+2020%2E+Facts+as+experts%3A+Adaptable+and+interpretable+neural+memory+over+symbolic+knowledge%2E+arXiv%3A2007%2E00849+%5Bcs%5D%2E+ArXiv%3A+2007%2E00849%2E)
57. Ellen M. Voorhees. 1999. The TREC-8 Question Answering Track Report. In Proceedings of TREC-8, pages 77–82. [find](https://scholar.google.com/scholar?q=Ellen+M%2E+Voorhees%2E+1999%2E+The+TREC%2D8+Question+Answering+Track+Report%2E+In+Proceedings+of+TREC%2D8%2C+pages+77%E2%80%9382%2E) [our card](https://md.research.elamur.ai/free/en/papers/voorhees-2000-trec-8-question-answering-track/)
58. Ellen M. Voorhees. 2002. Overview of the TREC 2002 question answering track. In Proceedings of The Eleventh Text REtrieval Conference, TREC 2002, Gaithersburg, Maryland, USA, November 19–22, 2002, volume 500–251 of NIST Special Publication. National Institute of Standards and Technology (NIST). [find](https://scholar.google.com/scholar?q=Ellen+M%2E+Voorhees%2E+2002%2E+Overview+of+the+TREC+2002+question+answering+track%2E+In+Proceedings+of+The+Eleventh+Text+REtrieval+Conference%2C+TREC+2002%2C+Gaithersburg%2C+Maryland%2C+USA%2C+November+19%E2%80%9322%2C+2002%2C+vol)
59. Ellen M. Voorhees and Donna K. Harman, editors. 1999. Proceedings of The Eighth Text REtrieval Conference, TREC 1999, Gaithersburg, Maryland, USA, November 17–19, 1999, volume 500–246 of NIST Special Publication. National Institute of Standards and Technology (NIST). [find](https://scholar.google.com/scholar?q=Ellen+M%2E+Voorhees+and+Donna+K%2E+Harman%2C+editors%2E+1999%2E+Proceedings+of+The+Eighth+Text+REtrieval+Conference%2C+TREC+1999%2C+Gaithersburg%2C+Maryland%2C+USA%2C+November+17%E2%80%9319%2C+1999%2C+volume+500%E2%80%93246+of+NIST+Special+)
60. William Wang, Angelina Wang, Aviv Tamar, Xi Chen, and Pieter Abbeel. 2018. Safer classification by synthesis. arXiv:1711.08534 [cs, stat]. ArXiv: 1711.08534. — [arXiv:1711.08534](https://arxiv.org/abs/1711.08534) [find](https://scholar.google.com/scholar?q=William+Wang%2C+Angelina+Wang%2C+Aviv+Tamar%2C+Xi+Chen%2C+and+Pieter+Abbeel%2E+2018%2E+Safer+classification+by+synthesis%2E+arXiv%3A1711%2E08534+%5Bcs%2C+stat%5D%2E+ArXiv%3A+1711%2E08534%2E+%E2%80%94+%5BarXiv%3A1711%2E08534%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
61. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+Remi+Louf%2C+Morgan+Funtowicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara+M) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
62. Jinfeng Xiao, Lidan Wang, Franck Dernoncourt, Trung Bui, Tong Sun, and Jiawei Han. 2021. Open-domain question answering with pre-constructed question spaces. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Worksho… [find](https://scholar.google.com/scholar?q=Jinfeng+Xiao%2C+Lidan+Wang%2C+Franck+Dernoncourt%2C+Trung+Bui%2C+Tong+Sun%2C+and+Jiawei+Han%2E+2021%2E+Open%2Ddomain+question+answering+with+pre%2Dconstructed+question+spaces%2E+In+Proceedings+of+the+2021+Conference+of+t)
63. Wei Yang, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. Data augmentation for BERT fine-tuning in open-domain question answering. arXiv:1904.06652 [cs]. ArXiv: 1904.06652. — [arXiv:1904.06652](https://arxiv.org/abs/1904.06652) [find](https://scholar.google.com/scholar?q=Wei+Yang%2C+Yuqing+Xie%2C+Luchen+Tan%2C+Kun+Xiong%2C+Ming+Li%2C+and+Jimmy+Lin%2E+2019%2E+Data+augmentation+for+BERT+fine%2Dtuning+in+open%2Ddomain+question+answering%2E+arXiv%3A1904%2E06652+%5Bcs%5D%2E+ArXiv%3A+1904%2E06652%2E+%E2%80%94+%5BarXiv%3A)
64. Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, and Madian Khabsa. 2021. Studying strategically: Learning to mask for closed-book QA. arXiv:2012.15856 [cs]. ArXiv: 2012.15856. — [arXiv:2012.15856](https://arxiv.org/abs/2012.15856) [find](https://scholar.google.com/scholar?q=Qinyuan+Ye%2C+Belinda+Z%2E+Li%2C+Sinong+Wang%2C+Benjamin+Bolte%2C+Hao+Ma%2C+Wen%2Dtau+Yih%2C+Xiang+Ren%2C+and+Madian+Khabsa%2E+2021%2E+Studying+strategically%3A+Learning+to+mask+for+closed%2Dbook+QA%2E+arXiv%3A2012%2E15856+%5Bcs%5D%2E+ArX)
65. A Appendices A.1 Details on Dataset Splits For NQ we use the standard open-domain splits introduced by Lee et al. (2019), and train-development splits used by Karpukhin et al. (2020). For TriviaQA, we use the standard open-domain splits, which correspond to the unfiltered-train and unfiltered-dev re… [find](https://scholar.google.com/scholar?q=A+Appendices+A%2E1+Details+on+Dataset+Splits+For+NQ+we+use+the+standard+open%2Ddomain+splits+introduced+by+Lee+et+al%2E+%282019%29%2C+and+train%2Ddevelopment+splits+used+by+Karpukhin+et+al%2E+%282020%29%2E+For+TriviaQA%2C+we)
66. A.2 Further Details on Passage Selection [find](https://scholar.google.com/scholar?q=A%2E2+Further+Details+on+Passage+Selection)