---
title: "Simple Entity-Centric Questions Challenge Dense Retrievers"
description: "The authors build EntityQuestions from 24 common Wikidata relations. 31 references, cited by 87 papers here."
authors: ["Christopher Sciavolino", "Zexuan Zhong", "Jinhyuk Lee", "Danqi Chen"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.496/"
doi: "10.18653/v1/2021.emnlp-main.496"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sciavolino-2021-simple-entity-centric-questions-challenge/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sciavolino-2021-simple-entity-centric-questions-challenge/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Simple Entity-Centric Questions Challenge Dense Retrievers

# Simple Entity-Centric Questions Challenge Dense Retrievers

**Authors:** Christopher Sciavolino, [Zexuan Zhong](https://md.research.elamur.ai/free/en/authors/zexuan-zhong/), [Jinhyuk Lee](https://md.research.elamur.ai/free/en/authors/jinhyuk-lee/) and [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.496/) · [PDF](https://aclanthology.org/2021.emnlp-main.496.pdf) · [DOI 10.18653/v1/2021.emnlp-main.496](https://doi.org/10.18653/v1/2021.emnlp-main.496) · [OpenAlex](https://openalex.org/W3201233724)

**Citations:** 81 citations (OpenAlex, 2026-10-14) · [87](https://md.research.elamur.ai/free/en/papers/sciavolino-2021-simple-entity-centric-questions-challenge/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Dense retrievers such as DPR are often seen as having replaced sparse methods like BM25 in open-domain QA. This paper shows that they fail on very simple factoid questions about entities. The authors build EntityQuestions from 24 common Wikidata relations. They sample facts from T-REx, so each answer is backed by Wikipedia text, and turn every triple into a question with a hand-written template, for example asking where a person was born. On this benchmark DPR falls far behind BM25, even when trained on several QA datasets. The authors then separate the effect of entities from the effect of question wording. DPR handles popular entities and entities seen in training well, but it degrades on rare ones. It generalizes to new entities mostly when the question template appeared during fine-tuning. Data augmentation helps the targeted relation but transfers poorly to other relations and can hurt NQ. Pairing a more robust passage encoder (trained on PAQ) with specialized question encoders comes close to BM25. The study exposes robustness limits of dense retrieval and motivates entity-aware retrievers.

## Key points
- EntityQuestions covers 24 Wikidata relations, with up to 1,000 template-based questions per relation drawn from T-REx triples. Dataset and code are released at github.com/princeton-nlp/EntityQuestions.
- Average top-20 accuracy is 49.7% for DPR (NQ), 56.7% for DPR (multi-dataset) and 71.2% for BM25. On NQ the ranking flips, with DPR (NQ) at 80.1 and BM25 at 64.5. On place-of-birth questions BM25 beats DPR by 49.8 points. REALM averages only 19.6%.
- Fine-tuning on a single pattern raises DPR's place-of-birth accuracy from 25.4 to 73.9. Updating only the passage encoder reaches 72.8, while updating only the question encoder reaches just 45.4.
- With the passage index fixed, per-relation question encoders reach 68.4 on top of DPR-PAQ, compared with 61.6 on top of DPR-NQ and 71.2 for BM25.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 61 works · 36,471 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (1): [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 145 works

## Cite

Cite as: Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee and Danqi Chen. 2021. Simple Entity-Centric Questions Challenge Dense Retrievers. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sciavolino2021simple,
  title = {{Simple Entity-Centric Questions Challenge Dense Retrievers}},
  author = {Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee and Danqi Chen},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.496},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.496}
}
```

- [← previous On Releasing Annotator-Level Labels and Information in Datasets](https://md.research.elamur.ai/free/en/papers/prabhakaran-2021-releasing-annotator-level-labels-information/)
- [next → Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction](https://md.research.elamur.ai/free/en/papers/sainz-2021-label-verbalization-entailment-effective-zero/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

31 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Petr Baudiš and Jan Šedivỳ. 2015. Modeling of the question answering task in the yodaqa system. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 222– 228. Springer. [find](https://scholar.google.com/scholar?q=Petr+Baudi%C5%A1+and+Jan+%C5%A0edivy%CC%80%2E+2015%2E+Modeling+of+the+question+answering+task+in+the+yodaqa+system%2E+In+International+Conference+of+the+Cross%2DLanguage+Evaluation+Forum+for+European+Languages%2C+pages+222%E2%80%93+2)
2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. [Semantic parsing on Freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics.
3. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Nicola+De+Cao%2C+Gautier+Izacard%2C+Sebastian+Riedel%2C+and+Fabio+Petroni%2E+2021%2E+Autoregressive+entity+retrieval%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
4. Anthony Chen, Pallavi Gudipati, Shayne Longpre, Xiao Ling, and Sameer Singh. 2021. Evaluating entity disambiguation and the role of popularity in retrieval-based NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc… [find](https://scholar.google.com/scholar?q=Anthony+Chen%2C+Pallavi+Gudipati%2C+Shayne+Longpre%2C+Xiao+Ling%2C+and+Sameer+Singh%2E+2021%2E+Evaluating+entity+disambiguation+and+the+role+of+popularity+in+retrieval%2Dbased+NLP%2E+In+Proceedings+of+the+59th+Annual)
5. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading Wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870– 1879, Vancouver, Canada. Association for Computational Li…
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
7. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. [T-REx: A large scale alignment of natural language with knowledge base triples](https://md.research.elamur.ai/free/en/papers/elsahar-2018-t-rex-large-scale-alignment/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluati…
8. Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020. [Entities as experts: Sparse memory access with entity supervision](https://md.research.elamur.ai/free/en/papers/fevry-2020-entities-as-experts-sparse-memory/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4937–4951, Online. Associatio…
9. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-augmented language model pre-training. In International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Kenton+Lee%2C+Zora+Tung%2C+Panupong+Pasupat%2C+and+Ming%2DWei+Chang%2E+2020%2E+REALM%3A+Retrieval%2Daugmented+language+model+pre%2Dtraining%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2E)
10. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver…
11. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769– 6781, Onl…
12. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. [Measuring bias in contextualized word representations](https://md.research.elamur.ai/free/en/papers/kurita-2019-measuring-bias-contextualized-word-representations/). In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics.
13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
14. Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021. [Learning dense representations of phrases at scale](https://md.research.elamur.ai/free/en/papers/lee-2021-learning-dense-representations-phrases-scale/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Paper…
15. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. [Latent retrieval for weakly supervised open domain question answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
16. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021a. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000–1008, Online. Assoc… [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2021a%2E+Question+and+answer+test%2Dtrain+overlap+in+open%2Ddomain+question+answering+datasets%2E+In+Proceedings+of+the+16th+Conference+of+the+European+C)
17. Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021b. [PAQ: 65 million probably-asked questions and what you can do with them](https://md.research.elamur.ai/free/en/papers/lewis-2021-paq-65-million-probably-asked/). ArXiv preprint, abs/2102.07033.
18. Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020. Efficient one-pass end-to-end entity linking for questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6433–6441, Online. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Belinda+Z%2E+Li%2C+Sewon+Min%2C+Srinivasan+Iyer%2C+Yashar+Mehdad%2C+and+Wen%2Dtau+Yih%2E+2020%2E+Efficient+one%2Dpass+end%2Dto%2Dend+entity+linking+for+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+)
19. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations. ArXiv preprint, abs/2102.10073. — [arXiv:2102.10073](https://arxiv.org/abs/2102.10073) [find](https://scholar.google.com/scholar?q=Jimmy+Lin%2C+Xueguang+Ma%2C+Sheng%2DChieh+Lin%2C+Jheng%2DHong+Yang%2C+Ronak+Pradeep%2C+and+Rodrigo+Nogueira%2E+2021%2E+Pyserini%3A+An+easy%2Dto%2Duse+python+toolkit+to+support+replicable+ir+research+with+sparse+and+dense+rep)
20. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Moin+Nadeem%2C+Anna+Bethke%2C+and+Siva+Reddy%2E+2021%2E+StereoSet%3A+Measuring+stereotypical+bias+in+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+L)
21. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. [RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/). In Proceedings of the 2021 Conference of the North American Chapter of the Ass…
22. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. [find](https://scholar.google.com/scholar?q=Stephen+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+BM25+and+beyond%2E+Foundations+and+Trends+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2E)
23. Yi Chern Tan and L. Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada… [find](https://scholar.google.com/scholar?q=Yi+Chern+Tan+and+L%2E+Elisa+Celis%2E+2019%2E+Assessing+social+and+intersectional+biases+in+contextualized+word+representations%2E+In+Advances+in+Neural+Information+Processing+Systems+32%3A+Annual+Conference+on+)
24. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. [BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models](https://md.research.elamur.ai/free/en/papers/thakur-2021-beir-heterogenous-benchmark-zero/). ArXiv preprint, abs/2104.08663.
25. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11). [find](https://scholar.google.com/scholar?q=Laurens+Van+der+Maaten+and+Geoffrey+Hinton%2E+2008%2E+Visualizing+data+using+t%2Dsne%2E+Journal+of+machine+learning+research%2C+9%2811%29%2E)
26. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85. [find](https://scholar.google.com/scholar?q=Denny+Vrandec%CC%8Cic%CC%81+and+Markus+Kr%C3%B6tzsch%2E+2014%2E+Wikidata%3A+a+free+collaborative+knowledgebase%2E+Communications+of+the+ACM%2C+57%2810%29%3A78%E2%80%9385%2E)
27. Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. [Zero-shot entity linking with dense entity retrieval](https://md.research.elamur.ai/free/en/papers/wu-2020-scalable-zero-shot-entity-linking/). In Empirical Methods in Natural Language Processing (EMNLP).
28. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. [Approximate nearest neighbor negative contrastive learning for dense text retrieval](https://md.research.elamur.ai/free/en/papers/xiong-2020-approximate-nearest-neighbor-negative-contrastive/). In International Conference on Learning Representations (ICLR).
29. Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442–6454, Online. Associat… [find](https://scholar.google.com/scholar?q=Ikuya+Yamada%2C+Akari+Asai%2C+Hiroyuki+Shindo%2C+Hideaki+Takeda%2C+and+Yuji+Matsumoto%2E+2020%2E+LUKE%3A+Deep+contextualized+entity+representations+with+entity%2Daware+self%2Dattention%2E+In+Proceedings+of+the+2020+Confe)
30. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. [End-to-end open-domain question answering with BERTserini](https://md.research.elamur.ai/free/en/papers/yang-2019-end-end-open-domain-question/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 72…
31. Relation DPR DPR BM25 (NQ) (multi) P36 What is the capital of [E]? 77.3 P407 Which language was [E] written in? 77.1 P26 Who is [E] married to? 35.6 P159 Where is the headquarter of [E]? 70.0 P276 Where is [E] located? 74.9 P40 Who is [E]’s child? 19.2 P176 Which company is [E] produced by? 61.7 P20… [find](https://scholar.google.com/scholar?q=Relation+DPR+DPR+BM25+%28NQ%29+%28multi%29+P36+What+is+the+capital+of+%5BE%5D%3F+77%2E3+P407+Which+language+was+%5BE%5D+written+in%3F+77%2E1+P26+Who+is+%5BE%5D+married+to%3F+35%2E6+P159+Where+is+the+headquarter+of+%5BE%5D%3F+70%2E0+P276+Whe)

*Updated 24 October 2026*