---
title: "Revealing the Importance of Semantic Retrieval for Machine"
description: "They build a transparent hierarchical pipeline over Wikipedia with four stages: term-based filtering… 18 references, cited by 62 papers here."
authors: ["Yixin Nie", "Songhe Wang", "Mohit Bansal"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1258/"
doi: "10.18653/v1/d19-1258"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nie-2019-revealing-importance-semantic-retrieval-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nie-2019-revealing-importance-semantic-retrieval-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Revealing the Importance of Semantic Retrieval for Machine Reading at Scale

# Revealing the Importance of Semantic Retrieval for Machine Reading at Scale

**Authors:** [Yixin Nie](https://md.research.elamur.ai/free/en/authors/yixin-nie/), Songhe Wang and [Mohit Bansal](https://md.research.elamur.ai/free/en/authors/mohit-bansal/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1258/) · [PDF](https://aclanthology.org/D19-1258.pdf) · [DOI 10.18653/v1/d19-1258](https://doi.org/10.18653/v1/d19-1258) · [OpenAlex](https://openalex.org/W2970139579)

**Citations:** 127 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/nie-2019-revealing-importance-semantic-retrieval-machine/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Machine Reading at Scale (MRS) systems must find relevant evidence in a huge text collection and then reason over it. Most prior work judged them only by final-task accuracy and treated retrieval mainly as a way to maximize coverage. The authors argue that intermediate retrieval quality deserves equal attention. They build a transparent hierarchical pipeline over Wikipedia with four stages: term-based filtering (TF-IDF plus keyword rules), then neural paragraph ranking, then neural sentence ranking, and finally a task model for either QA or NLI-style verification. Every neural component is a BERT-Base model. Each retriever is trained as a binary classifier, with negative examples drawn from the stage just before it.

The pipeline set new leaderboard records on both HotpotQA and FEVER. Ablations show that paragraph-level and sentence-level retrieval are each necessary. Good retrieval does more than raise recall: it saves computation further down the pipeline and changes what data the downstream models see during both training and inference. This offers practical design guidance for open-domain QA and fact-checking.

## Key points
- On HotpotQA the system reached 45.32% answer EM and 25.14% joint EM. That is about 8 points above the previous best on answer EM, and roughly twice the earlier joint EM. On FEVER it scored 67.26%, about 3 points above earlier published systems.
- For HotpotQA yes/no questions, the QA model has two extra tokens ("yes", "no") inserted after [CLS]. These compete directly with extracted answer spans.
- Taking more paragraphs beyond about two (k_p) hurt answer and joint EM on HotpotQA. The downstream QA model is sensitive to paragraph-level noise.
- At the sentence level, the best threshold for answer EM and FEVER label accuracy was h_s = 0.2, while the best for retrieval metrics was h_s = 0.5. Downstream models therefore tolerate some sentence noise in exchange for higher recall.
- A manual analysis of 200 HotpotQA examples found 49.0% overall accuracy, with the strongest results on Yes/No questions (70.6%). Code is released at github.com/easonnie/semanticRetrievalMRS.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of North Carolina at Chapel Hill](https://md.research.elamur.ai/free/en/universities/university-of-north-carolina-at-chapel-hill/) — University, Chapel Hill, United States · 37 works · 14,332 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (1): [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 145 works

## Cite

Cite as: Yixin Nie, Songhe Wang and Mohit Bansal. 2019. Revealing the Importance of Semantic Retrieval for Machine Reading at Scale. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nie2019revealing,
  title = {{Revealing the Importance of Semantic Retrieval for Machine Reading at Scale}},
  author = {Yixin Nie, Songhe Wang and Mohit Bansal},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1258},
  url = {https://doi.org/10.18653/v1/d19-1258}
}
```

- [← previous Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation](https://md.research.elamur.ai/free/en/papers/loureiro-2019-language-modelling-makes-sense-propagating/)
- [next → Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts](https://md.research.elamur.ai/free/en/papers/breitfeller-2019-finding-microaggressions-wild-case-locating/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

18 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Conference+on+Empirical+Methods+in+Natural+Lan)
2. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading Wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Association for Computational Linguistics (ACL).
3. Christopher Clark and Matt Gardner. 2017. [Simple and effective multi-paragraph reading comprehension](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/). Association for Computational Linguistics (ACL).
4. Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W Bruce Croft. 2017. Neural ranking models with weak supervision. In SIGIR. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Hamed+Zamani%2C+Aliaksei+Severyn%2C+Jaap+Kamps%2C+and+W+Bruce+Croft%2E+2017%2E+Neural+ranking+models+with+weak+supervision%2E+In+SIGIR%2E)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
6. Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In CIKM. [find](https://scholar.google.com/scholar?q=Jiafeng+Guo%2C+Yixing+Fan%2C+Qingyao+Ai%2C+and+W+Bruce+Croft%2E+2016%2E+A+deep+relevance+matching+model+for+ad%2Dhoc+retrieval%2E+In+CIKM%2E)
7. Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. [Ukp-athene: Multi-sentence textual entailment for claim verification](https://md.research.elamur.ai/free/en/papers/hanselowski-2018-ukp-athene-multi-sentence-textual/). In The 1st Workshop on Fact Extraction and Verification.
8. Phu Mon Htut, Samuel R Bowman, and Kyunghyun Cho. 2018. Training a ranking function for open-domain question answering. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Phu+Mon+Htut%2C+Samuel+R+Bowman%2C+and+Kyunghyun+Cho%2E+2018%2E+Training+a+ranking+function+for+open%2Ddomain+question+answering%2E+In+NAACL%2DHLT%2E)
9. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In CIKM. [find](https://scholar.google.com/scholar?q=Po%2DSen+Huang%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+Li+Deng%2C+Alex+Acero%2C+and+Larry+Heck%2E+2013%2E+Learning+deep+structured+semantic+models+for+web+search+using+clickthrough+data%2E+In+CIKM%2E)
10. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Sergey+Ioffe+and+Christian+Szegedy%2E+2015%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+In+International+Conference+on+International+Conference+on+Machi)
11. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Association for Computationa…
12. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. [Constructing datasets for multi-hop reading comprehension across documents](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/). Transactions of the Association of Computational Linguistics, 6:287–302.
13. Jinhyuk Lee, Seongjun Yun, Hyunjae Kim, Miyoung Ko, and Jaewoo Kang. 2018. Ranking paragraphs for improving answer recall in open-domain question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Jinhyuk+Lee%2C+Seongjun+Yun%2C+Hyunjae+Kim%2C+Miyoung+Ko%2C+and+Jaewoo+Kang%2E+2018%2E+Ranking+paragraphs+for+improving+answer+recall+in+open%2Ddomain+question+answering%2E+In+Conference+on+Empirical+Methods+in+Natur)
14. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Conference on Empirical Methods in Natural Language Processing (EMNLP).
15. Bill MacCartney and Christopher D Manning. 2009. Natural language inference. Citeseer. [find](https://scholar.google.com/scholar?q=Bill+MacCartney+and+Christopher+D+Manning%2E+2009%2E+Natural+language+inference%2E+Citeseer%2E)
16. Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Ucl machine reading group: Four factor framework for fact finding (hexaf). In The 1st Workshop on Fact Extraction and Verification. [find](https://scholar.google.com/scholar?q=Takuma+Yoneda%2C+Jeff+Mitchell%2C+Johannes+Welbl%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2018%2E+Ucl+machine+reading+group%3A+Four+factor+framework+for+fact+finding+%28hexaf%29%2E+In+The+1st+Workshop+on+Fact+Extra)
17. Qibin Chen Hongxia Yang Jie Tang Ming Ding, Chang Zhou. 2019. Cognitive graph for multi-hop reading comprehension at scale. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Qibin+Chen+Hongxia+Yang+Jie+Tang+Ming+Ding%2C+Chang+Zhou%2E+2019%2E+Cognitive+graph+for+multi%2Dhop+reading+comprehension+at+scale%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E)
18. Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017. Learning to match using local and distributed representations of text for web search. In WWW. [find](https://scholar.google.com/scholar?q=Bhaskar+Mitra%2C+Fernando+Diaz%2C+and+Nick+Craswell%2E+2017%2E+Learning+to+match+using+local+and+distributed+representations+of+text+for+web+search%2E+In+WWW%2E)

*Updated 24 October 2026*