---
title: "Compositional Questions Do Not Necessitate — Min 2019"
description: "The paper asks whether questions that appear compositional actually force a reading comprehension system to combine evidence from several paragraphs."
authors: ["Sewon Min", "Eric Wallace", "Sameer Singh", "Matt Gardner", "Hannaneh Hajishirzi", "Luke Zettlemoyer"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1416/"
doi: "10.18653/v1/p19-1416"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/min-2019-compositional-questions-do-not-necessitate/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/min-2019-compositional-questions-do-not-necessitate/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / Compositional Questions Do Not Necessitate Multi-hop Reasoning

# Compositional Questions Do Not Necessitate Multi-hop Reasoning

**Authors:** [Sewon Min](https://md.research.elamur.ai/free/en/authors/sewon-min/), [Eric Wallace](https://md.research.elamur.ai/free/en/authors/eric-wallace/), [Sameer Singh](https://md.research.elamur.ai/free/en/authors/sameer-singh/), [Matt Gardner](https://md.research.elamur.ai/free/en/authors/matt-gardner/), [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1416/) · [PDF](https://aclanthology.org/P19-1416.pdf) · [DOI 10.18653/v1/p19-1416](https://doi.org/10.18653/v1/p19-1416) · [OpenAlex](https://openalex.org/W2953163841)

**Citations:** 136 citations (openalex-snapshot · 2026-10-14) · [95](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks whether questions that appear compositional actually force a reading comprehension system to combine evidence from several paragraphs. Focusing on HotpotQA, the authors argue that whether reasoning is truly multi-hop depends on both the question and the evidence supplied with it. They build single-paragraph BERT, a model that reads each candidate paragraph separately, so it cannot link facts across them. It then returns the answer from the paragraph it judges most likely to contain one. In the distractor setting, this model performs close to dedicated multi-hop systems. A human study and a manual categorization of bridge questions point to the main causes: distractors that are easy to rule out by entity type, and questions whose extra conditions are redundant. The authors also try harder distractors, chosen adversarially and filtered by type. These lower accuracy at first, but retraining recovers most of the loss. The study implies that future multi-hop datasets must select evidence carefully, either through stronger distractors or through better retrieval in open-domain settings.

## Key points
- Single-paragraph BERT scores 67.08 F1 on the HotpotQA distractor validation set, near multi-hop leaderboard models. It reaches only 38.06 F1 in the open-domain (Fullwiki) setting with bigram TF-IDF retrieval.
- On 200 bridge questions, NLP graduate students scored 87.37 F1 with all ten paragraphs. With one gold paragraph withheld, they still scored 82.06 F1.
- Of 100 manually analyzed bridge questions, only 27% truly required multi-hop reasoning. Another 35% were solvable because of weak distractors, and 8% were non-compositional single-hop questions. For comparison questions, the model performs near chance on the multi-hop types.
- Adversarially selected distractors reduced F1 from 67.08 to 46.84, and type-filtered distractors reduced it to 40.73. After retraining, the model recovered to 60.10 and 58.42 F1, respectively.
- With 500 TF-IDF-retrieved paragraphs, F1 is 39.12. It rises to 53.12 when the two gold paragraphs are added, which shows that retrieval failures dominate the open-domain errors.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

95 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 84 not yet.

- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://md.research.elamur.ai/free/en/papers/asai-2019-learning-retrieve-reasoning-paths-over/)
- [Evaluating Models' Local Decision Boundaries via Contrast Sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-pretrained-transformers-improve-out-distribution/)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/)
- [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://md.research.elamur.ai/free/en/papers/jiang-2020-hover-dataset-many-hop-fact/)
- [Robustifying Multi-hop QA through Pseudo-Evidentiality Training](https://md.research.elamur.ai/free/en/papers/lee-2021-robustifying-multi-hop-qa-through/)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://md.research.elamur.ai/free/en/papers/levy-2024-same-task-more-tokens-impact/)
- [A Discrete Hard EM Approach for Weakly Supervised Question Answering](https://md.research.elamur.ai/free/en/papers/min-2019-discrete-hard-em-approach-weakly/)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://md.research.elamur.ai/free/en/papers/pang-2022-quality-question-answering-long-input/)
- [Unsupervised Question Decomposition for Question Answering](https://md.research.elamur.ai/free/en/papers/perez-2020-unsupervised-question-decomposition-question-answering/)
- [Break It Down: A Question Understanding Benchmark](https://md.research.elamur.ai/free/en/papers/wolfson-2020-break-it-down-question-understanding/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{min2019compositional,
  title = {{Compositional Questions Do Not Necessitate Multi-hop Reasoning}},
  author = {Min, Sewon and Wallace, Eric and Singh, Sameer and Gardner, Matt and Hajishirzi, Hannaneh and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1416},
  url = {https://doi.org/10.18653/v1/p19-1416}
}
```

- [← previous Comparing Sense Categorization Between English PropBank and English WordNet](https://md.research.elamur.ai/free/en/papers/bakay-2019-comparing-sense-categorization-between-english/)
- [next → Compressive Transformers for Long-Range Sequence Modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/)

## References

18 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In EMNLP. [find](https://scholar.google.com/scholar?q=Jonathan+Berant%2C+Andrew+Chou%2C+Roy+Frostig%2C+and+Percy+Liang%2E+2013%2E+Semantic+parsing+on+freebase+from+question%2Danswer+pairs%2E+In+EMNLP%2E)
2. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Adam+Fisch%2C+Jason+Weston%2C+and+Antoine+Bordes%2E+2017%2E+Reading+wikipedia+to+answer+open%2Ddomain+questions%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/)
3. Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In NAACL. [find](https://scholar.google.com/scholar?q=Jifan+Chen+and+Greg+Durrett%2E+2019%2E+Understanding+dataset+design+choices+for+multi%2Dhop+reasoning%2E+In+NAACL%2E)
4. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Christopher+Clark+and+Matt+Gardner%2E+2018%2E+Simple+and+effective+multi%2Dparagraph+reading+comprehension%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2E)
6. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
7. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
8. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR%2E)
9. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. TACL. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Koc%CC%8Cisky%CC%80%2C+Jonathan+Schwarz%2C+Phil+Blunsom%2C+Chris+Dyer%2C+Karl+Moritz+Hermann%2C+Ga%CC%81bor+Melis%2C+and+Edward+Grefenstette%2E+2018%2E+The+NarrativeQA+reading+comprehension+challenge%2E+TACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)
10. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+PyTorch%2E+In+NIPS+Aut)
11. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
12. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR. [find](https://scholar.google.com/scholar?q=Minjoon+Seo%2C+Aniruddha+Kembhavi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2017%2E+Bidirectional+attention+flow+for+machine+comprehension%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)
13. Saku Sugawara, Kentaro fInui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In EMNLP. [find](https://scholar.google.com/scholar?q=Saku+Sugawara%2C+Kentaro+fInui%2C+Satoshi+Sekine%2C+and+Akiko+Aizawa%2E+2018%2E+What+makes+reading+comprehension+questions+easier%3F+In+EMNLP%2E)
14. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In NAACL. [find](https://scholar.google.com/scholar?q=Alon+Talmor+and+Jonathan+Berant%2E+2018%2E+The+web+as+a+knowledge%2Dbase+for+answering+complex+questions%2E+In+NAACL%2E)
15. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2017. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. In TACL. [find](https://scholar.google.com/scholar?q=Johannes+Welbl%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2017%2E+Constructing+Datasets+for+Multi%2Dhop+Reading+Comprehension+Across+Documents%2E+In+TACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/)
16. Caiming Xiong, Victor Zhong, and Richard Socher. 2018. DCN+: Mixed objective and deep residual coattention for question answering. In ICLR. [find](https://scholar.google.com/scholar?q=Caiming+Xiong%2C+Victor+Zhong%2C+and+Richard+Socher%2E+2018%2E+DCN%2B%3A+Mixed+objective+and+deep+residual+coattention+for+question+answering%2E+In+ICLR%2E)
17. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+W+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D+Manning%2E+2018%2E+HotpotQA%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
18. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. In ICLR. [find](https://scholar.google.com/scholar?q=Adams+Wei+Yu%2C+David+Dohan%2C+Minh%2DThang+Luong%2C+Rui+Zhao%2C+Kai+Chen%2C+Mohammad+Norouzi%2C+and+Quoc+V%2E+Le%2E+2018%2E+Qanet%3A+Combining+local+convolution+with+global+self%2Dattention+for+reading+comprehension%2E+In+ICL) [our card](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/)