--- title: "♫ MuSiQue: Multihop Questions via Single-hop — Trivedi 2022" description: "Existing multihop QA benchmarks such as HotpotQA can often be solved through shortcuts." authors: ["Harsh Trivedi", "Niranjan Balasubramanian", "Tushar Khot", "Ashish Sabharwal"] year: 2022 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2022.tacl-1.31/" doi: "10.1162/tacl_a_00475" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/trivedi-2022-musique-multihop-questions-single-hop/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/trivedi-2022-musique-multihop-questions-single-hop/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / ♫ MuSiQue: Multihop Questions via Single-hop Question Composition # ♫ MuSiQue: Multihop Questions via Single-hop Question Composition **Authors:** [Harsh Trivedi](https://md.research.elamur.ai/free/en/authors/harsh-trivedi/), [Niranjan Balasubramanian](https://md.research.elamur.ai/free/en/authors/niranjan-balasubramanian/), [Tushar Khot](https://md.research.elamur.ai/free/en/authors/tushar-khot/), [Ashish Sabharwal](https://md.research.elamur.ai/free/en/authors/ashish-sabharwal/) · **Year:** 2022 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.tacl-1.31/) · [PDF](https://aclanthology.org/2022.tacl-1.31.pdf) · [DOI 10.1162/tacl\_a\_00475](https://doi.org/10.1162/tacl_a_00475) · [OpenAlex](https://openalex.org/W3190126809) **Citations:** 267 citations (openalex-snapshot · 2026-10-14) · [117](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Existing multihop QA benchmarks such as HotpotQA can often be solved through shortcuts. A model can reach the right answer without linking evidence across the reasoning steps, which the paper calls disconnected reasoning. Trivedi and colleagues build a dataset that rules this out by design. Instead of having crowdworkers write multihop questions directly, they assemble them from the bottom up. Single-hop questions taken from existing Wikipedia-based datasets are chained together, and a pair is kept only if the later question cannot be answered without the earlier question's answer. The pipeline also limits train–test overlap at the level of each single-hop question and adds hard distractor paragraphs. The result is MuSiQue-Ans, with about 25K questions of 2–4 hops. A second version, MuSiQue-Full, pairs each answerable question with an unanswerable counterpart. Compared with HotpotQA and 2WikiMultihopQA, both versions are harder and much less open to shortcut exploitation. The work gives the field a more reliable test of genuine multi-step reasoning, and its gold decompositions are useful for building interpretable models. ## Key points - MuSiQue-Ans has about 25K questions with 2–4 hops, arranged in six reasoning-graph shapes. MuSiQue-Full has about 50K questions forming answerable/unanswerable contrast pairs, and the unanswerable versions are made by removing the paragraphs that contain the bridge entity. - The gap between humans and the best model on answer F1 is about 27 points, compared with roughly 10 for HotpotQA and 5 for 2WikiMultihopQA, which is about a 3× increase. On supporting-paragraph identification a 14-point gap remains, while models already exceed humans on the other two datasets. - A single-hop model loses about 30 F1 points compared with its performance on earlier datasets, which shows the dataset is much harder to cheat via disconnected reasoning. - Ablations show that each pipeline step matters: - **Disconnection filter:** without it, both multihop and shortcut-based models score much higher. - **Leakage-aware split:** with a random split, a single-paragraph model reaches 85.1 answer F1. - **Distractors:** paragraphs taken from the supporting contexts of other questions in the pool ("positive distractors"), with 20 per question, are harder than distractors retrieved from full Wikipedia. - Training with gold decompositions improves answer F1 by about 9 points and support F1 by about 7 points, but performance still falls short of human level. Code and data are at github.com/stonybrooknlp/musique. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 117 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 106 not yet. - [Retrieval-Augmented Generation for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/) - [HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models](https://md.research.elamur.ai/free/en/papers/gutierrez-2024-hipporag-neurobiologically-inspired-long-term/) - [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) - [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/jin-2025-search-r1-training-llms-reason/) - [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://md.research.elamur.ai/free/en/papers/li-2023-loogle-long-context-language-models/) - [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/song-2025-r1-searcher-incentivizing-search/) - [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/) - [MEUR: A Benchmark for Evaluating Vision-Language Models on Multimodal Event Understanding and Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2026-meur-benchmark-evaluating-vision-language/) - [Retrieval meets Long Context Large Language Models](https://md.research.elamur.ai/free/en/papers/xu-2023-retrieval-meets-long-context-large/) - [Reasoning Graph-Structured Question Answering: Datasets and Insights from LLM Benchmarking](https://md.research.elamur.ai/free/en/papers/yone-2026-reasoning-graph-structured-question-answering/) - [CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps](https://md.research.elamur.ai/free/en/papers/yun-2026-crit-qa-evaluating-multi-hop/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{trivedi2022musique, title = {{♫ MuSiQue: Multihop Questions via Single-hop Question Composition}}, author = {Trivedi, Harsh and Balasubramanian, Niranjan and Khot, Tushar and Sabharwal, Ashish}, journal = {Transactions of the Association for Computational Linguistics}, year = {2022}, doi = {10.1162/tacl\_a\_00475}, url = {https://doi.org/10.1162/tacl_a_00475} } ``` - [← previous “I'm sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://md.research.elamur.ai/free/en/papers/smith-2022-i-m-sorry-hear-that/) - [next → 深度知识追踪模型综述和性能比较](https://md.research.elamur.ai/free/en/papers/wang-2022-performance-comparison-deep/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Matthew+E%2E+Peters%2C+and+Arman+Cohan%2E+2020%2E+Longformer%3A+The+long%2Ddocument+transformer%2E+arXiv%3A2004%2E05150%2E) 2. Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In NAACL-HLT. https://doi.org/10.18653/v1/N19-1405 [↗](https://doi.org/10.18653/v1/N19-1405)