---
title: "CRiT-QA: Evaluating Multi-hop Reasoning with — Yun 2026"
description: "Multi-hop QA benchmarks are meant to test whether LLMs can combine evidence spread across several documents."
authors: ["Jungmin Yun", "June Hyoung Kwon", "Youngbin Kim"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.410/"
doi: "10.63317/2jvxj7kwecuo"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yun-2026-crit-qa-evaluating-multi-hop/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yun-2026-crit-qa-evaluating-multi-hop/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps

# CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps

**Authors:** Jungmin Yun, June Hyoung Kwon, Youngbin Kim · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.410/) · [PDF](https://aclanthology.org/2026.lrec-1.410.pdf) · [DOI 10.63317/2jvxj7kwecuo](https://doi.org/10.63317/2jvxj7kwecuo) · [OpenAlex](https://openalex.org/W7160014640)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
Multi-hop QA benchmarks are meant to test whether LLMs can combine evidence spread across several documents. High scores can mislead, though. Models may answer from facts memorized during pretraining and ignore the supplied passages, or they may use shortcuts such as picking the only entity of the expected type or relying on a single paragraph.

CRiT-QA (Counterfactual Reasoning with Traps) is a diagnostic dataset built to close both loopholes. It starts from MuSiQue, whose questions come with explicit sub-question decompositions. An automated LLM-based pipeline keeps the original questions but replaces the whole factual reasoning chain and its evidence with counterfactual entities, so the answer can only be derived from the given context. It then adds multi-anchor distractor chains: believable but wrong paths that branch off at different hops, kept type-consistent through named entity recognition.

Both proprietary and open-source LLMs lose considerable accuracy on CRiT-QA compared with standard benchmarks, and the loss grows with chain length. The resource helps separate genuine evidence aggregation from recall and heuristics, which matters for building trustworthy RAG systems.

## Key points
- The source data is MuSiQue (Trivedi et al., 2022). Its step-by-step decompositions make it possible to rewrite every hop of a chain, and only gold evidence paragraphs are retained.
- There are two complementary perturbations:
  - counterfactual replacement of the entire reasoning chain, which defeats parametric memory and data contamination;
  - NER-checked distractor chains diverging at various hops, which defeat type-matching and single-paragraph shortcuts.
- Strong proprietary models (GPT-4o, Gemini-2.5-Pro) drop markedly when moving from 2-hop to 4-hop questions.
- Open-source LLaMA-3-8B and Qwen2.5-7B lose more than half of their accuracy.
- Unlike CofCA, which generates new questions from rewritten documents, CRiT-QA keeps the original multi-hop questions. It also combines counterfactuals and distractors in one framework, allowing difficulty to be raised step by step (LREC 2026, pp. 5246–5255).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yun2026crit,
  title = {{CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps}},
  author = {Yun, Jungmin and Kwon, June Hyoung and Kim, Youngbin},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2jvxj7kwecuo},
  url = {https://doi.org/10.63317/2jvxj7kwecuo}
}
```

- [← previous Creating a Hybrid Rule and Neural Network Based Semantic Tagger Using Silver Standard Data: The PyMUSAS Framework for Multilingual Semantic Annotation](https://md.research.elamur.ai/free/en/papers/moore-2026-creating-hybrid-rule-neural-network/)
- [next → Cross-Corpus CEFR Classification through Artificial Learners Perplexities](https://md.research.elamur.ai/free/en/papers/stearns-2026-cross-corpus-cefr-classification-through/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Neeladri Bhuiya, Viktor Schlegel, and Stefan Winkler. 2024. Seemingly plausible distractors in multi-hop reasoning: Are large language models attentive readers? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2514–2528. [find](https://scholar.google.com/scholar?q=Neeladri+Bhuiya%2C+Viktor+Schlegel%2C+and+Stefan+Winkler%2E+2024%2E+Seemingly+plausible+distractors+in+multi%2Dhop+reasoning%3A+Are+large+language+models+attentive+readers%3F+In+Proceedings+of+the+2024+Conference+o)
2. Baolong Bi, Shenghua Liu, Yiwei Wang, Yilong Xu, Junfeng Fang, Lingrui Mei, and Xueqi Cheng. 2025. Parameters vs. context: Fine-grained control of knowledge reliance in language models. arXiv preprint arXiv:2503.15888. — [arXiv:2503.15888](https://arxiv.org/abs/2503.15888) [find](https://scholar.google.com/scholar?q=Baolong+Bi%2C+Shenghua+Liu%2C+Yiwei+Wang%2C+Yilong+Xu%2C+Junfeng+Fang%2C+Lingrui+Mei%2C+and+Xueqi+Cheng%2E+2025%2E+Parameters+vs%2E+context%3A+Fine%2Dgrained+control+of+knowledge+reliance+in+language+models%2E+arXiv+preprint)
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
4. Yuefei Chen, Vivek K. Singh, Jing Ma, and Ruxiang Tang. 2025. Counterbench: A benchmark for counterfactuals reasoning in large language models. [find](https://scholar.google.com/scholar?q=Yuefei+Chen%2C+Vivek+K%2E+Singh%2C+Jing+Ma%2C+and+Ruxiang+Tang%2E+2025%2E+Counterbench%3A+A+benchmark+for+counterfactuals+reasoning+in+large+language+models%2E)
5. Sitao Cheng, Liangming Pan, Xunjian Yin, Xinyi Wang, and William Yang Wang. 2024. Understanding the interplay between parametric and contextual knowledge for large language models. arXiv preprint arXiv:2410.08414. — [arXiv:2410.08414](https://arxiv.org/abs/2410.08414) [find](https://scholar.google.com/scholar?q=Sitao+Cheng%2C+Liangming+Pan%2C+Xunjian+Yin%2C+Xinyi+Wang%2C+and+William+Yang+Wang%2E+2024%2E+Understanding+the+interplay+between+parametric+and+contextual+knowledge+for+large+language+models%2E+arXiv+preprint+arXi)
6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep… [find](https://scholar.google.com/scholar?q=Gheorghe+Comanici%2C+Eric+Bieber%2C+Mike+Schaekermann%2C+Ice+Pasupat%2C+Noveen+Sachdeva%2C+Inderjit+Dhillon%2C+Marcel+Blistein%2C+Ori+Ram%2C+Dan+Zhang%2C+Evan+Rosen%2C+et+al%2E+2025%2E+Gemini+2%2E5%3A+Pushing+the+frontier+with+a) [our card](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/)
7. Jiayu Ding, Siyuan Wang, Qin Chen, and Zhongyu Wei. 2021. Reasoning chain based adversarial attack for multi-hop question answering. arXiv preprint arXiv:2112.09658. — [arXiv:2112.09658](https://arxiv.org/abs/2112.09658) [find](https://scholar.google.com/scholar?q=Jiayu+Ding%2C+Siyuan+Wang%2C+Qin+Chen%2C+and+Zhongyu+Wei%2E+2021%2E+Reasoning+chain+based+adversarial+attack+for+multi%2Dhop+question+answering%2E+arXiv+preprint+arXiv%3A2112%2E09658%2E+%E2%80%94+%5BarXiv%3A2112%2E09658%5D%28https%3A%2F%2Farxiv)
8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv– 2407. [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Amy+Yang%2C+Angela+Fan%2C+et+al%2E+2024%2E+The+llama+3+herd+of+models%2E+arXiv+e%2Dpri)
9. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6491–6501. [find](https://scholar.google.com/scholar?q=Wenqi+Fan%2C+Yujuan+Ding%2C+Liangbo+Ning%2C+Shijie+Wang%2C+Hengyun+Li%2C+Dawei+Yin%2C+Tat%2DSeng+Chua%2C+and+Qing+Li%2E+2024%2E+A+survey+on+rag+meeting+llms%3A+Towards+retrieval%2Daugmented+large+language+models%2E+In+Proceedi) [our card](https://md.research.elamur.ai/free/en/papers/fan-2024-survey-rag-meeting-llms-towards/)
10. Jinyuan Fang, Zaiqiao Meng, and Craig MacDonald. 2024. TRACE the evidence: Constructing knowledge-grounded reasoning chains for retrieval-augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8472–8494, Miami, Florida, USA. Association for Computationa… [find](https://scholar.google.com/scholar?q=Jinyuan+Fang%2C+Zaiqiao+Meng%2C+and+Craig+MacDonald%2E+2024%2E+TRACE+the+evidence%3A+Constructing+knowledge%2Dgrounded+reasoning+chains+for+retrieval%2Daugmented+generation%2E+In+Findings+of+the+Association+for+Compu)
11. Jörg Frohberg and Frank Binder. 2022. CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Frohberg+and+Frank+Binder%2E+2022%2E+CRASS%3A+A+novel+data+set+and+benchmark+to+test+counterfactual+reasoning+of+large+language+models%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+and+Evaluatio)
12. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. [find](https://scholar.google.com/scholar?q=Yunfan+Gao%2C+Yun+Xiong%2C+Xinyu+Gao%2C+Kangxiang+Jia%2C+Jinliu+Pan%2C+Yuxi+Bi%2C+Yi+Dai%2C+Jiawei+Sun%2C+Meng+Wang%2C+and+Haofen+Wang%2E+2024%2E+Retrieval%2Daugmented+generation+for+large+language+models%3A+A+survey%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/)
13. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+2021%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+)
14. Wangzhen Guo, Qinkang Gong, Yanghui Rao, and Hanjiang Lai. 2023. Counterfactual multihop QA: A cause-effect approach for reducing disconnected reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4214–4226, Toronto, Cana… [find](https://scholar.google.com/scholar?q=Wangzhen+Guo%2C+Qinkang+Gong%2C+Yanghui+Rao%2C+and+Hanjiang+Lai%2E+2023%2E+Counterfactual+multihop+QA%3A+A+cause%2Deffect+approach+for+reducing+disconnected+reasoning%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+t)
15. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm… [find](https://scholar.google.com/scholar?q=Xanh+Ho%2C+Anh%2DKhoa+Duong+Nguyen%2C+Saku+Sugawara%2C+and+Akiko+Aizawa%2E+2020%2E+Constructing+a+multi%2Dhop+QA+dataset+for+comprehensive+evaluation+of+reasoning+steps%2E+In+Proceedings+of+the+28th+International+Con) [our card](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/)
16. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2023. Analyzing the effectiveness of the underlying reasoning tasks in multi-hop question answering. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1163–1180, Dubrovnik, Croatia. Association for Computat… [find](https://scholar.google.com/scholar?q=Xanh+Ho%2C+Anh%2DKhoa+Duong+Nguyen%2C+Saku+Sugawara%2C+and+Akiko+Aizawa%2E+2023%2E+Analyzing+the+effectiveness+of+the+underlying+reasoning+tasks+in+multi%2Dhop+question+answering%2E+In+Findings+of+the+Association+for)
17. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. [find](https://scholar.google.com/scholar?q=Aaron+Hurst%2C+Adam+Lerer%2C+Adam+P+Goucher%2C+Adam+Perelman%2C+Aditya+Ramesh%2C+Aidan+Clark%2C+AJ+Ostrow%2C+Akila+Welihinda%2C+Alan+Hayes%2C+Alec+Radford%2C+et+al%2E+2024%2E+Gpt%2D4o+system+card%2E+arXiv+preprint+arXiv%3A2410%2E212) [our card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/)
18. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
19. Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo Jose Taylor, and Dan Roth. 2024. A peek into token bias: Large language models are not yet genuine reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4… [find](https://scholar.google.com/scholar?q=Bowen+Jiang%2C+Yangxinyu+Xie%2C+Zhuoqun+Hao%2C+Xiaomeng+Wang%2C+Tanwi+Mallick%2C+Weijie+J+Su%2C+Camillo+Jose+Taylor%2C+and+Dan+Roth%2E+2024%2E+A+peek+into+token+bias%3A+Large+language+models+are+not+yet+genuine+reasoners)
20. Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2726–2736, Florence, Italy. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Yichen+Jiang+and+Mohit+Bansal%2E+2019%2E+Avoiding+reasoning+shortcuts%3A+Adversarial+evaluation%2C+training%2C+and+model+development+for+multi%2Dhop+QA%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associatio)
21. Jeonghoon Kim, Heesoo Jung, Hyeju Jang, and Hogun Park. 2024. Improving multi-hop logical reasoning in knowledge graphs with context-aware query representation learning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15978–15991, Bangkok, Thailand. Association for Comp… [find](https://scholar.google.com/scholar?q=Jeonghoon+Kim%2C+Heesoo+Jung%2C+Hyeju+Jang%2C+and+Hogun+Park%2E+2024%2E+Improving+multi%2Dhop+logical+reasoning+in+knowledge+graphs+with+context%2Daware+query+representation+learning%2E+In+Findings+of+the+Association)
22. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ethan+Perez%2C+Aleksandra+Piktus%2C+Fabio+Petroni%2C+Vladimir+Karpukhin%2C+Naman+Goyal%2C+Heinrich+K%C3%BCttler%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Tim+Rockt%C3%A4schel%2C+et+al%2E+2020%2E+Retrieval%2Daugmented+generation+f)
23. Ruosen Li, Zimu Wang, Son Tran, Lei Xia, and Xinya Du. 2024. Meqa: A benchmark for multi-hop event-centric question answering with explanations. Advances in Neural Information Processing Systems, 37:126835–126862. [find](https://scholar.google.com/scholar?q=Ruosen+Li%2C+Zimu+Wang%2C+Son+Tran%2C+Lei+Xia%2C+and+Xinya+Du%2E+2024%2E+Meqa%3A+A+benchmark+for+multi%2Dhop+event%2Dcentric+question+answering+with+explanations%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A1)
24. Hao Liu, Zhengren Wang, Xi Chen, Zhiyu Li, Feiyu Xiong, Qinhan Yu, and Wentao Zhang. 2025. Hoprag: Multi-hop reasoning for logic-aware retrieval-augmented generation. [find](https://scholar.google.com/scholar?q=Hao+Liu%2C+Zhengren+Wang%2C+Xi+Chen%2C+Zhiyu+Li%2C+Feiyu+Xiong%2C+Qinhan+Yu%2C+and+Wentao+Zhang%2E+2025%2E+Hoprag%3A+Multi%2Dhop+reasoning+for+logic%2Daware+retrieval%2Daugmented+generation%2E)
25. Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2023. DisentQA: Disentangling parametric and contextual knowledge with counterfactual question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Ella+Neeman%2C+Roee+Aharoni%2C+Or+Honovich%2C+Leshem+Choshen%2C+Idan+Szpektor%2C+and+Omri+Abend%2E+2023%2E+DisentQA%3A+Disentangling+parametric+and+contextual+knowledge+with+counterfactual+question+answering%2E+In+Proc)
26. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. LogicBench: Towards systematic evaluation of logical reasoning ability of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computati… [find](https://scholar.google.com/scholar?q=Mihir+Parmar%2C+Nisarg+Patel%2C+Neeraj+Varshney%2C+Mutsumi+Nakamura%2C+Man+Luo%2C+Santosh+Mashetty%2C+Arindam+Mitra%2C+and+Chitta+Baral%2E+2024%2E+LogicBench%3A+Towards+systematic+evaluation+of+logical+reasoning+ability+)
27. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Muru+Zhang%2C+Sewon+Min%2C+Ludwig+Schmidt%2C+Noah+Smith%2C+and+Mike+Lewis%2E+2023%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2E+In+Findings+of+the+Association+for+Computation)
28. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
29. Viktor Schlegel, Marco Valentino, Andre Freitas, Goran Nenadic, and Riza Batista-Navarro. 2020. A framework for evaluation of machine reading comprehension gold standards. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5359–5369, Marseille, France. European Languag… [find](https://scholar.google.com/scholar?q=Viktor+Schlegel%2C+Marco+Valentino%2C+Andre+Freitas%2C+Goran+Nenadic%2C+and+Riza+Batista%2DNavarro%2E+2020%2E+A+framework+for+evaluation+of+machine+reading+comprehension+gold+standards%2E+In+Proceedings+of+the+Twelft)
30. Julian Schnitzler, Xanh Ho, Jiahao Huang, Florian Boudin, Saku Sugawara, and Akiko Aizawa. 2024. Morehopqa: More than multi-hop reasoning. arXiv preprint arXiv:2406.13397. — [arXiv:2406.13397](https://arxiv.org/abs/2406.13397) [find](https://scholar.google.com/scholar?q=Julian+Schnitzler%2C+Xanh+Ho%2C+Jiahao+Huang%2C+Florian+Boudin%2C+Saku+Sugawara%2C+and+Akiko+Aizawa%2E+2024%2E+Morehopqa%3A+More+than+multi%2Dhop+reasoning%2E+arXiv+preprint+arXiv%3A2406%2E13397%2E+%E2%80%94+%5BarXiv%3A2406%2E13397%5D%28https%3A%2F)
31. Sudhanshu Suryawanshi, Shreyas Waghmode, Ritesh Sawant, and Megha Gupta. 2025. A knowledge graph-based rag for cross-document information extraction. In 2025 5th International Conference on Pervasive Computing and Social Networking (ICPCSN), pages 1401–1406. IEEE. [find](https://scholar.google.com/scholar?q=Sudhanshu+Suryawanshi%2C+Shreyas+Waghmode%2C+Ritesh+Sawant%2C+and+Megha+Gupta%2E+2025%2E+A+knowledge+graph%2Dbased+rag+for+cross%2Ddocument+information+extraction%2E+In+2025+5th+International+Conference+on+Pervasive+)
32. Yixuan Tang and Yi Yang. 2024. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. arXiv preprint arXiv:2401.15391. [find](https://scholar.google.com/scholar?q=Yixuan+Tang+and+Yi+Yang%2E+2024%2E+Multihop%2Drag%3A+Benchmarking+retrieval%2Daugmented+generation+for+multi%2Dhop+queries%2E+arXiv+preprint+arXiv%3A2401%2E15391%2E) [our card](https://md.research.elamur.ai/free/en/papers/tang-2024-multihop-rag-benchmarking-retrieval-augmented/)
33. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. [find](https://scholar.google.com/scholar?q=Harsh+Trivedi%2C+Niranjan+Balasubramanian%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2022%2E+MuSiQue%3A+Multihop+questions+via+single%2Dhop+question+composition%2E+Transactions+of+the+Association+for+Computational+Lin) [our card](https://md.research.elamur.ai/free/en/papers/trivedi-2022-musique-multihop-questions-single-hop/)
34. Jian Wu, Linyi Yang, Manabu Okumura, and Yue Zhang. 2024a. Mrke: The multi-hop reasoning evaluation of llms by knowledge edition. arXiv preprint arXiv:2402.11924. — [arXiv:2402.11924](https://arxiv.org/abs/2402.11924) [find](https://scholar.google.com/scholar?q=Jian+Wu%2C+Linyi+Yang%2C+Manabu+Okumura%2C+and+Yue+Zhang%2E+2024a%2E+Mrke%3A+The+multi%2Dhop+reasoning+evaluation+of+llms+by+knowledge+edition%2E+arXiv+preprint+arXiv%3A2402%2E11924%2E+%E2%80%94+%5BarXiv%3A2402%2E11924%5D%28https%3A%2F%2Farxiv%2Eor)
35. Jian Wu, Linyi Yang, Zhen Wang, Manabu Okumura, and Yue Zhang. 2025. Cofca: A step-wise counterfactual multi-hop qa benchmark. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jian+Wu%2C+Linyi+Yang%2C+Zhen+Wang%2C+Manabu+Okumura%2C+and+Yue+Zhang%2E+2025%2E+Cofca%3A+A+step%2Dwise+counterfactual+multi%2Dhop+qa+benchmark%2E+In+The+Thirteenth+International+Conference+on+Learning+Representations%2E)
36. Kevin Wu, Eric Wu, and James Zou. 2024b. Clasheval: Quantifying the tug-of-war between an llm’s internal prior and external evidence. Advances in neural information processing systems, 37:33402–33422. [find](https://scholar.google.com/scholar?q=Kevin+Wu%2C+Eric+Wu%2C+and+James+Zou%2E+2024b%2E+Clasheval%3A+Quantifying+the+tug%2Dof%2Dwar+between+an+llm%E2%80%99s+internal+prior+and+external+evidence%2E+Advances+in+neural+information+processing+systems%2C+37%3A33402%E2%80%9333422%2E)
37. Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024. Do large language models latently perform multi-hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10210– 10229, Bangkok, Thailand.… [find](https://scholar.google.com/scholar?q=Sohee+Yang%2C+Elena+Gribovskaya%2C+Nora+Kassner%2C+Mor+Geva%2C+and+Sebastian+Riedel%2E+2024%2E+Do+large+language+models+latently+perform+multi%2Dhop+reasoning%3F+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Assoc)
38. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369… [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D%2E+Manning%2E+2018%2E+HotpotQA%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+I) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
39. Wenhao Yu, Meng Jiang, Peter Clark, and Ashish Sabharwal. 2023a. IfQA: A dataset for open-domain question answering under counterfactual presuppositions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8276–8288, Singapore. Association for Computation… [find](https://scholar.google.com/scholar?q=Wenhao+Yu%2C+Meng+Jiang%2C+Peter+Clark%2C+and+Ashish+Sabharwal%2E+2023a%2E+IfQA%3A+A+dataset+for+open%2Ddomain+question+answering+under+counterfactual+presuppositions%2E+In+Proceedings+of+the+2023+Conference+on+Empir) [our card](https://md.research.elamur.ai/free/en/papers/yu-2023-ifqa-dataset-open-domain-question/)
40. Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023b. CREPE: Open-domain question answering with false presuppositions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10457–10480, Toronto, Canada. Associatio… [find](https://scholar.google.com/scholar?q=Xinyan+Yu%2C+Sewon+Min%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+2023b%2E+CREPE%3A+Open%2Ddomain+question+answering+with+false+presuppositions%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+f) [our card](https://md.research.elamur.ai/free/en/papers/yu-2023-crepe-open-domain-question-answering/)