---
title: "FanOutQA — paper by Zhu et al. (2024) — summary and ideas"
description: "The authors define three settings: closed-book (parametric knowledge only), open-book (retrieval… 20 references, cited by 36 papers here."
authors: ["Andrew Zhu", "Alyssa Hwang", "Liam Dugan", "Chris Callison-Burch"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2024.acl-short.2/"
doi: "10.18653/v1/2024.acl-short.2"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2024-fanoutqa-multi-hop-multi-document/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2024-fanoutqa-multi-hop-multi-document/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models

# FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models

**Authors:** [Andrew Zhu](https://md.research.elamur.ai/free/en/authors/andrew-zhu/), [Alyssa Hwang](https://md.research.elamur.ai/free/en/authors/alyssa-hwang/), [Liam Dugan](https://md.research.elamur.ai/free/en/authors/liam-dugan/) and [Chris Callison-Burch](https://md.research.elamur.ai/free/en/authors/chris-callison-burch/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-short.2/) · [PDF](https://aclanthology.org/2024.acl-short.2.pdf) · [DOI 10.18653/v1/2024.acl-short.2](https://doi.org/10.18653/v1/2024.acl-short.2) · [OpenAlex](https://openalex.org/W4402671478)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [36](https://md.research.elamur.ai/free/en/papers/zhu-2024-fanoutqa-multi-hop-multi-document/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Many real-world requests are "fan-out" questions. To answer one, a system must first identify a set of entities and then gather and combine facts about each from separate documents. Existing benchmarks such as HotpotQA, LongBench and ZeroSCROLLS mostly test reasoning within one document or across only a few. FanOutQA fills this gap with an English Wikipedia–based dataset. Students wrote the questions along with step-by-step decompositions into sub-questions, each answerable from a single article. The authors define three settings: closed-book (parametric knowledge only), open-book (retrieval over a fixed Wikipedia snapshot) and evidence-provided (the relevant articles are supplied). They evaluate seven LLMs, including GPT-4, LLaMA 2, Claude 2.1 and Mixtral-8x7B. The best models score under 50% in the closed- and open-book settings. Open-book results are often worse than closed-book because long retrieved passages crowd out the original question. Humans reach about 85% in the open-book setting. The benchmark therefore exposes a clear weakness in long-context, cross-document reasoning.

## Key points
- **Dataset size and coverage:** 1,034 top-level questions, 7,305 human-written sub-questions and 4,121 distinct Wikipedia articles.
  - The questions were written by 379 university students.
  - The data is split into 310 dev and 724 test questions.
  - A public leaderboard is hosted at fanoutqa.com.
- **Scale of each question:** on average, a question needs about 7 hops and roughly 172k tokens of evidence from about 7 documents. The knowledge base is a Wikipedia snapshot from November 20, 2023.
- **Closed-book results:** Claude had the lowest loose accuracy at 0.341, while Mixtral had the highest at 0.470. Errors were mostly plausible hallucinations, and GPT-4 declined to answer 44% of the time.
- **Evidence-provided results:** accuracy correlated strongly with model context length (r² = 0.782).
- **Metrics and release:**
  - Evaluation uses loose/strict string accuracy, ROUGE, BLEURT and a GPT-4 judge.
  - Open-book runs use Kani with BM25+ chunk retrieval.
  - The data is released under CC BY-SA 4.0 and the Python package under MIT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 221 works · 63,234 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works

## Cite

Cite as: Andrew Zhu, Alyssa Hwang, Liam Dugan and Chris Callison-Burch. 2024. FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhu2024fanoutqa,
  title = {{FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models}},
  author = {Andrew Zhu, Alyssa Hwang, Liam Dugan and Chris Callison-Burch},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-short.2},
  url = {https://doi.org/10.18653/v1/2024.acl-short.2}
}
```

- [← previous Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving](https://md.research.elamur.ai/free/en/papers/quan-2024-verification-refinement-natural-language-explanations/)
- [next → From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models](https://md.research.elamur.ai/free/en/papers/welleck-2024-decoding-meta-generation-inference-time/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

20 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508.
2. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
3. Yuanhua Lv and ChengXiang Zhai. 2011. Lower-bounding term frequency normalization. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM ’11, page 7–16, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Yuanhua+Lv+and+ChengXiang+Zhai%2E+2011%2E+Lower%2Dbounding+term+frequency+normalization%2E+In+Proceedings+of+the+20th+ACM+International+Conference+on+Information+and+Knowledge+Management%2C+CIKM+%E2%80%9911%2C+page+7%E2%80%9316%2C)
4. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan…
5. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. [Compositional questions do not necessitate multi-hop reasoning](https://md.research.elamur.ai/free/en/papers/min-2019-compositional-questions-do-not-necessitate/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4249–4257, Florence, Italy. Associat…
6. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. [Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm…
7. Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madelei… — [Text and Code Embeddings by Contrastive Pre-Training](https://md.research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/)
8. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In International Conference on Learning Representations.
9. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
10. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
11. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. [BLEURT: Learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.
12. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
13. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. [ZeroSCROLLS: A zero-shot benchmark for long text understanding](https://md.research.elamur.ai/free/en/papers/shaham-2023-zeroscrolls-zero-shot-benchmark-long/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, Singapore. Association for Computational Linguistics.
14. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. [SCROLLS: Standardized CompaRison over long language sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Tushar Khot, H…
15. Alon Talmor and Jonathan Berant. 2018. [The web as a knowledge-base for answering complex questions](https://md.research.elamur.ai/free/en/papers/talmor-2018-web-as-knowledge-base-answering/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651, New Orleans, L…
16. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
17. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. [find](https://scholar.google.com/scholar?q=Harsh+Trivedi%2C+Niranjan+Balasubramanian%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2022%2E+MuSiQue%3A+Multi%2Dhop+questions+via+single%2Dhop+question+composition%2E+Transactions+of+the+Association+for+Computational+Li)
18. Jian Wu, Linyi Yang, Yuliang Ji, Wenhao Huang, Börje F. Karlsson, and Manabu Okumura. 2024. Gendec: A robust generative question-decomposition method for multi-hop reasoning. [find](https://scholar.google.com/scholar?q=Jian+Wu%2C+Linyi+Yang%2C+Yuliang+Ji%2C+Wenhao+Huang%2C+B%C3%B6rje+F%2E+Karlsson%2C+and+Manabu+Okumura%2E+2024%2E+Gendec%3A+A+robust+generative+question%2Ddecomposition+method+for+multi%2Dhop+reasoning%2E)
19. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
20. Andrew Zhu, Liam Dugan, Alyssa Hwang, and Chris Callison-Burch. 2023. Kani: A lightweight and highly hackable framework for building language model applications. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 65–77, Singapore. Associatio… [find](https://scholar.google.com/scholar?q=Andrew+Zhu%2C+Liam+Dugan%2C+Alyssa+Hwang%2C+and+Chris+Callison%2DBurch%2E+2023%2E+Kani%3A+A+lightweight+and+highly+hackable+framework+for+building+language+model+applications%2E+In+Proceedings+of+the+3rd+Workshop+for)

*Updated 31 October 2026*