---
title: "ASQA: Factoid Questions Meet Long-Form — Stelmakh 2022"
description: "Long-form question answering has advanced slowly compared with short factoid QA."
authors: ["Ivan Stelmakh", "Yi Luan", "Bhuwan Dhingra", "Ming-Wei Chang"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.566/"
doi: "10.18653/v1/2022.emnlp-main.566"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/stelmakh-2022-asqa-factoid-questions-meet-long/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/stelmakh-2022-asqa-factoid-questions-meet-long/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / ASQA: Factoid Questions Meet Long-Form Answers

# ASQA: Factoid Questions Meet Long-Form Answers

**Authors:** Ivan Stelmakh, [Yi Luan](https://md.research.elamur.ai/free/en/authors/yi-luan/), [Bhuwan Dhingra](https://md.research.elamur.ai/free/en/authors/bhuwan-dhingra/), [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/) · **Year:** 2022 · **Venue:** Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.566/) · [PDF](https://aclanthology.org/2022.emnlp-main.566.pdf) · [DOI 10.18653/v1/2022.emnlp-main.566](https://doi.org/10.18653/v1/2022.emnlp-main.566) · [OpenAlex](https://openalex.org/W4385573898)

**Citations:** 71 citations (openalex-snapshot · 2026-10-14) · [168](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Long-form question answering has advanced slowly compared with short factoid QA. Two things hold it back: good training data is scarce, and there is no clear standard for what makes an answer good. The authors address this with ASQA, built from ambiguous questions in AmbigQA. Each question is paired with a crowdsourced paragraph that explains why the question is ambiguous and connects all valid short answers into one coherent text, using supporting Wikipedia evidence. Because each disambiguated sub-question has a known answer, correctness can be checked: a good answer should let a reader resolve every interpretation.

On this basis they introduce the DR score, which merges ROUGE-L (fluency and conciseness) with Disambig-F1 (whether a QA model can recover the short answers from the generated text). A small human study shows DR tracks human judgments closely. Baselines combine JPR retrieval with T5-large and remain far below human performance. This makes ASQA a measurable benchmark for both retrieval and summarization in long-form QA.

## Key points
- ASQA contains long-form answers for 6,316 ambiguous questions. They are drawn from the 7,207 AmbigQA questions flagged as ambiguous, out of 14,042 total.
- DR is the geometric mean of ROUGE-L and Disambig-F1. Disambig-F1 relies on a QA model trained on SQuAD v2.
- Human evaluation on 45 development questions found that DR had the highest Pearson correlation with the overall human score HO (95.0).
- The best baseline, T5-O-5, reaches 32.1 DR. Human performance lies between 40.6 and 61.8 DR.
- Analysis shows that both retrieval and summarization need work:
  - JPR@5 trails an oracle by 14.4 Disambig-F1.
  - T5 loses information from the passages it is given.
  - Concise answers matter: human references average about 65 words, while JPR@1 outputs average 196.8 words.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

168 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 159 not yet.

- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/)
- [Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models](https://md.research.elamur.ai/free/en/papers/bohnet-2022-attributed-question-answering-evaluation-modeling/)
- [Enabling Large Language Models to Generate Text with Citations](https://md.research.elamur.ai/free/en/papers/gao-2023-enabling-large-language-models-generate/)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/)
- [PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering](https://md.research.elamur.ai/free/en/papers/han-2023-prewome-exploiting-presuppositions-as-working/)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/)
- [The Mark Fades: Adaptive Evolutionary Paraphrase-based Attack against LLM Watermarks](https://md.research.elamur.ai/free/en/papers/zhao-2026-mark-fades-adaptive-evolutionary-paraphrase/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{stelmakh2022asqa,
  title = {{ASQA: Factoid Questions Meet Long-Form Answers}},
  author = {Stelmakh, Ivan and Luan, Yi and Dhingra, Bhuwan and Chang, Ming-Wei},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.566},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.566}
}
```

- [← previous Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://md.research.elamur.ai/free/en/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/)
- [next → Assessing the Linguistic Knowledge in Arabic Pre-trained Language Models Using Minimal Pairs](https://md.research.elamur.ai/free/en/papers/alrajhi-2022-assessing-linguistic-knowledge-arabic-pre/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

32 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. q 2 : Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,… [find](https://scholar.google.com/scholar?q=Or+Honovich%2C+Leshem+Choshen%2C+Roee+Aharoni%2C+Ella+Neeman%2C+Idan+Szpektor%2C+and+Omri+Abend%2E+2021%2E+q+2+%3A+Evaluating+factual+consistency+in+knowledge%2Dgrounded+dialogues+via+question+generation+and+question+a)
2. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Ming%2DWei+Chang%2C+and+Kristina+Toutanova%2E+2019%2E+Latent+retrieval+for+weakly+supervised+open+domain+question+answering%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Comput) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)
3. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880, Online. Association for Computa… [find](https://scholar.google.com/scholar?q=Gautier+Izacard+and+Edouard+Grave%2E+2021%2E+Leveraging+passage+retrieval+with+generative+models+for+open+domain+question+answering%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+As) [our card](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/)
4. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Robin+Jia+and+Percy+Liang%2E+2017%2E+Adversarial+examples+for+evaluating+reading+comprehension+systems%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+202) [our card](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
5. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver… [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+Proceedings+of+the+55th+Annual+Meeting+) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
6. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli… [find](https://scholar.google.com/scholar?q=Vladimir+Karpukhin%2C+Barlas+Oguz%2C+Sewon+Min%2C+Patrick+Lewis%2C+Ledell+Wu%2C+Sergey+Edunov%2C+Danqi+Chen%2C+and+Wen%2Dtau+Yih%2E+2020%2E+Dense+passage+retrieval+for+open%2Ddomain+question+answering%2E+In+Proceedings+of+th) [our card](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)
7. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328. [find](https://scholar.google.com/scholar?q=Tom%C3%A1%C5%A1+Koc%CC%8Cisk%C3%BD%2C+Jonathan+Schwarz%2C+Phil+Blunsom%2C+Chris+Dyer%2C+Karl+Moritz+Hermann%2C+G%C3%A1bor+Melis%2C+and+Edward+Grefenstette%2E+2018%2E+The+NarrativeQA+reading+comprehension+challenge%2E+Transactions+of+the+Associ) [our card](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)
8. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4940–4957, Online. Association for Computat… [find](https://scholar.google.com/scholar?q=Kalpesh+Krishna%2C+Aurko+Roy%2C+and+Mohit+Iyyer%2E+2021%2E+Hurdles+to+progress+in+long%2Dform+question+answering%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Compu) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/)
9. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Red%2Dfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+Toutanova%2C+Llion+Jones%2C+Mat)
10. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
11. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198. [find](https://scholar.google.com/scholar?q=Peter+J+Liu%2C+Mohammad+Saleh%2C+Etienne+Pot%2C+Ben+Goodrich%2C+Ryan+Sepassi%2C+Lukasz+Kaiser%2C+and+Noam+Shazeer%2E+2018%2E+Generating+wikipedia+by+summarizing+long+sequences%2E+arXiv+preprint+arXiv%3A1801%2E10198%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/)
12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+C)
13. Sewon Min, Kenton Lee, Ming-Wei Chang, Kristina Toutanova, and Hannaneh Hajishirzi. 2021. Joint passage ranking for diverse multi-answer retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6997–7008, Online and Punta Cana, Dominican Republic. A… [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Kristina+Toutanova%2C+and+Hannaneh+Hajishirzi%2E+2021%2E+Joint+passage+ranking+for+diverse+multi%2Danswer+retrieval%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+M)
14. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2020%2E+AmbigQA%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+La)
15. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. [find](https://scholar.google.com/scholar?q=Reiichiro+Nakano%2C+Jacob+Hilton%2C+Suchir+Balaji%2C+Jeff+Wu%2C+Long+Ouyang%2C+Christina+Kim%2C+Christopher+Hesse%2C+Shantanu+Jain%2C+Vineet+Kosaraju%2C+William+Saunders%2C+et+al%2E+2021%2E+Webgpt%3A+Browser%2Dassisted+question%2D) [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
16. Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1063–1072, Vancouver, Can… [find](https://scholar.google.com/scholar?q=Preksha+Nema%2C+Mitesh+M%2E+Khapra%2C+Anirban+Laha%2C+and+Balaraman+Ravindran%2E+2017%2E+Diversity+driven+attention+model+for+query%2Dbased+abstractive+summarization%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+th)
17. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated MAchine Reading COmprehension dataset. CoRR, abs/1611.09268. [find](https://scholar.google.com/scholar?q=Tri+Nguyen%2C+Mir+Rosenberg%2C+Xia+Song%2C+Jianfeng+Gao%2C+Saurabh+Tiwary%2C+Rangan+Majumder%2C+and+Li+Deng%2E+2016%2E+MS+MARCO%3A+A+human+generated+MAchine+Reading+COmprehension+dataset%2E+CoRR%2C+abs%2F1611%2E09268%2E) [our card](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)
18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2019%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
19. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+SQuAD%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
20. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
21. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. [find](https://scholar.google.com/scholar?q=Siva+Reddy%2C+Danqi+Chen%2C+and+Christopher+D%2E+Manning%2E+2019%2E+CoQA%3A+A+conversational+question+answering+challenge%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A249%E2%80%93266%2E) [our card](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)
22. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Langu)
23. Claude Sammut and Geoffrey I. Webb, editors. 2010. TF–IDF, pages 986–987. Springer US, Boston, MA. [find](https://scholar.google.com/scholar?q=Claude+Sammut+and+Geoffrey+I%2E+Webb%2C+editors%2E+2010%2E+TF%E2%80%93IDF%2C+pages+986%E2%80%93987%2E+Springer+US%2C+Boston%2C+MA%2E)
24. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008– 3021. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+2020%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural)
25. Retrieval, SIGIR ’00, page 200–207, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Retrieval%2C+SIGIR+%E2%80%9900%2C+page+200%E2%80%93207%2C+New+York%2C+NY%2C+USA%2E+Association+for+Computing+Machinery%2E)
26. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comput)
27. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+Remi+Louf%2C+Morgan+Funtow%2Dicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
28. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369… [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D%2E+Manning%2E+2018%2E+HotpotQA%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+I) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
29. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapte… [find](https://scholar.google.com/scholar?q=Ming+Zhong%2C+Da+Yin%2C+Tao+Yu%2C+Ahmad+Zaidi%2C+Mutethia+Mutuma%2C+Rahul+Jha%2C+Ahmed+Hassan+Awadallah%2C+Asli+Celikyilmaz%2C+Yang+Liu%2C+Xipeng+Qiu%2C+and+Dragomir+Radev%2E+2021%2E+QMSum%3A+A+new+benchmark+for+query%2Dbased+mu) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2021-qmsum-new-benchmark-query-based/)
30. Haitian Sun, William W Cohen, and Ruslan Salakhutdinov. 2021. Conditionalqa: A complex reading comprehension dataset with conditional answers. arXiv preprint arXiv:2110.06884. [find](https://scholar.google.com/scholar?q=Haitian+Sun%2C+William+W+Cohen%2C+and+Ruslan+Salakhutdinov%2E+2021%2E+Conditionalqa%3A+A+complex+reading+comprehension+dataset+with+conditional+answers%2E+arXiv+preprint+arXiv%3A2110%2E06884%2E)
31. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200, Vancouver, Canada. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Adam+Trischler%2C+Tong+Wang%2C+Xingdi+Yuan%2C+Justin+Harris%2C+Alessandro+Sordoni%2C+Philip+Bachman%2C+and+Kaheer+Suleman%2E+2017%2E+NewsQA%3A+A+machine+comprehension+dataset%2E+In+Proceedings+of+the+2nd+Workshop+on+Repr) [our card](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/)
32. Ellen M. Voorhees and Dawn M. Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information [find](https://scholar.google.com/scholar?q=Ellen+M%2E+Voorhees+and+Dawn+M%2E+Tice%2E+2000%2E+Building+a+question+answering+test+collection%2E+In+Proceedings+of+the+23rd+Annual+International+ACM+SIGIR+Conference+on+Research+and+Development+in+Information)