---
title: "Evidentiality-guided Generation for — Asai 2022"
description: "Retrieval-augmented generators such as FiD often produce answers from retrieved passages that look relevant but do not actually support the output."
authors: ["Akari Asai", "Matt Gardner", "Hannaneh Hajishirzi"]
year: 2022
venue: "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2022.naacl-main.162/"
doi: "10.18653/v1/2022.naacl-main.162"
arxiv: "2112.08688"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/asai-2022-evidentiality-guided-generation-knowledge-intensive/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/asai-2022-evidentiality-guided-generation-knowledge-intensive/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2022](https://md.research.elamur.ai/free/papers/2022/) / Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks

# Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks

**Authors:** [Akari Asai](https://md.research.elamur.ai/free/authors/akari-asai/), [Matt Gardner](https://md.research.elamur.ai/free/authors/matt-gardner/), [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/authors/hannaneh-hajishirzi/) · **Year:** 2022 · **Venue:** Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.naacl-main.162/) · [PDF](https://aclanthology.org/2022.naacl-main.162.pdf) · [DOI 10.18653/v1/2022.naacl-main.162](https://doi.org/10.18653/v1/2022.naacl-main.162) · [arXiv 2112.08688](https://arxiv.org/abs/2112.08688) · [OpenAlex](https://openalex.org/W4225992558)

**Citations:** 31 citations (openalex-snapshot · 2026-10-14) · [29](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Retrieval-augmented generators such as FiD often produce answers from retrieved passages that look relevant but do not actually support the output. They may exploit lexical overlap or a matching answer string, or they may hallucinate. Asai, Gardner and Hajishirzi address this by teaching the generator to judge each passage's evidentiality. Their model is trained jointly on two tasks: producing the output and classifying every input passage as evidence-positive or evidence-negative. Gold evidence labels are rarely available, so the authors build a task-agnostic pipeline for silver labels. A RoBERTa-based labeler is trained on partial gold annotations combined with passages identified through leave-one-out generation, which tests whether removing a passage breaks the correct prediction. The trained labeler then annotates all training passages. The approach was tested on open-domain QA, fact verification and knowledge-grounded dialogue. It consistently beat plain FiD and set new best results on three benchmarks. This matters because it improves the generator itself rather than the retriever, and it needs no task-specific heuristics.

## Key points
- The method was evaluated on five datasets: Natural Questions Open, TriviaQA (unfiltered), FaVIQ-Ambig, FEVER and Wizard of Wikipedia. On three of them (Wizard of Wikipedia, FEVER and FaVIQ-Ambig), it outperformed all previously reported systems.
- The base generator is FiD (T5 encoder-decoder), and an auxiliary evidentiality-prediction head is added on top. The silver labeler M is a RoBERTa binary classifier that takes the query, the gold output and one passage.
- In human evaluation, the silver evidentiality labels were about 95% accurate. The labeler often caught negative passages that contained the answer string only by coincidence.
- Ablations show that both the multi-task objective and the silver-label mining contribute to the gains. The gains appear mainly on harder examples that require reading beyond the top-ranked passage.
- Code and trained models are released at github.com/AkariAsai/evidentiality_qa.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

29 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 28 not yet.

- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://md.research.elamur.ai/free/papers/kamalloo-2023-evaluating-open-domain-question-answering/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{asai2022evidentiality,
  title = {{Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks}},
  author = {Asai, Akari and Gardner, Matt and Hajishirzi, Hannaneh},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.162},
  eprint = {2112.08688},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.162}
}
```

- [← previous End-to-end Multilingual Coreference Resolution with Mention Head Prediction](https://md.research.elamur.ai/free/papers/prazak-2022-end-end-multilingual-coreference-resolution/)
- [next → Exploiting Sentiment and Common Sense for Zero-shot Stance Detection](https://md.research.elamur.ai/free/papers/luo-2022-exploiting-sentiment-common-sense-zero/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/papers/2022/)

## References

45 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Asma Ben Abacha and Dina Demner-Fushman. 2019. A question-entailment approach to question answering. BMC bioinformatics. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha+and+Dina+Demner%2DFushman%2E+2019%2E+A+question%2Dentailment+approach+to+question+answering%2E+BMC+bioinformatics%2E)
2. Akari Asai and Eunsol Choi. 2021. Challenges in information seeking QA: Unanswerable questions and paragraph retrieval. In ACL. [find](https://scholar.google.com/scholar?q=Akari+Asai+and+Eunsol+Choi%2E+2021%2E+Challenges+in+information+seeking+QA%3A+Unanswerable+questions+and+paragraph+retrieval%2E+In+ACL%2E)
3. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2021. Improving language models by retrieving from trillions of tokens. [find](https://scholar.google.com/scholar?q=Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Jordan+Hoffmann%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Katie+Millican%2C+George+van+den+Driessche%2C+Jean%2DBaptiste+Lespiau%2C+Bogdan+Damoc%2C+Aidan+Clark%2C+et+al%2E+2021%2E+Improving+lan)
4. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. [find](https://scholar.google.com/scholar?q=Tom+B+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear)
5. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In ACL. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Adam+Fisch%2C+Jason+Weston%2C+and+Antoine+Bordes%2E+2017%2E+Reading+Wikipedia+to+answer+open%2Ddomain+questions%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/chen-2017-reading-wikipedia-answer-open-domain/)
6. Jifan Chen, Eunsol Choi, and Greg Durrett. 2021. Can nli models verify qa systems’ predictions? In EMNLP Findings. [find](https://scholar.google.com/scholar?q=Jifan+Chen%2C+Eunsol+Choi%2C+and+Greg+Durrett%2E+2021%2E+Can+nli+models+verify+qa+systems%E2%80%99+predictions%3F+In+EMNLP+Findings%2E)
7. Jifan Chen, Shih-ting Lin, and Greg Durrett. 2019. Multi-hop question answering via reasoning chains. [find](https://scholar.google.com/scholar?q=Jifan+Chen%2C+Shih%2Dting+Lin%2C+and+Greg+Durrett%2E+2019%2E+Multi%2Dhop+question+answering+via+reasoning+chains%2E)
8. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+ICLR%2E)
9. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In ICLR. [find](https://scholar.google.com/scholar?q=Emily+Dinan%2C+Stephen+Roller%2C+Kurt+Shuster%2C+Angela+Fan%2C+Michael+Auli%2C+and+Jason+Weston%2E+2019%2E+Wizard+of+wikipedia%3A+Knowledge%2Dpowered+conversational+agents%2E+In+ICLR%2E)
10. Martin Fajcik, Martin Docekal, Karel Ondrej, and Pavel Smrz. 2021. R2-D2: A modular baseline for open-domain question answering. In EMNLP Findings. [find](https://scholar.google.com/scholar?q=Martin+Fajcik%2C+Martin+Docekal%2C+Karel+Ondrej%2C+and+Pavel+Smrz%2E+2021%2E+R2%2DD2%3A+A+modular+baseline+for+open%2Ddomain+question+answering%2E+In+EMNLP+Findings%2E)
11. Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, and Alfio Gliozzo. 2021. Robust retrieval augmented generation for zero-shot slot filling. In EMNLP. [find](https://scholar.google.com/scholar?q=Michael+Glass%2C+Gaetano+Rossiello%2C+Md+Faisal+Mahbub+Chowdhury%2C+and+Alfio+Gliozzo%2E+2021%2E+Robust+retrieval+augmented+generation+for+zero%2Dshot+slot+filling%2E+In+EMNLP%2E)
12. Sanda Harabagiu and Andrew Hickl. 2006. Methods for using textual entailment in open-domain question answering. In ACL. [find](https://scholar.google.com/scholar?q=Sanda+Harabagiu+and+Andrew+Hickl%2E+2006%2E+Methods+for+using+textual+entailment+in+open%2Ddomain+question+answering%2E+In+ACL%2E)
13. Srinivasan Iyer, Sewon Min, Yashar Mehdad, and Wen-tau Yih. 2021. RECONSIDER: Improved reranking using span-focused cross-attention for open domain question answering. In NAACL. [find](https://scholar.google.com/scholar?q=Srinivasan+Iyer%2C+Sewon+Min%2C+Yashar+Mehdad%2C+and+Wen%2Dtau+Yih%2E+2021%2E+RECONSIDER%3A+Improved+reranking+using+span%2Dfocused+cross%2Dattention+for+open+domain+question+answering%2E+In+NAACL%2E)
14. Gautier Izacard and Edouard Grave. 2021a. Distilling knowledge from reader to retriever for question answering. In ICLR. [find](https://scholar.google.com/scholar?q=Gautier+Izacard+and+Edouard+Grave%2E+2021a%2E+Distilling+knowledge+from+reader+to+retriever+for+question+answering%2E+In+ICLR%2E)
15. Gautier Izacard and Edouard Grave. 2021b. Leveraging passage retrieval with generative models for open domain question answering. In EACL. [find](https://scholar.google.com/scholar?q=Gautier+Izacard+and+Edouard+Grave%2E+2021b%2E+Leveraging+passage+retrieval+with+generative+models+for+open+domain+question+answering%2E+In+EACL%2E) [our card](https://md.research.elamur.ai/free/papers/izacard-2021-leveraging-passage-retrieval-generative-models/)
16. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S%2E+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
17. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Vladimir+Karpukhin%2C+Barlas+Og%CC%86uz%2C+Sewon+Min%2C+Ledell+Wu%2C+Sergey+Edunov%2C+Danqi+Chen%2C+and+Wen%2Dtau+Yih%2E+2020%2E+Dense+passage+retrieval+for+open%2Ddomain+question+answering%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)
18. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR%2E)
19. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In NAACL. [find](https://scholar.google.com/scholar?q=Kalpesh+Krishna%2C+Aurko+Roy%2C+and+Mohit+Iyyer%2E+2021%2E+Hurdles+to+progress+in+long%2Dform+question+answering%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/papers/krishna-2021-hurdles-progress-long-form-question/)
20. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Red%2Dfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+Toutanova%2C+Llion+Jones%2C+Mat) [our card](https://md.research.elamur.ai/free/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
21. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In ACL. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Ming%2DWei+Chang%2C+and+Kristina+Toutanova%2E+2019%2E+Latent+retrieval+for+weakly+supervised+open+domain+question+answering%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/lee-2019-latent-retrieval-weakly-supervised-open/)
22. Kyungjae Lee, Seung-won Hwang, Sang-eun Han, and Dohyeon Lee. 2021. Robustifying multi-hop QA through pseudo-evidentiality training. In ACL. [find](https://scholar.google.com/scholar?q=Kyungjae+Lee%2C+Seung%2Dwon+Hwang%2C+Sang%2Deun+Han%2C+and+Dohyeon+Lee%2E+2021%2E+Robustifying+multi%2Dhop+QA+through+pseudo%2Devidentiality+training%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/lee-2021-robustifying-multi-hop-qa-through/)
23. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020a%2E+BART%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+) [our card](https://md.research.elamur.ai/free/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020b%2E+BART%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+) [our card](https://md.research.elamur.ai/free/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
25. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020c. Retrieval-augmented generation for knowledge-intensive nlp tasks. In NeruIPS. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ethan+Perez%2C+Aleksandra+Piktus%2C+Fabio+Petroni%2C+Vladimir+Karpukhin%2C+Naman+Goyal%2C+Heinrich+K%C3%BCttler%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Tim+Rockt%C3%A4schel%2C+Sebastian+Riedel%2C+and+Douwe+Kiela%2E+2020c%2E+Ret)
26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+bert+pretraining+approach%2E)
27. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Shayne+Longpre%2C+Kartik+Perisetla%2C+Anthony+Chen%2C+Nikhil+Ramesh%2C+Chris+DuBois%2C+and+Sameer+Singh%2E+2021%2E+Entity%2Dbased+knowledge+conflicts+in+question+answering%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/papers/longpre-2021-entity-based-knowledge-conflicts-question/)
28. Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oguz, Veselin Stoyanov, and Gargi Ghosh. 2021. Multi-task retrieval for knowledge-intensive tasks. In ACL. [find](https://scholar.google.com/scholar?q=Jean+Maillard%2C+Vladimir+Karpukhin%2C+Fabio+Petroni%2C+Wen%2Dtau+Yih%2C+Barlas+Oguz%2C+Veselin+Stoyanov%2C+and+Gargi+Ghosh%2E+2021%2E+Multi%2Dtask+retrieval+for+knowledge%2Dintensive+tasks%2E+In+ACL%2E)
29. Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. A discrete hard em approach for weakly supervised question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Danqi+Chen%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2019%2E+A+discrete+hard+em+approach+for+weakly+supervised+question+answering%2E+In+EMNLP%2E)
30. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In EMNLP. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2020%2E+AmbigQA%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+EMNLP%2E)
31. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In ACL. [find](https://scholar.google.com/scholar?q=Mike+Mintz%2C+Steven+Bills%2C+Rion+Snow%2C+and+Daniel+Jurafsky%2E+2009%2E+Distant+supervision+for+relation+extraction+without+labeled+data%2E+In+ACL%2E)
32. Kosuke Nishida, Kyosuke Nishida, Nagata Masaaki, Atsushi Otsuka, Itsumi Saito, Hisako Asano, and Junji Tomita. 2019. Answering while summarizing: Multi-task learning for multi-hop qa with evidence extraction. In ACL. [find](https://scholar.google.com/scholar?q=Kosuke+Nishida%2C+Kyosuke+Nishida%2C+Nagata+Masaaki%2C+Atsushi+Otsuka%2C+Itsumi+Saito%2C+Hisako+Asano%2C+and+Junji+Tomita%2E+2019%2E+Answering+while+summarizing%3A+Multi%2Dtask+learning+for+multi%2Dhop+qa+with+evidence+ext)
33. Ashwin Paranjape, Omar Khattab, Christopher Potts, Matei Zaharia, and Christopher D Manning. 2021. Hindsight: Posterior-guided training of retrievers for improved open-ended generation. [find](https://scholar.google.com/scholar?q=Ashwin+Paranjape%2C+Omar+Khattab%2C+Christopher+Potts%2C+Matei+Zaharia%2C+and+Christopher+D+Manning%2E+2021%2E+Hindsight%3A+Posterior%2Dguided+training+of+retrievers+for+improved+open%2Dended+generation%2E)
34. Jungsoo Park, Sewon Min, Jaewoo Kang, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. FaVIQ: Fact verification from information seeking questions. [find](https://scholar.google.com/scholar?q=Jungsoo+Park%2C+Sewon+Min%2C+Jaewoo+Kang%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+2021%2E+FaVIQ%3A+Fact+verification+from+information+seeking+questions%2E)
35. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and… [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Francisco+Massa%2C+Adam+Lerer%2C+James+Bradbury%2C+Gregory+Chanan%2C+Trevor+Killeen%2C+Zeming+Lin%2C+Natalia+Gimelshein%2C+Luca+Antiga%2C+Alban+Desmaison%2C+Andreas+Kopf%2C+Edward+Yang%2C+Zachary+De) [our card](https://md.research.elamur.ai/free/papers/paszke-2019-pytorch-imperative-style-high-performance/)
36. Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Patrick+Lewis%2C+Wen%2Dtau+Yih%2C+Kyunghyun+Cho%2C+and+Douwe+Kiela%2E+2020%2E+Unsupervised+question+decomposition+for+question+answering%2E+In+EMNLP%2E)
37. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In NAACL. [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Aleksandra+Piktus%2C+Angela+Fan%2C+Patrick+Lewis%2C+Majid+Yazdani%2C+Nicola+De+Cao%2C+James+Thorne%2C+Yacine+Jernite%2C+Vladimir+Karpukhin%2C+Jean+Maillard%2C+Vassilis+Plachouras%2C+Tim+Rockt%C3%A4schel%2C+and+Se) [our card](https://md.research.elamur.ai/free/papers/petroni-2020-kilt-benchmark-knowledge-intensive-language/)
38. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
39. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval. [find](https://scholar.google.com/scholar?q=Stephen+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+BM25+and+beyond%2E+Foundations+and+Trends+in+Information+Retrieval%2E) [our card](https://md.research.elamur.ai/free/papers/robertson-2009-probabilistic-relevance-framework-bm25/)
40. Bogdan Sacaleanu, Constantin Orasan, Christian Spurk, Shiyan Ou, Oscar Ferrandez, Milen Kouylekov, and Matteo Negri. 2008. Entailment-based question answering for structured data. In COLING. [find](https://scholar.google.com/scholar?q=Bogdan+Sacaleanu%2C+Constantin+Orasan%2C+Christian+Spurk%2C+Shiyan+Ou%2C+Oscar+Ferrandez%2C+Milen+Kouylekov%2C+and+Matteo+Negri%2E+2008%2E+Entailment%2Dbased+question+answering+for+structured+data%2E+In+COLING%2E)
41. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In NAACL. [find](https://scholar.google.com/scholar?q=James+Thorne%2C+Andreas+Vlachos%2C+Christos+Christodoulopoulos%2C+and+Arpit+Mittal%2E+2018%2E+FEVER%3A+a+large%2Dscale+dataset+for+fact+extraction+and+VERification%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/papers/thorne-2018-fever-large-scale-dataset-fact/)
42. Harsh Trivedi, Heeyoung Kwon, Tushar Khot, Ashish Sabharwal, and Niranjan Balasubramanian. 2019. Repurposing entailment for multi-hop question answering tasks. In NAACL. [find](https://scholar.google.com/scholar?q=Harsh+Trivedi%2C+Heeyoung+Kwon%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Niranjan+Balasubramanian%2E+2019%2E+Repurposing+entailment+for+multi%2Dhop+question+answering+tasks%2E+In+NAACL%2E)
43. Peng Xu, Davis Liang, Zhiheng Huang, and Bing Xiang. 2021. Attention-guided generative models for extractive question answering. [find](https://scholar.google.com/scholar?q=Peng+Xu%2C+Davis+Liang%2C+Zhiheng+Huang%2C+and+Bing+Xiang%2E+2021%2E+Attention%2Dguided+generative+models+for+extractive+question+answering%2E)
44. Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2019. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Vikas+Yadav%2C+Steven+Bethard%2C+and+Mihai+Surdeanu%2E+2019%2E+Quick+and+%28not+so%29+dirty%3A+Unsupervised+selection+of+justification+sentences+for+multi%2Dhop+question+answering%2E+In+EMNLP%2E)
45. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D%2E+Manning%2E+2018%2E+HotpotQA%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+I) [our card](https://md.research.elamur.ai/free/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)