---
title: "Finding Generalizable Evidence by Learning to Convince Q&A"
description: "The paper asks whether the passage sentences that best support an answer can be found… 31 references, cited by 22 papers here."
authors: ["Ethan Perez", "Siddharth Karamcheti", "Rob Fergus", "Jason Weston", "Douwe Kiela", "Kyunghyun Cho"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1244/"
doi: "10.18653/v1/d19-1244"
arxiv: "1909.05863"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/perez-2019-finding-generalizable-evidence-learning-convince/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/perez-2019-finding-generalizable-evidence-learning-convince/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/) / Finding Generalizable Evidence by Learning to Convince Q&A Models

# Finding Generalizable Evidence by Learning to Convince Q&A Models

**Authors:** [Ethan Perez](https://md.research.elamur.ai/free/en/authors/ethan-perez/), [Siddharth Karamcheti](https://md.research.elamur.ai/free/en/authors/siddharth-karamcheti/), [Rob Fergus](https://md.research.elamur.ai/free/en/authors/rob-fergus/), [Jason Weston](https://md.research.elamur.ai/free/en/authors/jason-weston/), [Douwe Kiela](https://md.research.elamur.ai/free/en/authors/douwe-kiela/) and [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1244/) · [PDF](https://aclanthology.org/D19-1244.pdf) · [DOI 10.18653/v1/d19-1244](https://doi.org/10.18653/v1/d19-1244) · [arXiv 1909.05863](https://arxiv.org/abs/1909.05863) · [OpenAlex](https://openalex.org/W2970630590)

**Citations:** 21 citations (OpenAlex, 2026-10-14) · [22](https://md.research.elamur.ai/free/en/papers/perez-2019-finding-generalizable-evidence-learning-convince/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether the passage sentences that best support an answer can be found automatically, and whether that evidence is useful beyond the model that produced it. The authors use multiple-choice reading comprehension (RACE and DREAM) as a testbed. Each answer option gets its own "evidence agent," which picks passage sentences that raise a pretrained QA "judge" model's probability for that option. The judge sees only the selected sentences, not the whole passage. Judges include TFIDF and fastText similarity baselines plus fine-tuned BERT BASE and BERT LARGE. Agents either search exhaustively or are learned BERT BASE predictors of the judge's response, and they can act alone or compete (free-for-all or round robin). Evidence chosen to convince one judge also shifts other models and human readers toward the supported answer. Humans keep most of their accuracy while reading about a fifth of the passage, and QA models generalize better to longer passages and harder questions. This points to model-guided evidence selection as a practical aid for reading and robust QA.

## Key points
- **Datasets:** RACE has about 100K written passages with 4 answer options per question and is split into middle- and high-school exams. DREAM has about 10K dialogue passages with 3 options per question.
- **Judge accuracy on full passages:** BERT LARGE reaches 69.4% on RACE and 64.9% on DREAM. The TFIDF and fastText judges fall far lower, roughly 30–45%.
- **Human study:** People answered from agent-selected sentences only, about 3.3 vs 18.2 sentences on RACE and 2.4 vs 12.2 on DREAM. Their accuracy was 73.2% vs 82.3% (RACE) and 83.8% vs 93.0% (DREAM), keeping roughly 90% of full-passage accuracy.
- **Generalization:** Answering from the most convincing evidence helps judges transfer from short to long passages, and from middle-school to high-school RACE questions.
- **Learned agents:** Training an agent only requires querying the judge, not gold answer labels. Agents can therefore be trained on unlabeled or out-of-distribution data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 183 works · 228,558 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 201 works · 100,180 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

## Cite

Cite as: Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela and Kyunghyun Cho. 2019. Finding Generalizable Evidence by Learning to Convince Q&A Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{perez2019finding,
  title = {{Finding Generalizable Evidence by Learning to Convince Q\&A Models}},
  author = {Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela and Kyunghyun Cho},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1244},
  eprint = {1909.05863},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1244}
}
```

- [← previous Neural Decipherment via Minimum-Cost Flow: From Ugaritic to Linear B](https://md.research.elamur.ai/free/en/papers/luo-2019-neural-decipherment-minimum-cost-flow/)
- [next → OlloBot - Towards A Text-Based Arabic Health Conversational Agent: Evaluation and Results](https://md.research.elamur.ai/free/en/papers/fadhil-2019-ollobot-text-based-arabic-health/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/)

## References

31 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. 2019. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR. [find](https://scholar.google.com/scholar?q=Robert+Geirhos%2C+Patricia+Rubisch%2C+Claudio+Michaelis%2C+Matthias+Bethge%2C+Felix+A%2E+Wichmann%2C+and+Wieland+Brendel%2E+2019%2E+Imagenet%2Dtrained+CNNs+are+biased+towards+texture%3B+increasing+shape+bias+improves+acc)
2. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. [Generating natural language adversarial examples](https://md.research.elamur.ai/free/en/papers/alzantot-2018-generating-natural-language-adversarial-examples/). In EMNLP, pages 2890–2896.
3. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. [Annotation artifacts in natural language inference data](https://md.research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/). In NAACL, pages 107–112.
4. Kristjan Arumae and Fei Liu. 2018. Reinforced extractive summarization with question-focused rewards. [find](https://scholar.google.com/scholar?q=Kristjan+Arumae+and+Fei+Liu%2E+2018%2E+Reinforced+extractive+summarization+with+question%2Dfocused+rewards%2E)
5. Geoffrey Irving and Amanda Askell. 2019. AI safety needs social scientists. Distill. [find](https://scholar.google.com/scholar?q=Geoffrey+Irving+and+Amanda+Askell%2E+2019%2E+AI+safety+needs+social+scientists%2E+Distill%2E)
6. Geoffrey Irving, Paul Christiano, and Dario Amodei. 2018. AI safety via debate. CoRR, abs/1805.00899. [find](https://scholar.google.com/scholar?q=Geoffrey+Irving%2C+Paul+Christiano%2C+and+Dario+Amodei%2E+2018%2E+AI+safety+via+debate%2E+CoRR%2C+abs%2F1805%2E00899%2E)
7. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In EMNLP, pages 2021–2031.
8. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In EACL, pages 427–431. [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Tomas+Mikolov%2E+2017%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+EACL%2C+pages+427%E2%80%93431%2E)
9. Diederik P. Kingma and Jimmy Ba. 2015. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In ICLR.
10. Bernhard Kratzwald and Stefan Feuerriegel. 2019. Learning from on-line user feedback in neural question answering on the web. In WWW, pages 906– 916. [find](https://scholar.google.com/scholar?q=Bernhard+Kratzwald+and+Stefan+Feuerriegel%2E+2019%2E+Learning+from+on%2Dline+user+feedback+in+neural+question+answering+on+the+web%2E+In+WWW%2C+pages+906%E2%80%93+916%2E)
11. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [RACE: Large-scale ReAding Comprehension dataset from Examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In EMNLP, pages 785–794.
12. Barbara Landau, Linda B. Smith, and Susan S. Jones. 1988. The importance of shape in early lexical learning. Cognitive Development, 3(3):299–321. [find](https://scholar.google.com/scholar?q=Barbara+Landau%2C+Linda+B%2E+Smith%2C+and+Susan+S%2E+Jones%2E+1988%2E+The+importance+of+shape+in+early+lexical+learning%2E+Cognitive+Development%2C+3%283%29%3A299%E2%80%93321%2E)
13. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. [Rationalizing neural predictions](https://md.research.elamur.ai/free/en/papers/lei-2016-rationalizing-neural-predictions/). In EMNLP, pages 107–117.
14. Jiwei Li, Will Monroe, and Daniel Jurafsky. 2016. Understanding neural networks through representation erasure. CoRR, abs/1612.08220. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Daniel+Jurafsky%2E+2016%2E+Understanding+neural+networks+through+representation+erasure%2E+CoRR%2C+abs%2F1612%2E08220%2E)
15. Alexander H. Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017. [Parlai: A dialog research software platform](https://md.research.elamur.ai/free/en/papers/miller-2017-parlai-dialog-research-software-platform/). CoRR, abs/1705.06476.
16. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS-W. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+pytorch%2E+In+NIPS%2DW%2E)
17. Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. 2018. [Film: Visual reasoning with a general conditioning layer](https://md.research.elamur.ai/free/en/papers/perez-2017-film-visual-reasoning-general-conditioning/). In AAAI, pages 3942–3951.
18. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In ACL, pages 856–865. [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2018%2E+Semantically+equivalent+adversarial+rules+for+debugging+NLP+models%2E+In+ACL%2C+pages+856%E2%80%93865%2E)
19. M. Schuster and K. Nakajima. 2012. Japanese and korean voice search. In ICASSP, pages 5149–5152. [find](https://scholar.google.com/scholar?q=M%2E+Schuster+and+K%2E+Nakajima%2E+2012%2E+Japanese+and+korean+voice+search%2E+In+ICASSP%2C+pages+5149%E2%80%935152%2E)
20. Chenglei Si. 2019. Bert for multiple choice machine comprehension. [find](https://scholar.google.com/scholar?q=Chenglei+Si%2E+2019%2E+Bert+for+multiple+choice+machine+comprehension%2E)
21. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. [What makes reading comprehension questions easier?](https://md.research.elamur.ai/free/en/papers/sugawara-2018-what-makes-reading-comprehension-questions/) In EMNLP, pages 4208– 4219.
22. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. Dream: A challenge dataset and models for dialogue-based reading comprehension. TACL, 7:217–231. [find](https://scholar.google.com/scholar?q=Kai+Sun%2C+Dian+Yu%2C+Jianshu+Chen%2C+Dong+Yu%2C+Yejin+Choi%2C+and+Claire+Cardie%2E+2019%2E+Dream%3A+A+challenge+dataset+and+models+for+dialogue%2Dbased+reading+comprehension%2E+TACL%2C+7%3A217%E2%80%93231%2E)
23. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. [Intriguing properties of neural networks](https://md.research.elamur.ai/free/en/papers/szegedy-2013-intriguing-properties-neural-networks/). CoRR, abs/1312.6199.
24. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [FEVER: a large-scale dataset for fact extraction and VERification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In NAACL, pages 809–819.
25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, NeurIPS, pages 5998–6008. Curran Asso…
26. Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, Dan Roth, and David McAllester. 2018a. Evidence extraction for machine reading comprehension with deep probabilistic logic. CoRR, abs/1902.08852. — [arXiv:1902.08852](https://arxiv.org/abs/1902.08852) [find](https://scholar.google.com/scholar?q=Hai+Wang%2C+Dian+Yu%2C+Kai+Sun%2C+Jianshu+Chen%2C+Dong+Yu%2C+Dan+Roth%2C+and+David+McAllester%2E+2018a%2E+Evidence+extraction+for+machine+reading+comprehension+with+deep+probabilistic+logic%2E+CoRR%2C+abs%2F1902%2E08852%2E+%E2%80%94+%5B)
27. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018b. Evidence aggregation for answer re-ranking in open-domain question answering. In ICLR. [find](https://scholar.google.com/scholar?q=Shuohang+Wang%2C+Mo+Yu%2C+Jing+Jiang%2C+Wei+Zhang%2C+Xiaoxiao+Guo%2C+Shiyu+Chang%2C+Zhiguo+Wang%2C+Tim+Klinger%2C+Gerald+Tesauro%2C+and+Murray+Campbell%2E+2018b%2E+Evidence+aggregation+for+answer+re%2Dranking+in+open%2Ddomain+)
28. Mo Yu, Shiyu Chang, and Tommi S Jaakkola. 2019. Learning corresponded rationales for text matching. [find](https://scholar.google.com/scholar?q=Mo+Yu%2C+Shiyu+Chang%2C+and+Tommi+S+Jaakkola%2E+2019%2E+Learning+corresponded+rationales+for+text+matching%2E)
29. Christian S. Perone, Roberto Silveira, and Thomas S. Paula. 2018. Evaluation of sentence embeddings in downstream and linguistic probing tasks. CoRR, abs/1806.06259. — [arXiv:1806.06259](https://arxiv.org/abs/1806.06259) [find](https://scholar.google.com/scholar?q=Christian+S%2E+Perone%2C+Roberto+Silveira%2C+and+Thomas+S%2E+Paula%2E+2018%2E+Evaluation+of+sentence+embeddings+in+downstream+and+linguistic+probing+tasks%2E+CoRR%2C+abs%2F1806%2E06259%2E+%E2%80%94+%5BarXiv%3A1806%2E06259%5D%28https%3A%2F%2Farxiv)
30. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
31. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In EMNLP, pages 2383–2392.

*Updated 31 October 2026*