---
title: "Robustifying Multi-hop QA through — Lee 2021"
description: "Multi-hop QA models often get the right answer without real reasoning."
authors: ["Kyungjae Lee", "Seung-won Hwang", "Sang-eun Han", "Dohyeon Lee"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.476/"
doi: "10.18653/v1/2021.acl-long.476"
arxiv: "2107.03242"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/lee-2021-robustifying-multi-hop-qa-through/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/lee-2021-robustifying-multi-hop-qa-through/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2021](https://md.research.elamur.ai/free/papers/2021/page/3/) / Robustifying Multi-hop QA through Pseudo-Evidentiality Training

# Robustifying Multi-hop QA through Pseudo-Evidentiality Training

**Authors:** [Kyungjae Lee](https://md.research.elamur.ai/free/authors/kyungjae-lee/), Seung-won Hwang, Sang-eun Han, Dohyeon Lee · **Year:** 2021 · **Venue:** Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.476/) · [PDF](https://aclanthology.org/2021.acl-long.476.pdf) · [DOI 10.18653/v1/2021.acl-long.476](https://doi.org/10.18653/v1/2021.acl-long.476) · [arXiv 2107.03242](https://arxiv.org/abs/2107.03242) · [OpenAlex](https://openalex.org/W3175291199)

**Citations:** 8 citations (openalex-snapshot · 2026-10-14) · [11](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Multi-hop QA models often get the right answer without real reasoning. They exploit shortcuts such as the expected entity type instead of linking all the needed facts. The authors want to penalise such unsupported answers without paying for annotated reasoning chains. Working with the HotpotQA distractor setting and a RoBERTa reader, they add two training signals. The first is answerability. The second is evidentiality: whether the passage actually contains the evidence that justifies the answer.

Evidence-negative examples are created by stripping or diluting context around the answer sentence. Evidence-positive examples come from an Interpreter module that scores sentences by inserting and erasing them and measuring the resulting change in answer confidence. These pseudo-labels feed a selective regularization scheme. It trains a deliberately biased branch and decorrelates the main model from it, so the model stays confident when evidence is complete and becomes cautious when it is not.

The approach improves QA over baselines on both the original and challenge sets of HotpotQA. It can also be stacked on a stronger existing model. This matters because it makes multi-hop readers more robust at no extra annotation cost.

## Key points
- Data: HotpotQA (112k questions). Each question has 10 candidate paragraphs, 2 relevant and 8 distractors. Evaluation uses word-level answer F1, plus a challenge set testing disconnected reasoning.
- Three kinds of evidence-negative inputs are built automatically:
  - the answer sentence alone;
  - the answer sentence plus distractor text;
  - one relevant paragraph plus one distractor paragraph.
- Pseudo-evidence quality, measured as sentence-level evidence selection against human labels:
  - The Interpreter on the full model reaches F1 69.35 and recall 86.59.
  - The variant trained with plain regularization R reaches F1 70.30 and recall 87.10.
  - The unsupervised retrieval baseline AIR gets F1 66.16 but the best precision, 63.06.
- Ablations show performance drops when evidence-positive data or evidentiality labels are removed, or when R̂ is replaced by plain R.
- A dev-set confidence analysis uses about 7K question–passage pairs per set. It shows the full model raises confidence on evidence-positive inputs and lowers it on answer-sentence-only inputs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 10 not yet.

- [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://md.research.elamur.ai/free/papers/asai-2022-evidentiality-guided-generation-knowledge-intensive/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lee2021robustifying,
  title = {{Robustifying Multi-hop QA through Pseudo-Evidentiality Training}},
  author = {Lee, Kyungjae and Hwang, Seung-won and Han, Sang-eun and Lee, Dohyeon},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.476},
  eprint = {2107.03242},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.acl-long.476}
}
```

- [← previous Revisiting Shallow Discourse Parsing in the PDTB-3: Handling Intra-sentential Implicits](https://md.research.elamur.ai/free/papers/zhao-2021-revisiting-shallow-discourse-parsing-pdtb/)
- [next → RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/papers/qu-2021-rocketqa-optimized-training-approach-dense/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/papers/2021/page/3/)

## References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2019. Learning to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Akari+Asai%2C+Kazuma+Hashimoto%2C+Hannaneh+Hajishirzi%2C+Richard+Socher%2C+and+Caiming+Xiong%2E+2019%2E+Learning+to+retrieve+reasoning+paths+over+wikipedia+graph+for+question+answering%2E+In+International+Conferenc) [our card](https://md.research.elamur.ai/free/papers/asai-2019-learning-retrieve-reasoning-paths-over/)
2. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Matthew+E+Peters%2C+and+Arman+Cohan%2E+2020%2E+Longformer%3A+The+long%2Ddocument+transformer%2E+arXiv+preprint+arXiv%3A2004%2E05150%2E)
3. Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jifan+Chen+and+Greg+Durrett%2E+2019%2E+Understanding+dataset+design+choices+for+multi%2Dhop+reasoning%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computationa)
4. Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2020. Hierarchical graph network for multi-hop question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8823–8838. [find](https://scholar.google.com/scholar?q=Yuwei+Fang%2C+Siqi+Sun%2C+Zhe+Gan%2C+Rohit+Pillai%2C+Shuohang+Wang%2C+and+Jingjing+Liu%2E+2020%2E+Hierarchical+graph+network+for+multi%2Dhop+question+answering%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Meth)
5. Gabriel Grand and Yonatan Belinkov. 2019. Adversarial regularization for visual question answering: Strengths, shortcomings, and side effects. In Proceedings of the Second Workshop on Shortcomings in Vision and Language, pages 1–13. [find](https://scholar.google.com/scholar?q=Gabriel+Grand+and+Yonatan+Belinkov%2E+2019%2E+Adversarial+regularization+for+visual+question+answering%3A+Strengths%2C+shortcomings%2C+and+side+effects%2E+In+Proceedings+of+the+Second+Workshop+on+Shortcomings+in+)
6. Dirk Groeneveld, Tushar Khot, Ashish Sabharwal, et al. 2020. A simple yet strong pipeline for hotpotqa. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8839–8845. [find](https://scholar.google.com/scholar?q=Dirk+Groeneveld%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+et+al%2E+2020%2E+A+simple+yet+strong+pipeline+for+hotpotqa%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP)
7. Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556. [find](https://scholar.google.com/scholar?q=Sarthak+Jain+and+Byron+C+Wallace%2E+2019%2E+Attention+is+not+explanation%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Langua) [our card](https://md.research.elamur.ai/free/papers/jain-2019-attention-is-not-explanation/)
8. Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop qa. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2726– 2736. [find](https://scholar.google.com/scholar?q=Yichen+Jiang+and+Mohit+Bansal%2E+2019%2E+Avoiding+reasoning+shortcuts%3A+Adversarial+evaluation%2C+training%2C+and+model+development+for+multi%2Dhop+qa%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associatio)
9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
10. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4249–4257. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Eric+Wallace%2C+Sameer+Singh%2C+Matt+Gardner%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2019%2E+Compositional+questions+do+not+necessitate+multi%2Dhop+reasoning%2E+In+Proceedings+of+the+57th+Annual+)
11. Yixin Nie, Songhe Wang, and Mohit Bansal. 2019. Revealing the importance of semantic retrieval for machine reading at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN… [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Songhe+Wang%2C+and+Mohit+Bansal%2E+2019%2E+Revealing+the+importance+of+semantic+retrieval+for+machine+reading+at+scale%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Langu)
12. Yilin Niu, Fangkai Jiao, Mantong Zhou, Ting Yao, Minlie Huang, et al. 2020. A self-training method for machine reading comprehension with soft evidence extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3916–3927. [find](https://scholar.google.com/scholar?q=Yilin+Niu%2C+Fangkai+Jiao%2C+Mantong+Zhou%2C+Ting+Yao%2C+Minlie+Huang%2C+et+al%2E+2020%2E+A+self%2Dtraining+method+for+machine+reading+comprehension+with+soft+evidence+extraction%2E+In+Proceedings+of+the+58th+Annual+Me)
13. Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, and Kyunghyun Cho. 2019. Finding generalizable evidence by learning to convince q\\&a models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference… [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Siddharth+Karamcheti%2C+Rob+Fergus%2C+Jason+Weston%2C+Douwe+Kiela%2C+and+Kyunghyun+Cho%2E+2019%2E+Finding+generalizable+evidence+by+learning+to+convince+q%5C%5C%26a+models%2E+In+Proceedings+of+the+2019+Confe)
14. Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. 2019. Dynamically fused graph network for multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6140– 6150. [find](https://scholar.google.com/scholar?q=Lin+Qiu%2C+Yunxuan+Xiao%2C+Yanru+Qu%2C+Hao+Zhou%2C+Lei+Li%2C+Weinan+Zhang%2C+and+Yong+Yu%2E+2019%2E+Dynamically+fused+graph+network+for+multi%2Dhop+reasoning%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associatio)
15. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+squad%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/papers/rajpurkar-2018-know-what-you-don-t/)
16. Sofia Serrano and Noah A Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931–2951. [find](https://scholar.google.com/scholar?q=Sofia+Serrano+and+Noah+A+Smith%2E+2019%2E+Is+attention+interpretable%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+2931%E2%80%932951%2E) [our card](https://md.research.elamur.ai/free/papers/serrano-2019-is-attention-interpretable/)
17. Krishna Kumar Singh, Dhruv Mahajan, Kristen Grau-man, Yong Jae Lee, Matt Feiszli, and Deepti Ghadi-yaram. 2020. Don’t judge an object by its context: Learning to overcome contextual bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11070–11078. [find](https://scholar.google.com/scholar?q=Krishna+Kumar+Singh%2C+Dhruv+Mahajan%2C+Kristen+Grau%2Dman%2C+Yong+Jae+Lee%2C+Matt+Feiszli%2C+and+Deepti+Ghadi%2Dyaram%2E+2020%2E+Don%E2%80%99t+judge+an+object+by+its+context%3A+Learning+to+overcome+contextual+bias%2E+In+Proceedin)
18. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2020. Is multihop qa in dire condition? measuring and reducing disconnected reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8846–8863. [find](https://scholar.google.com/scholar?q=Harsh+Trivedi%2C+Niranjan+Balasubramanian%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2020%2E+Is+multihop+qa+in+dire+condition%3F+measuring+and+reducing+disconnected+reasoning%2E+In+Proceedings+of+the+2020+Conference)
19. Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621–633. [find](https://scholar.google.com/scholar?q=Lifu+Tu%2C+Garima+Lalwani%2C+Spandana+Gella%2C+and+He+He%2E+2020%2E+An+empirical+study+on+robustness+to+spurious+correlations+using+pre%2Dtrained+language+models%2E+Transactions+of+the+Association+for+Computational)
20. Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020. Mind the trade-off: Debiasing nlu models without degrading the in-distribution performance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8717–8729. [find](https://scholar.google.com/scholar?q=Prasetya+Ajie+Utama%2C+Nafise+Sadat+Moosavi%2C+and+Iryna+Gurevych%2E+2020%2E+Mind+the+trade%2Doff%3A+Debiasing+nlu+models+without+degrading+the+in%2Ddistribution+performance%2E+In+Proceedings+of+the+58th+Annual+Meeti)
21. Vikas Yadav, Steven Bethard, and Mihai Surdeanu. 2020. Unsupervised alignment-based iterative evidence retrieval for multi-hop question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4514– 4525. [find](https://scholar.google.com/scholar?q=Vikas+Yadav%2C+Steven+Bethard%2C+and+Mihai+Surdeanu%2E+2020%2E+Unsupervised+alignment%2Dbased+iterative+evidence+retrieval+for+multi%2Dhop+question+answering%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Asso)
22. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D+Manning%2E+2018%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E) [our card](https://md.research.elamur.ai/free/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
23. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2369%E2%80%932380%2E)