---
title: "QED — paper by Lamm et al. (2021) — summary and ideas"
description: "The authors define QED, a linguistically motivated format for explaining extractive answers. 31 references, cited by 48 papers here."
authors: ["Matthew Lamm", "Jennimaria Palomaki", "Chris Alberti", "Daniel Andor", "Eunsol Choi", "Livio Baldini Soares", "Michael Collins"]
year: 2021
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2021.tacl-1.48/"
doi: "10.1162/tacl_a_00398"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lamm-2021-qed-framework-dataset-explanations-question/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lamm-2021-qed-framework-dataset-explanations-question/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / QED: A Framework and Dataset for Explanations in Question Answering

# QED: A Framework and Dataset for Explanations in Question Answering

**Authors:** [Matthew Lamm](https://md.research.elamur.ai/free/en/authors/matthew-lamm/), [Jennimaria Palomaki](https://md.research.elamur.ai/free/en/authors/jennimaria-palomaki/), [Chris Alberti](https://md.research.elamur.ai/free/en/authors/chris-alberti/), [Daniel Andor](https://md.research.elamur.ai/free/en/authors/daniel-andor/), [Eunsol Choi](https://md.research.elamur.ai/free/en/authors/eunsol-choi/), [Livio Baldini Soares](https://md.research.elamur.ai/free/en/authors/livio-baldini-soares/) and [Michael Collins](https://md.research.elamur.ai/free/en/authors/michael-collins/) · **Year:** 2021 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2021](https://md.research.elamur.ai/free/en/venues/tacl/2021/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.tacl-1.48/) · [PDF](https://aclanthology.org/2021.tacl-1.48.pdf) · [DOI 10.1162/tacl\_a\_00398](https://doi.org/10.1162/tacl_a_00398) · [OpenAlex](https://openalex.org/W3085552234)

**Citations:** 43 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/lamm-2021-qed-framework-dataset-explanations-question/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Neural QA systems reach high accuracy but rarely show how they arrive at an answer, which limits trust and makes them harder to debug. The authors define QED, a linguistically motivated format for explaining extractive answers. An explanation consists of three parts. First, it picks one passage sentence that implies the answer. Second, it links noun phrases in the question and that sentence that denote the same entity or concept, including coreference and bridging cases. Third, it confirms that, once those shared referents are abstracted out, the sentence's remaining predicate entails the question's predicate. Experts applied this scheme to a subset of Google Natural Questions, and the resulting corpus is public. The paper reports baselines for two tasks: explaining an answer that is already given, and producing an answer and explanation together. In the joint setting, a modest amount of QED training data appears to improve answer accuracy. A large user study found that showing QED explanations helps non-expert raters catch mistakes made by a strong neural QA model.

## Key points
- The dataset is built on Natural Questions and contains 7,638 training and 1,353 dev examples. Three annotators labelled them without replication, spending about 2 minutes per item.
- About 10% of the examined NQ short answers turned out to be wrong. Among the valid ones, 77% could receive a single-sentence QED explanation.
- Explanations rest on referential equality, sentencehood, and predicate entailment. Bridging anaphora are handled through an implicit-modifier notation.
- Two baseline tasks are evaluated: post-hoc explanation generation and joint QA plus explanation.
- The resource is released at github.com/google-research-datasets/QED. The authors outline extensions to multi-hop, yes/no, and set-valued questions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 310 works · 284,480 citations
- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 149 works · 48,852 citations
- [The University of Texas at Austin](https://md.research.elamur.ai/free/en/universities/the-university-of-texas-at-austin/) — University, Austin, United States · 72 works · 55,722 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 242 works

## Cite

Cite as: Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares and Michael Collins. 2021. QED: A Framework and Dataset for Explanations in Question Answering. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{lamm2021qed,
  title = {{QED: A Framework and Dataset for Explanations in Question Answering}},
  author = {Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares and Michael Collins},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00398},
  url = {https://doi.org/10.1162/tacl_a_00398}
}
```

- [← previous Frequency Effects on Syntactic Rule Learning in Transformers](https://md.research.elamur.ai/free/en/papers/wei-2021-frequency-effects-syntactic-rule-learning/)
- [next → BERT Busters: Outlier Dimensions that Disrupt Transformers](https://md.research.elamur.ai/free/en/papers/kovaleva-2021-bert-busters-outlier-dimensions-that/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

31 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Herbert H. Clark and Catherine R. Marshall. 1981. Definite reference and mutual knowledge. Elements of Discourse Understanding. [find](https://scholar.google.com/scholar?q=Herbert+H%2E+Clark+and+Catherine+R%2E+Marshall%2E+1981%2E+Definite+reference+and+mutual+knowledge%2E+Elements+of+Discourse+Understanding%2E)
2. Sarthak Jain and Byron C. Wallace. 2019. [Attention is not explanation](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, Minneapolis, Minnesota. Associa…
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
4. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. [SpanBERT: Improving pre-training by representing and predicting spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/). Transactions of the Association for Computational Linguistics, 8:64–77. https://doi.org/10 .1162/tacl a 00300
5. Finale Doshi-Velez and Been Kim. 2017. [Towards a rigorous science of interpretable machine learning](https://md.research.elamur.ai/free/en/papers/doshi-velez-2017-towards-rigorous-science-interpretable-machine/). arXiv preprint arXiv:1702.08608.
6. Manfred Krifka. 2003. Bare NPS: Kind-referring, indefinites, both, or neither? Semantics and Linguistic Theory, 13:180–203. [find](https://scholar.google.com/scholar?q=Manfred+Krifka%2E+2003%2E+Bare+NPS%3A+Kind%2Dreferring%2C+indefinites%2C+both%2C+or+neither%3F+Semantics+and+Linguistic+Theory%2C+13%3A180%E2%80%93203%2E)
7. Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O. Riedl. 2019. Automated rationale generation: A technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 263–274. ACM. https://d… — [doi:10.1145/3301275](https://doi.org/10.1145/3301275) [find](https://scholar.google.com/scholar?q=Upol+Ehsan%2C+Pradyumna+Tambwekar%2C+Larry+Chan%2C+Brent+Harrison%2C+and+Mark+O%2E+Riedl%2E+2019%2E+Automated+rationale+generation%3A+A+technique+for+explainable+ai+and+its+effects+on+human+perceptions%2E+In+Proceeding)
8. Sawan Kumar and Partha Talukdar. 2020. [Nile: Natural language inference with faithful natural language explanations](https://md.research.elamur.ai/free/en/papers/kumar-2020-nile-natural-language-inference-faithful/). https://doi.org/10.3765/salt.v13i0.2880 [↗](https://doi.org/10.3765/salt.v13i0.2880)
9. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
10. Andrew Gelman and Jennifer Hill. 2006. Data Analysis Using Regression and Multilevel/Hierarchical Models, Cambridge University Press. https://doi.org/10.1017/CBO9780 511790942 — [doi:10.1017/cbo9780](https://doi.org/10.1017/cbo9780) [↗](https://doi.org/10.1017/CBO9780)
11. Ben Goodrich, Jonah Gabry, Imad Ali, and Sam Brilleman. 2020. rstanarm: Bayesian Applied Regression Modeling via Stan. R package version 2.19.3. [find](https://scholar.google.com/scholar?q=Ben+Goodrich%2C+Jonah+Gabry%2C+Imad+Ali%2C+and+Sam+Brilleman%2E+2020%2E+rstanarm%3A+Bayesian+Applied+Regression+Modeling+via+Stan%2E+R+package+version+2%2E19%2E3%2E)
12. Matthew Lamm, Arun Chaganty, Christopher D. Manning, Dan Jurafsky, and Percy Liang. 2018. Textual analogy parsing: What’s shared and what’s compared among analogous facts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 82–92, Brussels, Belgium. Assoc… [find](https://scholar.google.com/scholar?q=Matthew+Lamm%2C+Arun+Chaganty%2C+Christopher+D%2E+Manning%2C+Dan+Jurafsky%2C+and+Percy+Liang%2E+2018%2E+Textual+analogy+parsing%3A+What%E2%80%99s+shared+and+what%E2%80%99s+compared+among+analogous+facts%2E+In+Proceedings+of+the+2018+C)
13. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics. https://doi.org/10.3115/992133.992154 [↗](https://doi.org/10.3115/992133.992154)
14. Alon Jacovi and Yoav Goldberg. 2020. [Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?](https://md.research.elamur.ai/free/en/papers/jacovi-2020-faithfully-interpretable-nlp-systems-how/) In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198–4205, Online. Association for Computational Linguistics. https://d…
15. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end ution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Luheng+He%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2017%2E+End%2Dto%2Dend+ution%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+188%E2%80%93197%2C+Copenhagen%2C+D)
16. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ Questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/D16-1264 [↗](https://doi.org/10.18653/v1/D16-1264)
17. Peter Lipton. 2001, What good is an explanation?, Giora Hon and Sam S. Rakover, editors, Explanation: Theoretical Approaches and Applications, Springer Netherlands, Dordrecht, pages 43–59. https://doi.org/10 .1007/978-94-015-9731-9 2 [find](https://scholar.google.com/scholar?q=Peter+Lipton%2E+2001%2C+What+good+is+an+explanation%3F%2C+Giora+Hon+and+Sam+S%2E+Rakover%2C+editors%2C+Explanation%3A+Theoretical+Approaches+and+Applications%2C+Springer+Netherlands%2C+Dordrecht%2C+pages+43%E2%80%9359%2E+https%3A%2F%2Fdoi)
18. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. [CoQA: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266. https://doi.org/10.1162/tacl a 00266 [↗](https://doi.org/10.1162/tacl)
19. Line Mikkelsen. 2011. Copular clauses. In Claudia Maienborn, Klaus von Heusinger, and Paul Portner, editors, Semantics: An International Handbook of Natural Language Meaning, 2, pages 1805–1829, Berlin. Mouton De Gruyter. [find](https://scholar.google.com/scholar?q=Line+Mikkelsen%2E+2011%2E+Copular+clauses%2E+In+Claudia+Maienborn%2C+Klaus+von+Heusinger%2C+and+Paul+Portner%2C+editors%2C+Semantics%3A+An+International+Handbook+of+Natural+Language+Meaning%2C+2%2C+pages+1805%E2%80%931829%2C+Berli)
20. Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. 2020. Getting closer to AI complete question answering: A set of prerequisite real tasks. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8722–8731. https://doi.org/10.1609/aaai.v34i05.6398 — [doi:10.1609/aaai.v34i05.6398](https://doi.org/10.1609/aaai.v34i05.6398) [↗](https://doi.org/10.1609/aaai.v34i05.6398)
21. Eleni Miltsakaki, Rashmi Prasad, Aravind K. Joshi, and Bonnie L. Webber. 2004. [The Penn Discourse Treebank](https://md.research.elamur.ai/free/en/papers/miltsakaki-2004-penn-discourse-treebank/). In LREC.
22. Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M. Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2020. Towards transparent and explainable attention models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4206–4216, Online… [find](https://scholar.google.com/scholar?q=Akash+Kumar+Mohankumar%2C+Preksha+Nema%2C+Sharan+Narasimhan%2C+Mitesh+M%2E+Khapra%2C+Balaji+Vasan+Srinivasan%2C+and+Balaraman+Ravindran%2E+2020%2E+Towards+transparent+and+explainable+attention+models%2E+In+Proceedings+)
23. Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2662–2670. [find](https://scholar.google.com/scholar?q=Andrew+Slavin+Ross%2C+Michael+C%2E+Hughes%2C+and+Finale+Doshi%2DVelez%2E+2017%2E+Right+for+the+right+reasons%3A+Training+differentiable+models+by+constraining+their+explanations%2E+In+Proceedings+of+the+Twenty%2DSixth+)
24. Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. Wt5?! Training text-to-text models to explain their predictions. [find](https://scholar.google.com/scholar?q=Sharan+Narang%2C+Colin+Raffel%2C+Katherine+Lee%2C+Adam+Roberts%2C+Noah+Fiedel%2C+and+Karishma+Malkan%2E+2020%2E+Wt5%3F%21+Training+text%2Dto%2Dtext+models+to+explain+their+predictions%2E)
25. Bertrand Russell. 1905. On denoting. Mind, 14(56):479–493. https://doi.org/10 .1093/mind/XIV.4.479 [find](https://scholar.google.com/scholar?q=Bertrand+Russell%2E+1905%2E+On+denoting%2E+Mind%2C+14%2856%29%3A479%E2%80%93493%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E1093%2Fmind%2FXIV%2E4%2E479)
26. Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019. WIQA: A dataset for ‘‘What if...’’ reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… [find](https://scholar.google.com/scholar?q=Niket+Tandon%2C+Bhavana+Dalvi%2C+Keisuke+Sakaguchi%2C+Peter+Clark%2C+and+Antoine+Bosselut%2E+2019%2E+WIQA%3A+A+dataset+for+%E2%80%98%E2%80%98What+if%2E%2E%2E%E2%80%99%E2%80%99+reasoning+over+procedural+text%2E+In+Proceedings+of+the+2019+Conference+on+Emp)
27. Judea Pearl. 2019. The limitations of opaque learning machines. Possible Minds: Twenty-Five Ways of Looking at AI13–19. [find](https://scholar.google.com/scholar?q=Judea+Pearl%2E+2019%2E+The+limitations+of+opaque+learning+machines%2E+Possible+Minds%3A+Twenty%2DFive+Ways+of+Looking+at+AI13%E2%80%9319%2E)
28. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling multilingual unrestricted coreference in Onto-Notes. In EMNLP-CoNLL Shared Task. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+CoNLL%2D2012+Shared+Task%3A+Modeling+multilingual+unrestricted+coreference+in+Onto%2DNotes%2E+In+EMNLP%2DCoNLL+Shared+Ta)
29. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
30. Michael Tomasello, Malinda Carpenter, and Ulf Liszkowski. 2007. A new look at infant pointing. Child Development, 78(3):705–722. https://doi.org/10.1111/j.1467 -8624.2007.01025.x, PubMed: 17516997 — [doi:10.1111/j.1467](https://doi.org/10.1111/j.1467) [↗](https://doi.org/10.1111/j.1467)
31. We now describe the form of the e and a annotations, making reference to the following example. Subscripts indicate token positions: Sarah Wiegreffe and Yuval Pinter. 2019. [Attention is not not explanation](https://md.research.elamur.ai/free/en/papers/wiegreffe-2019-attention-is-not-not-explanation/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and…

*Updated 24 October 2026*