---
title: "Reframing Human-AI Collaboration for Generating Free-Text"
description: "This work tests whether few-shot prompting of GPT-3 can replace that supervision. 64 references, cited by 104 papers here."
authors: ["Sarah Wiegreffe", "Jack Hessel", "Swabha Swayamdipta", "Mark Riedl", "Yejin Choi"]
year: 2021
venue: "arXiv:2112.08674"
source: "https://arxiv.org/abs/2112.08674"
doi: "10.48550/arxiv.2112.08674"
arxiv: "2112.08674"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wiegreffe-2021-reframing-human-ai-collaboration-generating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wiegreffe-2021-reframing-human-ai-collaboration-generating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Reframing Human-AI Collaboration for Generating Free-Text Explanations

# Reframing Human-AI Collaboration for Generating Free-Text Explanations

**Authors:** [Sarah Wiegreffe](https://md.research.elamur.ai/free/en/authors/sarah-wiegreffe/), [Jack Hessel](https://md.research.elamur.ai/free/en/authors/jack-hessel/), [Swabha Swayamdipta](https://md.research.elamur.ai/free/en/authors/swabha-swayamdipta/), [Mark Riedl](https://md.research.elamur.ai/free/en/authors/mark-riedl/) and [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) · **Year:** 2021 · **Venue:** arXiv:2112.08674 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2112.08674) · [PDF](https://arxiv.org/pdf/2112.08674) · [DOI 10.48550/arxiv.2112.08674](https://doi.org/10.48550/arxiv.2112.08674) · [arXiv 2112.08674](https://arxiv.org/abs/2112.08674) · [OpenAlex](https://openalex.org/W4226056775)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [104](https://md.research.elamur.ai/free/en/papers/wiegreffe-2021-reframing-human-ai-collaboration-generating/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Free-text explanations for NLP decisions are usually produced by models trained on crowdsourced explanation corpora. Collecting these corpora is costly, and the resulting data is often inconsistent in quality. This work tests whether few-shot prompting of GPT-3 can replace that supervision. The authors evaluate on CommonsenseQA (with CoS-E and ECQA explanations) and SNLI (with e-SNLI). In head-to-head crowdworker studies, GPT-3 explanations prompted with carefully author-written examples were frequently judged better than the existing crowd-written ones. Raters still found weaknesses: the generated texts often repeat the input and only weakly justify the label. To address this, the paper changes what annotators are asked to do. Instead of writing explanations, they give yes/no acceptability ratings on several GPT-3 candidates per instance. These ratings train a T5/T0-based filter that picks the best candidate. This overgenerate-and-filter pipeline beats greedy decoding and strong baselines on human acceptability, novelty, and label support. The result suggests cheap binary judgments could replace expensive explanation authoring, although a clear gap to the oracle remains, especially for SNLI.

## Key points
- Prompts were built from 115 training instances, with 8–24 demonstrations per prompt. GPT-3 Davinci (~175B parameters, non-instruct) was conditioned on the gold label. Evaluation used fixed 250-instance test sets per task.
- When GPT-3 was prompted with each dataset's own crowd explanations, raters preferred its outputs over CoS-E (44.8% vs 20.3%). They still preferred ECQA and e-SNLI over GPT-3. With higher-quality prompts written by the authors, GPT-3 was preferred over CoS-E (78.9% vs 7.2%) and roughly matched ECQA (45.7% vs 44.5%).
- Under greedy decoding, fewer than half of GPT-3 explanations were rated acceptable by all annotators. Sampling 5 candidates per instance usually yields at least one unanimously acceptable explanation.
- The acceptability filter can be trained on only ~5k binary crowd labels. Agreement among annotators, measured with Krippendorff's α, was low to moderate, which reflects how subjective the task is.
- Appendix C finds that GPT-3 explains non-predicted answers about as convincingly as its own predictions. For this reason, the authors advise against real-world deployment. Code and data are released at github.com/allenai/few_shot_explanations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 304 works · 106,137 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 152 works · 27,763 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl and Yejin Choi. 2021. Reframing Human-AI Collaboration for Generating Free-Text Explanations. arXiv:2112.08674.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wiegreffe2021reframing,
  title = {{Reframing Human-AI Collaboration for Generating Free-Text Explanations}},
  author = {Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl and Yejin Choi},
  year = {2021},
  doi = {10.48550/arxiv.2112.08674},
  eprint = {2112.08674},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2112.08674}
}
```

- [← previous Linear Transformers Are Secretly Fast Weight Programmers](https://md.research.elamur.ai/free/en/papers/schlag-2021-linear-transformers-secretly-fast-weight/)
- [next → What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus](https://md.research.elamur.ai/free/en/papers/luccioni-2021-what-s-box-analysis-undesirable/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

64 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021. [Explanations for CommonsenseQA: New Dataset and Models](https://md.research.elamur.ai/free/en/papers/aggarwal-2021-explanations-commonsenseqa-new-dataset-models/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confe…
2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati…
3. Faeze Brahman, Vered Shwartz, Rachel Rudinger, and Yejin Choi. 2021. Learning to rationalize for non-monotonic reasoning with distant supervision. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, Online. [find](https://scholar.google.com/scholar?q=Faeze+Brahman%2C+Vered+Shwartz%2C+Rachel+Rudinger%2C+and+Yejin+Choi%2E+2021%2E+Learning+to+rationalize+for+non%2Dmonotonic+reasoning+with+distant+supervision%2E+In+Proceedings+of+the+35th+AAAI+Conference+on+Artific)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. [e-snli: Natural language inference with natural language explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). In Advances in Neural Information Processing Systems, volume 31, Montreal, Canada. Curran Associates, Inc.
6. Francisco Javier Chiyah Garcia, David A. Robb, Xingkun Liu, Atanas Laskov, Pedro Patron, and Helen Hastie. 2018. Explainable autonomy: A study of explanation styles for building clear mental models. In Proceedings of the 11th International Conference on Natural Language Generation, pages 99– 108, Ti… [find](https://scholar.google.com/scholar?q=Francisco+Javier+Chiyah+Garcia%2C+David+A%2E+Robb%2C+Xingkun+Liu%2C+Atanas+Laskov%2C+Pedro+Patron%2C+and+Helen+Hastie%2E+2018%2E+Explainable+autonomy%3A+A+study+of+explanation+styles+for+building+clear+mental+models%2E+I)
7. Miruna-Adriana Clinciu, Arash Eshghi, and Helen Hastie. 2021. A study of automatic metrics for the evaluation of natural language explanations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2376–2387, Online. Associ… [find](https://scholar.google.com/scholar?q=Miruna%2DAdriana+Clinciu%2C+Arash+Eshghi%2C+and+Helen+Hastie%2E+2021%2E+A+study+of+automatic+metrics+for+the+evaluation+of+natural+language+explanations%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Ch)
8. Upol Ehsan, Brent Harrison, Larry Chan, and Mark O Riedl. 2018. Rationalization: A neural machine translation approach to generating natural language explanations. In Proceedings of the 1st AAAI/ACM Conference on AI, Ethics, and Society (AIES), New Orleans, USA. [find](https://scholar.google.com/scholar?q=Upol+Ehsan%2C+Brent+Harrison%2C+Larry+Chan%2C+and+Mark+O+Riedl%2E+2018%2E+Rationalization%3A+A+neural+machine+translation+approach+to+generating+natural+language+explanations%2E+In+Proceedings+of+the+1st+AAAI%2FACM+C)
9. Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O Riedl. 2019. Automated rationale generation: A technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI), pages 263–274, Los Angel… [find](https://scholar.google.com/scholar?q=Upol+Ehsan%2C+Pradyumna+Tambwekar%2C+Larry+Chan%2C+Brent+Harrison%2C+and+Mark+O+Riedl%2E+2019%2E+Automated+rationale+generation%3A+A+technique+for+explainable+ai+and+its+effects+on+human+perceptions%2E+In+Proceedings)
10. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf… [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Yoav+Goldberg%2C+and+Jonathan+Berant%2E+2019%2E+Are+we+modeling+the+task+or+the+annotator%3F+an+investigation+of+annotator+bias+in+natural+language+understanding+datasets%2E+In+Proceedings+of+the+2019)
11. Alison Gopnik. 1998. Explanation as orgasm. Minds and Machines, 8(1):101–118. [find](https://scholar.google.com/scholar?q=Alison+Gopnik%2E+1998%2E+Explanation+as+orgasm%2E+Minds+and+Machines%2C+8%281%29%3A101%E2%80%93118%2E)
12. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. [Annotation artifacts in natural language inference data](https://md.research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
13. Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. 2020. [Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language?](https://md.research.elamur.ai/free/en/papers/hase-2020-leakage-adjusted-simulatability-can-models/) In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4351–4367, Online. Association for Computa…
14. Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In Proceedings of the 14th European Conference on Computer Vision (ECCV), Amsterdam, Netherlands. [find](https://scholar.google.com/scholar?q=Lisa+Anne+Hendricks%2C+Zeynep+Akata%2C+Marcus+Rohrbach%2C+Jeff+Donahue%2C+Bernt+Schiele%2C+and+Trevor+Darrell%2E+2016%2E+Generating+visual+explanations%2E+In+Proceedings+of+the+14th+European+Conference+on+Computer+Vi)
15. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In Proceedings of the 8th International Conference on Learning Representations (ICLR), Online.
16. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. [Surface form competition: Why the highest probability answer isn’t always right](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, Online and Punta Cana, Dominic…
17. Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai. In Proceedings of the 4th ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 624–635, Online. [find](https://scholar.google.com/scholar?q=Alon+Jacovi%2C+Ana+Marasovic%CC%81%2C+Tim+Miller%2C+and+Yoav+Goldberg%2E+2021%2E+Formalizing+trust+in+artificial+intelligence%3A+Prerequisites%2C+causes+and+goals+of+human+trust+in+ai%2E+In+Proceedings+of+the+4th+ACM+Conf)
18. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. [How can we know what language models know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) Transactions of the Association for Computational Linguistics, 8:423–438.
19. Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. 2021. e-vil: A dataset and benchmark for natural language explanations in vision-language tasks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Onli… [find](https://scholar.google.com/scholar?q=Maxime+Kayser%2C+Oana%2DMaria+Camburu%2C+Leonard+Salewski%2C+Cornelius+Emde%2C+Virginie+Do%2C+Zeynep+Akata%2C+and+Thomas+Lukasiewicz%2E+2021%2E+e%2Dvil%3A+A+dataset+and+benchmark+for+natural+language+explanations+in+vision)
20. Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016. Examples are not enough, learn to criticize! criticism for interpretability. In Advances in Neural Information Processing Systems, volume 29, Barcelona, Spain. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Been+Kim%2C+Rajiv+Khanna%2C+and+Oluwasanmi+O+Koyejo%2E+2016%2E+Examples+are+not+enough%2C+learn+to+criticize%21+criticism+for+interpretability%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+29%2C+Bar)
21. Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John F. Canny, and Zeynep Akata. 2018. Textual Explanations for Self-Driving Vehicles. In Proceedings of the 15th European Conference on Computer Vision (ECCV), Munich, Germany. [find](https://scholar.google.com/scholar?q=Jinkyu+Kim%2C+Anna+Rohrbach%2C+Trevor+Darrell%2C+John+F%2E+Canny%2C+and+Zeynep+Akata%2E+2018%2E+Textual+Explanations+for+Self%2DDriving+Vehicles%2E+In+Proceedings+of+the+15th+European+Conference+on+Computer+Vision+%28ECC)
22. Diederik P. Kingma and Jimmy Ba. 2015. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, USA.
23. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2011%2E+Computing+krippendorff%E2%80%99s+alpha%2Dreliability%2E)
24. Vivian Lai, Samuel Carton, and Chenhao Tan. 2020. Harnessing explanations to bridge ai and humans. In Fair and Responsible AI Workshop at the CHI Conference on Human Factors in Computing Systems, Online. [find](https://scholar.google.com/scholar?q=Vivian+Lai%2C+Samuel+Carton%2C+and+Chenhao+Tan%2E+2020%2E+Harnessing+explanations+to+bridge+ai+and+humans%2E+In+Fair+and+Responsible+AI+Workshop+at+the+CHI+Conference+on+Human+Factors+in+Computing+Systems%2C+Onli)
25. Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. 2022. [Can language models learn from explanations in context?](https://md.research.elamur.ai/free/en/papers/lampinen-2022-can-language-models-learn-explanations/) ArXiv preprint arXiv:2204.02329.
26. Veronica Latcinnik and Jonathan Berant. 2020. Explaining question answering models through text generation. ArXiv preprint arXiv:2004.05569. — [arXiv:2004.05569](https://arxiv.org/abs/2004.05569) [find](https://scholar.google.com/scholar?q=Veronica+Latcinnik+and+Jonathan+Berant%2E+2020%2E+Explaining+question+answering+models+through+text+generation%2E+ArXiv+preprint+arXiv%3A2004%2E05569%2E+%E2%80%94+%5BarXiv%3A2004%2E05569%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2004%2E05569%29)
27. David B Leake. 1991. Goal-based explanation evaluation. Cognitive Science, 15(4):509–545. [find](https://scholar.google.com/scholar?q=David+B+Leake%2E+1991%2E+Goal%2Dbased+explanation+evaluation%2E+Cognitive+Science%2C+15%284%29%3A509%E2%80%93545%2E)
28. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. [Rationalizing neural predictions](https://md.research.elamur.ai/free/en/papers/lei-2016-rationalizing-neural-predictions/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107–117, Austin, Texas. Association for Computational Linguistics.
29. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry,… — [Datasets: A Community Library for Natural Language Processing](https://md.research.elamur.ai/free/en/papers/lhoest-2021-datasets-community-library-natural-language/)
30. Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022. [Wanli: Worker and ai collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/). ArXiv preprint arXiv:2201.05955.
31. Tania Lombrozo. 2007. Simplicity and probability in causal explanation. Cognitive Psychology, 55(3):232–257. [find](https://scholar.google.com/scholar?q=Tania+Lombrozo%2E+2007%2E+Simplicity+and+probability+in+causal+explanation%2E+Cognitive+Psychology%2C+55%283%29%3A232%E2%80%93257%2E)
32. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). ArXiv preprint arXiv:2104.08786.
33. Bodhisattwa Prasad Majumder, Oana-Maria Camburu, Thomas Lukasiewicz, and Julian McAuley. 2021. Rationale-inspired natural language explanations with commonsense. ArXiv preprint arXiv:2106.13876. — [arXiv:2106.13876](https://arxiv.org/abs/2106.13876) [find](https://scholar.google.com/scholar?q=Bodhisattwa+Prasad+Majumder%2C+Oana%2DMaria+Camburu%2C+Thomas+Lukasiewicz%2C+and+Julian+McAuley%2E+2021%2E+Rationale%2Dinspired+natural+language+explanations+with+commonsense%2E+ArXiv+preprint+arXiv%3A2106%2E13876%2E+%E2%80%94+%5Bar)
34. Ana Marasović, Iz Beltagy, Doug Downey, and Matthew E Peters. 2021. [Few-shot self-rationalization with natural language prompts](https://md.research.elamur.ai/free/en/papers/marasovic-2022-few-shot-self-rationalization-natural/). ArXiv preprint arXiv:2111.08284.
35. Luca Massarelli, Fabio Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras, Fabrizio Silvestri, and Sebastian Riedel. 2020. How decoding strategies affect the verifiability of generated text. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 223–2… [find](https://scholar.google.com/scholar?q=Luca+Massarelli%2C+Fabio+Petroni%2C+Aleksandra+Piktus%2C+Myle+Ott%2C+Tim+Rockt%C3%A4schel%2C+Vassilis+Plachouras%2C+Fabrizio+Silvestri%2C+and+Sebastian+Riedel%2E+2020%2E+How+decoding+strategies+affect+the+verifiability+of+g)
36. Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. [WT5?! training text-to-text models to explain their predictions](https://md.research.elamur.ai/free/en/papers/narang-2020-wt5-training-text-text-models/). ArXiv preprint arXiv:2004.14546.
37. Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021. Prompting contrastive explanations for commonsense reasoning tasks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4179–4192, Online. Association for Compu… [find](https://scholar.google.com/scholar?q=Bhargavi+Paranjape%2C+Julian+Michael%2C+Marjan+Ghazvininejad%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2021%2E+Prompting+contrastive+explanations+for+commonsense+reasoning+tasks%2E+In+Findings+of+the+Associ)
38. Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018. Multimodal explanations: Justifying decisions and pointing to the evidence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8779–8… [find](https://scholar.google.com/scholar?q=Dong+Huk+Park%2C+Lisa+Anne+Hendricks%2C+Zeynep+Akata%2C+Anna+Rohrbach%2C+Bernt+Schiele%2C+Trevor+Darrell%2C+and+Marcus+Rohrbach%2E+2018%2E+Multimodal+explanations%3A+Justifying+decisions+and+pointing+to+the+evidence%2E+I)
39. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. [True few-shot learning with language models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/). In Advances in Neural Information Processing Systems, volume 34, pages 11054–11070, Online. Curran Associates, Inc.
40. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. [Hypothesis only baselines in natural language inference](https://md.research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/). In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Compu…
41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
42. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(140):1–67.
43. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. [Explain yourself! leveraging language models for commonsense reasoning](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4932–4942, Florence, Italy. Association for Computat…
44. Swarnadeep Saha, Prateek Yadav, Lisa Bauer, and Mohit Bansal. 2021. ExplaGraphs: An explanation graph generation task for structured commonsense reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7716–7740, Online and Punta Cana, Dominican Rep… [find](https://scholar.google.com/scholar?q=Swarnadeep+Saha%2C+Prateek+Yadav%2C+Lisa+Bauer%2C+and+Mohit+Bansal%2E+2021%2E+ExplaGraphs%3A+An+explanation+graph+generation+task+for+structured+commonsense+reasoning%2E+In+Proceedings+of+the+2021+Conference+on+Emp)
45. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. [Multitask prompted training enables zero-shot task generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/). In Proceedings of the 10th International Conference on Learning Repre…
46. Timo Schick and Hinrich Schütze. 2021a. [Exploiting cloze-questions for few-shot text classification and natural language inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Comp…
47. Timo Schick and Hinrich Schütze. 2021b. [It’s not just size that matters: Small language models are also few-shot learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Assoc…
48. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 4596–4604, Stockholm, Sweden. PMLR. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+Proceedings+of+the+35th+International+Conference+on+Machine+Learning+%28ICML%29%2C+pages+4596%E2%80%934604%2C+S)
49. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin…
50. Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4615–4629, Online. Association for Computational L… [find](https://scholar.google.com/scholar?q=Vered+Shwartz%2C+Peter+West%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Unsupervised+commonsense+question+answering+with+self%2Dtalk%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Meth)
51. Charles Spearman. 1987. The proof and measurement of association between two things. The American Journal of Psychology, 100(3/4):441–471. [find](https://scholar.google.com/scholar?q=Charles+Spearman%2E+1987%2E+The+proof+and+measurement+of+association+between+two+things%2E+The+American+Journal+of+Psychology%2C+100%283%2F4%29%3A441%E2%80%93471%2E)
52. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021, Online. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+2020%2E+Learning+to+summarize+from+human+feedback%2E+In+Advances+in+Neu)
53. Justin Sulik, Jeroen van Paridon, and Gary Lupyan. 2021. Explanations in the wild. [find](https://scholar.google.com/scholar?q=Justin+Sulik%2C+Jeroen+van+Paridon%2C+and+Gary+Lupyan%2E+2021%2E+Explanations+in+the+wild%2E)
54. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
55. Chenhao Tan. 2021. On the diversity and limits of human explanations. ArXiv preprint arXiv:2106.11988. [find](https://scholar.google.com/scholar?q=Chenhao+Tan%2E+2021%2E+On+the+diversity+and+limits+of+human+explanations%2E+ArXiv+preprint+arXiv%3A2106%2E11988%2E)
56. Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. 2020. [Scipy 1.0: Fundamental algorithms for scientific computing in python](https://md.research.elamur.ai/free/en/papers/virtanen-2019-scipy-1-0-fundamental-algorithms/). Nature Methods, 17(3):261–272.
57. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. [Symbolic knowledge distillation: from general language models to commonsense models](https://md.research.elamur.ai/free/en/papers/west-2022-symbolic-knowledge-distillation-general-language/). ArXiv preprint arXiv:2110.07178.
58. Sarah Wiegreffe and Ana Marasović. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, volume 34, Online. [find](https://scholar.google.com/scholar?q=Sarah+Wiegreffe+and+Ana+Marasovic%CC%81%2E+2021%2E+Teach+me+to+explain%3A+A+review+of+datasets+for+explainable+natural+language+processing%2E+In+Advances+in+Neural+Information+Processing+Systems+Datasets+and+Bench)
59. Sarah Wiegreffe, Ana Marasović, and Noah A. Smith. 2021. [Measuring association between labels and free-text rationales](https://md.research.elamur.ai/free/en/papers/wiegreffe-2021-measuring-association-between-labels-free/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10266–10284, Online and Punta Cana, Dominican Republic. Association for Computatio…
60. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
61. Yordan Yordanov, Vid Kocijan, Thomas Lukasiewicz, and Oana-Maria Camburu. 2021. Few-shot out-of-domain transfer learning of natural language explanations. In NeurIPS Workshop on Deep Generative Models and Downstream Applications. [find](https://scholar.google.com/scholar?q=Yordan+Yordanov%2C+Vid+Kocijan%2C+Thomas+Lukasiewicz%2C+and+Oana%2DMaria+Camburu%2E+2021%2E+Few%2Dshot+out%2Dof%2Ddomain+transfer+learning+of+natural+language+explanations%2E+In+NeurIPS+Workshop+on+Deep+Generative+Models)
62. Jeffrey C Zemla, Steven Sloman, Christos Bechlivanidis, and David A Lagnado. 2017. Evaluating everyday explanations. Psychonomic Bulletin & Review, 24(5):1488–1500. [find](https://scholar.google.com/scholar?q=Jeffrey+C+Zemla%2C+Steven+Sloman%2C+Christos+Bechlivanidis%2C+and+David+A+Lagnado%2E+2017%2E+Evaluating+everyday+explanations%2E+Psychonomic+Bulletin+%26+Review%2C+24%285%29%3A1488%E2%80%931500%2E)
63. Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In Proceedings of the 38th International Conference on Machine Learning (ICML), Online.
64. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences. ArXiv preprint arXiv:1909.08593. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2020%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+ArXiv+preprint+ar)

*Updated 2 November 2026*