---
title: "WT5?! Training Text-to-Text Models to Explain their Predictions"
description: "The authors present WT5 (\"Why, T5?\"). 20 references, cited by 100 papers here. Among them: 99."
authors: ["Sharan Narang", "Colin Raffel", "Katherine Lee", "Adam Roberts", "Noah Fiedel", "Karishma Malkan"]
year: 2020
venue: "arXiv:2004.14546"
source: "https://arxiv.org/abs/2004.14546"
doi: "10.48550/arxiv.2004.14546"
arxiv: "2004.14546"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/narang-2020-wt5-training-text-text-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/narang-2020-wt5-training-text-text-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / WT5?! Training Text-to-Text Models to Explain their Predictions

# WT5?! Training Text-to-Text Models to Explain their Predictions

**Authors:** [Sharan Narang](https://md.research.elamur.ai/free/en/authors/sharan-narang/), [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/), [Katherine Lee](https://md.research.elamur.ai/free/en/authors/katherine-lee/), [Adam Roberts](https://md.research.elamur.ai/free/en/authors/adam-roberts/), [Noah Fiedel](https://md.research.elamur.ai/free/en/authors/noah-fiedel/) and Karishma Malkan · **Year:** 2020 · **Venue:** arXiv:2004.14546 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2004.14546) · [PDF](https://arxiv.org/pdf/2004.14546) · [DOI 10.48550/arxiv.2004.14546](https://doi.org/10.48550/arxiv.2004.14546) · [arXiv 2004.14546](https://arxiv.org/abs/2004.14546) · [OpenAlex](https://openalex.org/W3023690688)

**Citations:** 99 citations (OpenAlex, 2026-10-14) · [100](https://md.research.elamur.ai/free/en/papers/narang-2020-wt5-training-text-text-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large pre-trained neural models now match or beat human accuracy on many NLP tasks, but they reveal little about why they reach a given decision. The authors present WT5 ("Why, T5?"). It reuses the text-to-text format of T5 so the model writes a natural-language justification right after its label. The loss, training loop and decoding stay unchanged. The input gets the prefix "explain", and the target becomes the label followed by "explanation:" and the rationale. Dropping the prefix yields plain classification, so examples with and without rationales can be mixed in one training run. For extractive benchmarks such as ERASER, generated rationales are mapped back to spans of the input. The largest variant, WT5-11B, achieves state-of-the-art results on explainability benchmarks such as e-SNLI and CoS-E, often approaching human quality. It also learns from a small pool of annotated explanations and carries explanation skills over to new domains and even new tasks. The work offers a simple, general recipe for self-explaining models. However, the authors caution that the explanations need not reflect the model's actual causal reasoning.

## Key points
- Built on pre-trained T5 checkpoints (trained on Common Crawl text, SentencePiece subword tokens). The experiments use the "Base" model (about 220 million parameters) and the "11B" model (about 11 billion parameters).
- Datasets include e-SNLI (NLI with human-written rationales), CoS-E (commonsense multiple-choice QA), Movie Reviews and the extractive ERASER suite. The authors argue that tasks like the Winograd Schema Challenge (WSC) cannot be handled by span-only explanations.
- In a semi-supervised setup, the training data is fully labeled but only a fraction of examples carry explanations. Training uses both kinds of examples together.
- Explanation ability transferred across domains (Movie Reviews → Amazon Reviews). It also transferred zero-shot across tasks (e-SNLI → CoS-E). The cross-task transfer worked only after aligning input formats (e.g., recasting CoS-E fields as "nli"/"premise:" and listing the three NLI labels as choices), and beam search made explanation generation more reliable.
- Training code is released in the google-research GitHub repository under `wt5`.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel and Karishma Malkan. 2020. WT5?! Training Text-to-Text Models to Explain their Predictions. arXiv:2004.14546.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{narang2020wt5,
  title = {{WT5?! Training Text-to-Text Models to Explain their Predictions}},
  author = {Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel and Karishma Malkan},
  year = {2020},
  doi = {10.48550/arxiv.2004.14546},
  eprint = {2004.14546},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.14546}
}
```

- [← previous We Can Detect Your Bias: Predicting the Political Ideology of News Articles](https://md.research.elamur.ai/free/en/papers/baly-2020-we-can-detect-your-bias/)
- [next → Exploring and Predicting Transferability across NLP Tasks](https://md.research.elamur.ai/free/en/papers/vu-2020-exploring-predicting-transferability-across-nlp/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

20 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. with natural language explanations. In Advances in Neural Information Processing Systems, 2018. [find](https://scholar.google.com/scholar?q=with+natural+language+explanations%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2018%2E)
2. Jacobsen, J.-H., Smeulders, A., and Oyallon, E. i- Levesque, H., Davis, E., and Morgenstern, L. The revnet: Deep invertible networks. arXiv preprint winograd schema challenge. In Thirteenth Interna-arXiv:1802.07088, 2018. tional Conference on the Principles of Knowledge Representation and Reasoning,… — [arXiv:1802.07088](https://arxiv.org/abs/1802.07088) [find](https://scholar.google.com/scholar?q=Jacobsen%2C+J%2E%2DH%2E%2C+Smeulders%2C+A%2E%2C+and+Oyallon%2C+E%2E+i%2D+Levesque%2C+H%2E%2C+Davis%2C+E%2E%2C+and+Morgenstern%2C+L%2E+The+revnet%3A+Deep+invertible+networks%2E+arXiv+preprint+winograd+schema+challenge%2E+In+Thirteenth+Interna%2Dar)
3. Jain, S. and Wallace, B. C. Attention is not explana-Liu, X., He, P., Chen, W., and Gao, J. Multi-task deep tion. arXiv preprint arXiv:1902.10186, 2019. neural networks for natural language understanding. — [arXiv:1902.10186](https://arxiv.org/abs/1902.10186) [find](https://scholar.google.com/scholar?q=Jain%2C+S%2E+and+Wallace%2C+B%2E+C%2E+Attention+is+not+explana%2DLiu%2C+X%2E%2C+He%2C+P%2E%2C+Chen%2C+W%2E%2C+and+Gao%2C+J%2E+Multi%2Dtask+deep+tion%2E+arXiv+preprint+arXiv%3A1902%2E10186%2C+2019%2E+neural+networks+for+natural+language+understand)
4. Jia, R. and Liang, P. Adversarial examples for evaluat-arXiv preprint arXiv:1901.[11504, 2019](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/). ing reading comprehension systems. arXiv preprint Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., arXiv:1707.07328, 2017. Ng, A. Y., and Potts, C. Learning word vectors Kalchbrenner, N., Grefenstette, E.,…
5. Khashabi, D., Chaturvedi, S., Roth, M., Upad- McAuley, J., Targett, C., Shi, Q., and Van Den Hen-hyay, S., and Roth, D. Looking Beyond the Sur-gel, A. Image-based recommendations on styles face: A Challenge Set for Reading Comprehen-and substitutes. In Proceedings of the 38th Inter-sion over Multipl… — [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/)
6. Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer Pruthi, D., Gupta, M., Dhingra, B., Neubig, G., and Lipton, Z. C. Learning to deceive with and detokenizer for neural text processing. arXiv attention-based explanations. arXiv preprint preprint arXiv:1808… — [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
7. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. ALBERT: A lite bert for Raffel, C., Luong, M.-T., Liu, P. J., Weiss, R. J., and Eck, D. Online and linear-time attention by self-supervised learning of language representations. enforcing monotonic alignments. In Proceedings arX… — [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/)
8. Rajani, N. F., McCann, B., Xiong, C., and Socher, R. [Explain yourself! leveraging language models for commonsense reasoning](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/). arXiv preprint arXiv:1906.02361, 2019.
9. Ruder, S. [An overview of multi-task learning in deep neural networks](https://md.research.elamur.ai/free/en/papers/ruder-2017-overview-multi-task-learning-deep/). arXiv preprint arXiv:1706.05098, 2017.
10. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword Zhang, Q. and Zhu, S. Visual interpretability for units. arXiv preprint arXiv:1508.[07909, 2015](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). deep learning: a survey. Frontiers of Information Technology and Electronic Engineering, 19, 02 2018.
11. Serrano, S. and Smith, N. A. Is attention inter-doi: 10.1631/FITEE.1700808. pretable? arXiv preprint arXiv:1906.03731, 2019. — [doi:10.1631/fitee.1700808](https://doi.org/10.1631/fitee.1700808) [find](https://scholar.google.com/scholar?q=Serrano%2C+S%2E+and+Smith%2C+N%2E+A%2E+Is+attention+inter%2Ddoi%3A+10%2E1631%2FFITEE%2E1700808%2E+pretable%3F+arXiv+preprint+arXiv%3A1906%2E03731%2C+2019%2E+%E2%80%94+%5Bdoi%3A10%2E1631%2Ffitee%2E1700808%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1631%2Ffitee%2E1700808%29)
12. Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235, 2018. [find](https://scholar.google.com/scholar?q=Shazeer%2C+N%2E+and+Stern%2C+M%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1804%2E04235%2C+2018%2E)
13. Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017. — [arXiv:1706.03825](https://arxiv.org/abs/1706.03825) [find](https://scholar.google.com/scholar?q=Smilkov%2C+D%2E%2C+Thorat%2C+N%2E%2C+Kim%2C+B%2E%2C+Vie%CC%81gas%2C+F%2E%2C+and+Wattenberg%2C+M%2E+Smoothgrad%3A+removing+noise+by+adding+noise%2E+arXiv+preprint+arXiv%3A1706%2E03825%2C+2017%2E+%E2%80%94+%5BarXiv%3A1706%2E03825%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1706%2E038)
14. Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, 2017. [find](https://scholar.google.com/scholar?q=Sundararajan%2C+M%2E%2C+Taly%2C+A%2E%2C+and+Yan%2C+Q%2E+Axiomatic+attribution+for+deep+networks%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning%2C+2017%2E)
15. Sutskever, I., Vinyals, O., and Le, Q. V. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in neural information processing systems, 2014.
16. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, 2017.
17. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neur…
18. Williams, A., Nangia, N., and Bowman, S. R. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). arXiv preprint arXiv:1704.05426, 2017.
19. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. [Show, attend and tell: Neural image caption generation with visual attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). In International conference on machine learning, 2015.
20. Zaidan, O. and Eisner, J. Modeling annotators: A generative approach to learning from annotator rationales. In Proceedings of the 2008 conference on Empirical methods in natural language processing, pp. 31–40, 2008. [find](https://scholar.google.com/scholar?q=Zaidan%2C+O%2E+and+Eisner%2C+J%2E+Modeling+annotators%3A+A+generative+approach+to+learning+from+annotator+rationales%2E+In+Proceedings+of+the+2008+conference+on+Empirical+methods+in+natural+language+processing%2C+p)

*Updated 2 November 2026*