---
title: "Towards Faithfully Interpretable NLP Systems — Jacovi 2020"
description: "This ACL 2020 position paper by Jacovi and Goldberg addresses a conceptual gap in NLP interpretability."
authors: ["Alon Jacovi", "Yoav Goldberg"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.386/"
doi: "10.18653/v1/2020.acl-main.386"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jacovi-2020-faithfully-interpretable-nlp-systems-how/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jacovi-2020-faithfully-interpretable-nlp-systems-how/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?

# Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?

**Authors:** [Alon Jacovi](https://md.research.elamur.ai/free/en/authors/alon-jacovi/), [Yoav Goldberg](https://md.research.elamur.ai/free/en/authors/yoav-goldberg/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.386/) · [PDF](https://aclanthology.org/2020.acl-main.386.pdf) · [DOI 10.18653/v1/2020.acl-main.386](https://doi.org/10.18653/v1/2020.acl-main.386) · [OpenAlex](https://openalex.org/W3015575765)

**Citations:** 480 citations (openalex-snapshot · 2026-10-14) · [267](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This ACL 2020 position paper by Jacovi and Goldberg addresses a conceptual gap in NLP interpretability. The field has no consistent, formal notion of a faithful explanation, meaning one that accurately reflects how a model actually reached its prediction. The paper introduces no dataset or model. Instead, the authors survey existing evaluation practice and argue that faithfulness is routinely confused with plausibility, which is how convincing an explanation looks to people.

They make three contributions. First, they distill the community's implicit working definitions into three explicit assumptions. Second, they give practical guidelines for faithfulness evaluation. Third, they criticise the habit of treating faithfulness as a yes/no property. Any interpretation is a lossy approximation of a highly non-linear model, so counter-examples will almost always exist, and showing that a method is "not faithful" teaches little. The authors therefore propose graded faithfulness, assessed per model or task, or per region of the input space.

The work matters because it gives researchers a shared vocabulary. It also warns against evaluation setups that quietly measure plausibility rather than faithfulness.

## Key points
- The paper offers five guidelines for faithfulness evaluation:
  - State explicitly which property is being evaluated.
  - Avoid human judgement of explanation quality.
  - Avoid reliance on gold labels.
  - Treat "inherently interpretable" claims, such as attention, as unverified.
  - Do not use end-user task performance in IUI/HCI settings as a proxy for faithfulness.
- Current approaches are organised around three assumptions. Two examples:
  - The Model Assumption links identical predictions to identical reasoning.
  - The Linearity Assumption holds that input parts carry independent importance, which is what justifies heat-map explanations.
- Heat-maps can be stress-tested in several ways:
  - Erasure removes the most or least relevant tokens and checks whether the prediction changes.
  - The comprehensiveness and sufficiency metrics, drawn from work by DeYoung et al. (2019) and Yu et al. (2019), give this idea a formal footing.
- A thought experiment shows the risk of utility-based evaluation. Explanations that are random yet look plausible only when the model is correct can still improve user performance.
- Strictly faithful interpretation is called a likely "unicorn". The authors set graded faithfulness criteria as an open challenge for the community.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

267 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 253 not yet.

- [A Diagnostic Study of Explainability Techniques for Text Classification](https://md.research.elamur.ai/free/en/papers/atanasova-2020-diagnostic-study-explainability-techniques-text/)
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://md.research.elamur.ai/free/en/papers/deyoung-2020-eraser-benchmark-evaluate-rationalized-nlp/)
- [A Survey on Automated Fact-Checking](https://md.research.elamur.ai/free/en/papers/guo-2022-survey-automated-fact-checking/)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://md.research.elamur.ai/free/en/papers/han-2020-explaining-black-box-predictions-unveiling/)
- [Human Rationales as Attribution Priors for Explainable Stance Detection](https://md.research.elamur.ai/free/en/papers/jayaram-2021-human-rationales-as-attribution-priors/)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/lanham-2023-measuring-faithfulness-chain-thought-reasoning/)
- [Faithful Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/lyu-2023-faithful-chain-thought-reasoning/)
- [Teaching language models to support answers with verified quotes](https://md.research.elamur.ai/free/en/papers/menick-2022-teaching-language-models-support-answers/)
- [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://md.research.elamur.ai/free/en/papers/schuster-2021-get-your-vitamin-c-robust/)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://md.research.elamur.ai/free/en/papers/turpin-2023-language-models-don-t-always/)
- [Measuring Association Between Labels and Free-Text Rationales](https://md.research.elamur.ai/free/en/papers/wiegreffe-2021-measuring-association-between-labels-free/)
- [STaR: Bootstrapping Reasoning With Reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/)
- [Explainability for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{jacovi2020faithfully,
  title = {{Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?}},
  author = {Jacovi, Alon and Goldberg, Yoav},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.386},
  url = {https://doi.org/10.18653/v1/2020.acl-main.386}
}
```

- [← previous Towards Emotion-aided Multi-modal Dialogue Act Classification](https://md.research.elamur.ai/free/en/papers/saha-2020-emotion-aided-multi-modal-dialogue/)
- [next → Toxic, Hateful, Offensive or Abusive? What Are We Really Classifying? An Empirical Analysis of Hate Speech Datasets](https://md.research.elamur.ai/free/en/papers/fortuna-2020-toxic-hateful-offensive-or-abusive/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

49 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ashraf M. Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, and Mohan S. Kankanhalli. 2018. Trends and trajectories for explainable, accountable and intelligible systems: An HCI research agenda. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, Montreal, QC, C… [find](https://scholar.google.com/scholar?q=Ashraf+M%2E+Abdul%2C+Jo+Vermeulen%2C+Danding+Wang%2C+Brian+Y%2E+Lim%2C+and+Mohan+S%2E+Kankanhalli%2E+2018%2E+Trends+and+trajectories+for+explainable%2C+accountable+and+intelligible+systems%3A+An+HCI+research+agenda%2E+In+Pro)
2. David Alvarez-Melis and Tommi S. Jaakkola. 2018. On the robustness of interpretability methods. CoRR, abs/1806.08049. — [arXiv:1806.08049](https://arxiv.org/abs/1806.08049) [find](https://scholar.google.com/scholar?q=David+Alvarez%2DMelis+and+Tommi+S%2E+Jaakkola%2E+2018%2E+On+the+robustness+of+interpretability+methods%2E+CoRR%2C+abs%2F1806%2E08049%2E+%E2%80%94+%5BarXiv%3A1806%2E08049%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1806%2E08049%29)
3. Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2016. ”what is relevant in a text document?”: An interpretable machine learning approach. CoRR, abs/1612.07843. — [arXiv:1612.07843](https://arxiv.org/abs/1612.07843) [find](https://scholar.google.com/scholar?q=Leila+Arras%2C+Franziska+Horn%2C+Gre%CC%81goire+Montavon%2C+Klaus%2DRobert+Mu%CC%88ller%2C+and+Wojciech+Samek%2E+2016%2E+%E2%80%9Dwhat+is+relevant+in+a+text+document%3F%E2%80%9D%3A+An+interpretable+machine+learning+approach%2E+CoRR%2C+abs%2F1612%2E0784)
4. Joris Baan, Maartje ter Hoeve, Marlies van der Wees, Anne Schuth, and Maarten de Rijke. 2019. Do transformer attention heads provide transparency in abstractive summarization? CoRR, abs/1907.00570. — [arXiv:1907.00570](https://arxiv.org/abs/1907.00570) [find](https://scholar.google.com/scholar?q=Joris+Baan%2C+Maartje+ter+Hoeve%2C+Marlies+van+der+Wees%2C+Anne+Schuth%2C+and+Maarten+de+Rijke%2E+2019%2E+Do+transformer+attention+heads+provide+transparency+in+abstractive+summarization%3F+CoRR%2C+abs%2F1907%2E00570%2E+%E2%80%94+)
5. Przemyslaw Biecek. 2018. DALEX: explainers for complex predictive models in R. J. Mach. Learn. Res., 19:84:1–84:5. [find](https://scholar.google.com/scholar?q=Przemyslaw+Biecek%2E+2018%2E+DALEX%3A+explainers+for+complex+predictive+models+in+R%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+19%3A84%3A1%E2%80%9384%3A5%2E)
6. Oana-Maria Camburu, Eleonora Giunchiglia, Jakob Foerster, Thomas Lukasiewicz, and Phil Blunsom. 2019. Can i trust the explainer? verifying post-hoc explanatory methods. [find](https://scholar.google.com/scholar?q=Oana%2DMaria+Camburu%2C+Eleonora+Giunchiglia%2C+Jakob+Foerster%2C+Thomas+Lukasiewicz%2C+and+Phil+Blunsom%2E+2019%2E+Can+i+trust+the+explainer%3F+verifying+post%2Dhoc+explanatory+methods%2E)
7. Vicente Iván Sánchez Carmona, Tim Rocktäschel, Sebastian Riedel, and Sameer Singh. 2015. Towards extracting faithful and descriptive representations of latent variable models. In AAAI Spring Symposia. [find](https://scholar.google.com/scholar?q=Vicente+Iva%CC%81n+Sa%CC%81nchez+Carmona%2C+Tim+Rockta%CC%88schel%2C+Sebastian+Riedel%2C+and+Sameer+Singh%2E+2015%2E+Towards+extracting+faithful+and+descriptive+representations+of+latent+variable+models%2E+In+AAAI+Spring+Sympos)
8. Supriyo Chakraborty, Richard Tomsett, Ramya Raghavendra, Daniel Harborne, Moustafa Alzantot, Federico Cerutti, Mani Srivastava, Alun Preece, Simon Julier, Raghuveer M Rao, et al. 2017. Interpretability of deep learning models: a survey of results. In 2017 IEEE SmartWorld, Ubiquitous Intelligence & C… [find](https://scholar.google.com/scholar?q=Supriyo+Chakraborty%2C+Richard+Tomsett%2C+Ramya+Raghavendra%2C+Daniel+Harborne%2C+Moustafa+Alzantot%2C+Federico+Cerutti%2C+Mani+Srivastava%2C+Alun+Preece%2C+Simon+Julier%2C+Raghuveer+M+Rao%2C+et+al%2E+2017%2E+Interpretabilit)
9. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2019. Eraser: A benchmark to evaluate rationalized nlp models. [find](https://scholar.google.com/scholar?q=Jay+DeYoung%2C+Sarthak+Jain%2C+Nazneen+Fatema+Rajani%2C+Eric+Lehman%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Byron+C%2E+Wallace%2E+2019%2E+Eraser%3A+A+benchmark+to+evaluate+rationalized+nlp+models%2E) [our card](https://md.research.elamur.ai/free/en/papers/deyoung-2020-eraser-benchmark-evaluate-rationalized-nlp/)
10. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608. [find](https://scholar.google.com/scholar?q=Finale+Doshi%2DVelez+and+Been+Kim%2E+2017%2E+Towards+a+rigorous+science+of+interpretable+machine+learning%2E+arXiv+preprint+arXiv%3A1702%2E08608%2E) [our card](https://md.research.elamur.ai/free/en/papers/doshi-2017-velez-towards-rigorous-science-interpretable-machine/)
11. Shi Feng and Jordan Boyd-Graber. 2019. What can ai do for me? evaluating machine learning interpretations in cooperative play. In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI 19, page 229239, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Shi+Feng+and+Jordan+Boyd%2DGraber%2E+2019%2E+What+can+ai+do+for+me%3F+evaluating+machine+learning+interpretations+in+cooperative+play%2E+In+Proceedings+of+the+24th+International+Conference+on+Intelligent+User+I)
12. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan L. Boyd-Graber. 2018. Pathologies of neural models make interpretation difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4,… [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+Alvin+Grissom+II%2C+Mohit+Iyyer%2C+Pedro+Rodriguez%2C+and+Jordan+L%2E+Boyd%2DGraber%2E+2018%2E+Pathologies+of+neural+models+make+interpretation+difficult%2E+In+Proceedings+of+the+2018+Conferen)
13. Karën Fort and Alain Couillault. 2016. Yes, we care! results of the ethics and natural language processing surveys. In LREC. [find](https://scholar.google.com/scholar?q=Kare%CC%88n+Fort+and+Alain+Couillault%2E+2016%2E+Yes%2C+we+care%21+results+of+the+ethics+and+natural+language+processing+surveys%2E+In+LREC%2E)
14. Reza Ghaeini, Xiaoli Z. Fern, and Prasad Tadepalli. 2018. Interpreting recurrent and attention-based neural models: a case study on natural language inference. CoRR, abs/1808.03894. [find](https://scholar.google.com/scholar?q=Reza+Ghaeini%2C+Xiaoli+Z%2E+Fern%2C+and+Prasad+Tadepalli%2E+2018%2E+Interpreting+recurrent+and+attention%2Dbased+neural+models%3A+a+case+study+on+natural+language+inference%2E+CoRR%2C+abs%2F1808%2E03894%2E)
15. Amirata Ghorbani, Abubakar Abid, and James Zou. 2019. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3681–3688. [find](https://scholar.google.com/scholar?q=Amirata+Ghorbani%2C+Abubakar+Abid%2C+and+James+Zou%2E+2019%2E+Interpretation+of+neural+networks+is+fragile%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+33%2C+pages+3681%E2%80%933688%2E)
16. Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pages 80–89. IEEE. [find](https://scholar.google.com/scholar?q=Leilani+H+Gilpin%2C+David+Bau%2C+Ben+Z+Yuan%2C+Ayesha+Bajwa%2C+Michael+Specter%2C+and+Lalana+Kagal%2E+2018%2E+Explaining+explanations%3A+An+overview+of+interpretability+of+machine+learning%2E+In+2018+IEEE+5th+Internati)
17. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM Comput. Surv., 51(5):93:1–93:42. [find](https://scholar.google.com/scholar?q=Riccardo+Guidotti%2C+Anna+Monreale%2C+Salvatore+Ruggieri%2C+Franco+Turini%2C+Fosca+Giannotti%2C+and+Dino+Pedreschi%2E+2018%2E+A+survey+of+methods+for+explaining+black+box+models%2E+ACM+Comput%2E+Surv%2E%2C+51%285%29%3A93%3A1%E2%80%9393%3A42)
18. L.A. Harrington, M.D. Morley, A. Šcedrov, and S.G. Simpson. 1985. Harvey Friedman’s Research on the Foundations of Mathematics. Studies in Logic and the Foundations of Mathematics. Elsevier Science. [find](https://scholar.google.com/scholar?q=L%2EA%2E+Harrington%2C+M%2ED%2E+Morley%2C+A%2E+S%CC%8Ccedrov%2C+and+S%2EG%2E+Simpson%2E+1985%2E+Harvey+Friedman%E2%80%99s+Research+on+the+Foundations+of+Mathematics%2E+Studies+in+Logic+and+the+Foundations+of+Mathematics%2E+Elsevier+Science%2E)
19. Bernease Herman. 2017. The promise and peril of human evaluation for model interpretability. CoRR, abs/1711.07414. Withdrawn. — [arXiv:1711.07414](https://arxiv.org/abs/1711.07414) [find](https://scholar.google.com/scholar?q=Bernease+Herman%2E+2017%2E+The+promise+and+peril+of+human+evaluation+for+model+interpretability%2E+CoRR%2C+abs%2F1711%2E07414%2E+Withdrawn%2E+%E2%80%94+%5BarXiv%3A1711%2E07414%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1711%2E07414%29)
20. Alon Jacovi, Oren Sar Shalom, and Yoav Goldberg. 2018. Understanding convolutional neural networks for text classification. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 56–65. [find](https://scholar.google.com/scholar?q=Alon+Jacovi%2C+Oren+Sar+Shalom%2C+and+Yoav+Goldberg%2E+2018%2E+Understanding+convolutional+neural+networks+for+text+classification%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+and+Interpr)
21. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Pap… [find](https://scholar.google.com/scholar?q=Sarthak+Jain+and+Byron+C%2E+Wallace%2E+2019%2E+Attention+is+not+explanation%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Langu) [our card](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/)
22. Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna M. Wallach, and Jennifer Wortman Vaughan. 2019. Interpreting interpretability: Understanding data scientists use of interpretability tools for machine learning. [find](https://scholar.google.com/scholar?q=Harmanpreet+Kaur%2C+Harsha+Nori%2C+Samuel+Jenkins%2C+Rich+Caruana%2C+Hanna+M%2E+Wallach%2C+and+Jennifer+Wortman+Vaughan%2E+2019%2E+Interpreting+interpretability%3A+Understanding+data+scientists+use+of+interpretability+)
23. Technologies, Volume 1 (Long Papers), pages 1069– 1078, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Technologies%2C+Volume+1+%28Long+Papers%29%2C+pages+1069%E2%80%93+1078%2C+New+Orleans%2C+Louisiana%2E+Association+for+Computational+Linguistics%2E)
24. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2017. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). [find](https://scholar.google.com/scholar?q=Been+Kim%2C+Martin+Wattenberg%2C+Justin+Gilmer%2C+Carrie+Cai%2C+James+Wexler%2C+Fernanda+Viegas%2C+and+Rory+Sayres%2E+2017%2E+Interpretability+beyond+feature+attribution%3A+Quantitative+testing+with+concept+activation+)
25. Nina Pörner, Hinrich Schütze, and Benjamin Roth. 2018. Evaluating neural network explanation methods using hybrid documents and morphological prediction. CoRR, abs/1801.06422. — [arXiv:1801.06422](https://arxiv.org/abs/1801.06422) [find](https://scholar.google.com/scholar?q=Nina+Po%CC%88rner%2C+Hinrich+Schu%CC%88tze%2C+and+Benjamin+Roth%2E+2018%2E+Evaluating+neural+network+explanation+methods+using+hybrid+documents+and+morphological+prediction%2E+CoRR%2C+abs%2F1801%2E06422%2E+%E2%80%94+%5BarXiv%3A1801%2E06422%5D%28h)
26. Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. 2019. The (un)reliability of saliency methods. In Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller, editors, Explainable AI… [find](https://scholar.google.com/scholar?q=Pieter%2DJan+Kindermans%2C+Sara+Hooker%2C+Julius+Adebayo%2C+Maximilian+Alber%2C+Kristof+T%2E+Schu%CC%88tt%2C+Sven+Da%CC%88hne%2C+Dumitru+Erhan%2C+and+Been+Kim%2E+2019%2E+The+%28un%29reliability+of+saliency+methods%2E+In+Wojciech+Samek%2C+Gr)
27. Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2019. An evaluation of the human-interpretability of explanation. CoRR, abs/1902.00006. — [arXiv:1902.00006](https://arxiv.org/abs/1902.00006) [find](https://scholar.google.com/scholar?q=Isaac+Lage%2C+Emily+Chen%2C+Jeffrey+He%2C+Menaka+Narayanan%2C+Been+Kim%2C+Sam+Gershman%2C+and+Finale+Doshi%2DVelez%2E+2019%2E+An+evaluation+of+the+human%2Dinterpretability+of+explanation%2E+CoRR%2C+abs%2F1902%2E00006%2E+%E2%80%94+%5BarXiv%3A1)
28. Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. 2019. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2019, Honolulu, HI, USA, January 27-28, 2019, pages 131–138. ACM. [find](https://scholar.google.com/scholar?q=Himabindu+Lakkaraju%2C+Ece+Kamar%2C+Rich+Caruana%2C+and+Jure+Leskovec%2E+2019%2E+Faithful+and+customizable+explanations+of+black+box+models%2E+In+Proceedings+of+the+2019+AAAI%2FACM+Conference+on+AI%2C+Ethics%2C+and+Soc)
29. Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive visualization and manipulation of attention-based neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 121–126, Copenhagen, Denmark. Associa… [find](https://scholar.google.com/scholar?q=Jaesong+Lee%2C+Joong%2DHwi+Shin%2C+and+Jun%2DSeok+Kim%2E+2017%2E+Interactive+visualization+and+manipulation+of+attention%2Dbased+neural+machine+translation%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Method)
30. Zachary C. Lipton. 2018. The mythos of model interpretability. Commun. ACM, 61(10):36–43. [find](https://scholar.google.com/scholar?q=Zachary+C%2E+Lipton%2E+2018%2E+The+mythos+of+model+interpretability%2E+Commun%2E+ACM%2C+61%2810%29%3A36%E2%80%9343%2E)
31. Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 4765–4774. [find](https://scholar.google.com/scholar?q=Scott+M%2E+Lundberg+and+Su%2DIn+Lee%2E+2017%2E+A+unified+approach+to+interpreting+model+predictions%2E+In+Advances+in+Neural+Information+Processing+Systems+30%3A+Annual+Conference+on+Neural+Information+Processing) [our card](https://md.research.elamur.ai/free/en/papers/lundberg-2017-unified-approach-interpreting-model-predictions/)
32. Sina Mohseni and Eric D. Ragan. 2018. A human-grounded evaluation benchmark for local explanations of machine learning. CoRR, abs/1801.05075. — [arXiv:1801.05075](https://arxiv.org/abs/1801.05075) [find](https://scholar.google.com/scholar?q=Sina+Mohseni+and+Eric+D%2E+Ragan%2E+2018%2E+A+human%2Dgrounded+evaluation+benchmark+for+local+explanations+of+machine+learning%2E+CoRR%2C+abs%2F1801%2E05075%2E+%E2%80%94+%5BarXiv%3A1801%2E05075%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1801%2E05075%29)
33. W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. 2019. Interpretable machine learning: definitions, methods, and applications. ArXiv, abs/1901.04592. — [arXiv:1901.04592](https://arxiv.org/abs/1901.04592) [find](https://scholar.google.com/scholar?q=W%2E+James+Murdoch%2C+Chandan+Singh%2C+Karl+Kumbier%2C+Reza+Abbasi%2DAsl%2C+and+Bin+Yu%2E+2019%2E+Interpretable+machine+learning%3A+definitions%2C+methods%2C+and+applications%2E+ArXiv%2C+abs%2F1901%2E04592%2E+%E2%80%94+%5BarXiv%3A1901%2E04592%5D%28ht)
34. Dong Nguyen. 2018. Comparing automatic and human evaluation of local explanations for text classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, a… [find](https://scholar.google.com/scholar?q=Dong+Nguyen%2E+2018%2E+Comparing+automatic+and+human+evaluation+of+local+explanations+for+text+classification%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Co)
35. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. CoRR, abs/1906.02361. [find](https://scholar.google.com/scholar?q=Nazneen+Fatema+Rajani%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2019%2E+Explain+yourself%21+leveraging+language+models+for+commonsense+reasoning%2E+CoRR%2C+abs%2F1906%2E02361%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/)
36. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “Why Should I Trust You?”: Explaining the predictions of any classifier. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 1135–1144, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2016%2E+%E2%80%9CWhy+Should+I+Trust+You%3F%E2%80%9D%3A+Explaining+the+predictions+of+any+classifier%2E+In+Proceedings+of+the+22Nd+ACM+SIGKDD+International+Conference+on) [our card](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/)
37. Cynthia Rudin. 2018. Please stop explaining black box models for high stakes decisions. CoRR, abs/1811.10154. [find](https://scholar.google.com/scholar?q=Cynthia+Rudin%2E+2018%2E+Please+stop+explaining+black+box+models+for+high+stakes+decisions%2E+CoRR%2C+abs%2F1811%2E10154%2E) [our card](https://md.research.elamur.ai/free/en/papers/rudin-2018-stop-explaining-black-box/)
38. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2931–2951. [find](https://scholar.google.com/scholar?q=Sofia+Serrano+and+Noah+A%2E+Smith%2E+2019%2E+Is+attention+interpretable%3F+In+Proceedings+of+the+57th+Conference+of+the+Association+for+Computational+Linguistics%2C+ACL+2019%2C+Florence%2C+Italy%2C+July+28%2D+August+2%2C) [our card](https://md.research.elamur.ai/free/en/papers/serrano-2019-is-attention-interpretable/)
39. Julia Strout, Ye Zhang, and Raymond J. Mooney. 2019. Do human rationales improve machine explanations? CoRR, abs/1905.13714. [find](https://scholar.google.com/scholar?q=Julia+Strout%2C+Ye+Zhang%2C+and+Raymond+J%2E+Mooney%2E+2019%2E+Do+human+rationales+improve+machine+explanations%3F+CoRR%2C+abs%2F1905%2E13714%2E)
40. Madhumita Sushil, Simon Šuster, and Walter Daelemans. 2018. Rule induction for global explanation of trained models. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 82–97, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Madhumita+Sushil%2C+Simon+S%CC%8Custer%2C+and+Walter+Daelemans%2E+2018%2E+Rule+induction+for+global+explanation+of+trained+models%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+and+Interpreting+)
41. Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019. Attention interpretability across NLP tasks. CoRR, abs/1909.11218. — [arXiv:1909.11218](https://arxiv.org/abs/1909.11218) [find](https://scholar.google.com/scholar?q=Shikhar+Vashishth%2C+Shyam+Upadhyay%2C+Gaurav+Singh+Tomar%2C+and+Manaal+Faruqui%2E+2019%2E+Attention+interpretability+across+NLP+tasks%2E+CoRR%2C+abs%2F1909%2E11218%2E+%E2%80%94+%5BarXiv%3A1909%2E11218%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1909%2E1121)
42. Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. CoRR, abs/1906.04284. [find](https://scholar.google.com/scholar?q=Jesse+Vig+and+Yonatan+Belinkov%2E+2019%2E+Analyzing+the+structure+of+attention+in+a+transformer+language+model%2E+CoRR%2C+abs%2F1906%2E04284%2E) [our card](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/)
43. Hilde J. P. Weerts, Werner van Ipenburg, and Mykola Pechenizkiy. 2019. A human-grounded evaluation of SHAP for alert processing. CoRR, abs/1907.03324. — [arXiv:1907.03324](https://arxiv.org/abs/1907.03324) [find](https://scholar.google.com/scholar?q=Hilde+J%2E+P%2E+Weerts%2C+Werner+van+Ipenburg%2C+and+Mykola+Pechenizkiy%2E+2019%2E+A+human%2Dgrounded+evaluation+of+SHAP+for+alert+processing%2E+CoRR%2C+abs%2F1907%2E03324%2E+%E2%80%94+%5BarXiv%3A1907%2E03324%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1907%2E0)
44. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, page… [find](https://scholar.google.com/scholar?q=Sarah+Wiegreffe+and+Yuval+Pinter%2E+2019%2E+Attention+is+not+not+explanation%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+International+Joint+Conf) [our card](https://md.research.elamur.ai/free/en/papers/wiegreffe-2019-attention-is-not-not-explanation/)
45. Lior Wolf, Tomer Galanti, and Tamir Hazan. 2019. A formal approach to explainability. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2019, Honolulu, HI, USA, January 27-28, 2019, pages 255–261. ACM. [find](https://scholar.google.com/scholar?q=Lior+Wolf%2C+Tomer+Galanti%2C+and+Tamir+Hazan%2E+2019%2E+A+formal+approach+to+explainability%2E+In+Proceedings+of+the+2019+AAAI%2FACM+Conference+on+AI%2C+Ethics%2C+and+Society%2C+AIES+2019%2C+Honolulu%2C+HI%2C+USA%2C+January+2)
46. Jialin Wu and Raymond J. Mooney. 2018. Faithful multimodal explanation for visual question answering. CoRR, abs/1809.02805. [find](https://scholar.google.com/scholar?q=Jialin+Wu+and+Raymond+J%2E+Mooney%2E+2018%2E+Faithful+multimodal+explanation+for+visual+question+answering%2E+CoRR%2C+abs%2F1809%2E02805%2E)
47. Wenting Xiong, Iftitahu Ni’mah, Juan M. G. Huesca, Werner van Ipenburg, Jan Veldsink, and Mykola Pechenizkiy. 2018. Looking deeper into deep learning model: Attribution-based explanations of textcnn. CoRR, abs/1811.03970. — [arXiv:1811.03970](https://arxiv.org/abs/1811.03970) [find](https://scholar.google.com/scholar?q=Wenting+Xiong%2C+Iftitahu+Ni%E2%80%99mah%2C+Juan+M%2E+G%2E+Huesca%2C+Werner+van+Ipenburg%2C+Jan+Veldsink%2C+and+Mykola+Pechenizkiy%2E+2018%2E+Looking+deeper+into+deep+learning+model%3A+Attribution%2Dbased+explanations+of+textcnn%2E+)
48. Mo Yu, Shiyu Chang, Yang Zhang, and Tommi S. Jaakkola. 2019. Rethinking cooperative rationalization: Introspective extraction and complement control. CoRR, abs/1910.13294. [find](https://scholar.google.com/scholar?q=Mo+Yu%2C+Shiyu+Chang%2C+Yang+Zhang%2C+and+Tommi+S%2E+Jaakkola%2E+2019%2E+Rethinking+cooperative+rationalization%3A+Introspective+extraction+and+complement+control%2E+CoRR%2C+abs%2F1910%2E13294%2E)
49. Omar Zaidan and Jason Eisner. 2008. Modeling annotators: A generative approach to learning from annotator rationales. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 31–40, Honolulu, Hawaii. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Omar+Zaidan+and+Jason+Eisner%2E+2008%2E+Modeling+annotators%3A+A+generative+approach+to+learning+from+annotator+rationales%2E+In+Proceedings+of+the+2008+Conference+on+Empirical+Methods+in+Natural+Language+Pro)