---
title: "NILE — paper by Kumar and Talukdar (2020)"
description: "NILE (Natural-language Inference over Label-specific Explanations) tackles a gap in explainable NLI. 28 references, cited by 72 papers here."
authors: ["Sawan Kumar", "Partha Talukdar"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.771/"
doi: "10.18653/v1/2020.acl-main.771"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kumar-2020-nile-natural-language-inference-faithful/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kumar-2020-nile-natural-language-inference-faithful/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / NILE : Natural Language Inference with Faithful Natural Language Explanations

# NILE : Natural Language Inference with Faithful Natural Language Explanations

**Authors:** Sawan Kumar and [Partha Talukdar](https://md.research.elamur.ai/free/en/authors/partha-talukdar/) · **Year:** 2020 · **Venue:** [Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2020](https://md.research.elamur.ai/free/en/venues/acl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.771/) · [PDF](https://aclanthology.org/2020.acl-main.771.pdf) · [DOI 10.18653/v1/2020.acl-main.771](https://doi.org/10.18653/v1/2020.acl-main.771) · [OpenAlex](https://openalex.org/W3027825046)

**Citations:** 110 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/kumar-2020-nile-natural-language-inference-faithful/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

NILE (Natural-language Inference over Label-specific Explanations) tackles a gap in explainable NLI. Existing systems produce free-text justifications, but it is hard to check whether those justifications actually drive the model's prediction, which is the property called faithfulness. NILE generates a candidate explanation for each possible label (entailment, contradiction, neutral) using label-specific GPT-2 generators fine-tuned on e-SNLI. A RoBERTa-based Explanation Processor then scores the candidates, optionally together with the premise–hypothesis pair, and chooses the label. Because the decision passes through explicit text, faithfulness can be tested by perturbing the explanations directly.

In automatic and human evaluation, NILE matches the label accuracy of earlier systems and yields better explanations. Its explanations also transfer better than the Explain-Then-Predict baseline to out-of-domain MNLI. The authors further show that generic erasure metrics can be misleading. A task-specific probe, which swaps in a random same-label explanation, shows that only NILE's predictions truly depend on its explanations. The work argues that faithfulness should be evaluated explicitly, alongside label and explanation accuracy.

## Key points

- Data: SNLI (about 570K pairs) extended with e-SNLI human explanations. After uninformative examples are removed, about 532K training examples remain.
- Models: gpt2-medium (about 345M parameters) serves as the per-label explanation generator, and roberta-base (about 125M) serves as the classifier. Processor variants include Independent, Aggregate and Append, plus NILE-PH and NILE-NS.
- On SNLI Dev, all variants reach roughly 91.2–91.7% accuracy. In MNLI transfer, three annotators judged explanations on 100 Dev samples, and NILE's explanations were more often correct than the baseline's.
- Shuffling probe: when explanations are swapped for random ones with the same label, NILE drops to about 31–35% accuracy, while NILE-NS keeps about 88–90%. This shows NILE-NS largely ignores the explanation content.
- Source code is released at github.com/SawanKumar28/nile.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Indian Institute of Science Bangalore](https://md.research.elamur.ai/free/en/universities/indian-institute-of-science-bangalore/) — University, Bengaluru, India · 8 works · 1,140 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Sawan Kumar and Partha Talukdar. 2020. NILE : Natural Language Inference with Faithful Natural Language Explanations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kumar2020nile,
  title = {{NILE : Natural Language Inference with Faithful Natural Language Explanations}},
  author = {Sawan Kumar and Partha Talukdar},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.771},
  url = {https://doi.org/10.18653/v1/2020.acl-main.771}
}
```

- [← previous The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT](https://md.research.elamur.ai/free/en/papers/tiedemann-2020-tatoeba-translation-challenge-realistic-data/)
- [next → Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://md.research.elamur.ai/free/en/papers/han-2020-explaining-black-box-predictions-unveiling/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Andreas, Dan Klein, and Sergey Levine. 2018. Learning with latent language. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2166–2179, New Orleans, Louisiana. Associ… [find](https://scholar.google.com/scholar?q=Jacob+Andreas%2C+Dan+Klein%2C+and+Sergey+Levine%2E+2018%2E+Learning+with+latent+language%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+)
2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
3. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. [e-SNLI: Natural Language Inference with Natural Language Explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). In Advances in Neural Information Processing Systems, pages 9539–9549.
4. Supriyo Chakraborty, Richard Tomsett, Ramya Raghavendra, Daniel Harborne, Moustafa Alzantot, Federico Cerutti, Mani Srivastava, Alun Preece, Simon Julier, Raghuveer M Rao, et al. 2017. Interpretability of Deep Learning Models: A Survey of Results. In 2017 IEEE SmartWorld, Ubiquitous Intelligence & C… [find](https://scholar.google.com/scholar?q=Supriyo+Chakraborty%2C+Richard+Tomsett%2C+Ramya+Raghavendra%2C+Daniel+Harborne%2C+Moustafa+Alzantot%2C+Federico+Cerutti%2C+Mani+Srivastava%2C+Alun+Preece%2C+Simon+Julier%2C+Raghuveer+M+Rao%2C+et+al%2E+2017%2E+Interpretabilit)
5. Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. 2018. Learning to Explain: An Information-Theoretic Perspective on Model Interpretation. In International Conference on Machine Learning, pages 882–891. [find](https://scholar.google.com/scholar?q=Jianbo+Chen%2C+Le+Song%2C+Martin+Wainwright%2C+and+Michael+Jordan%2E+2018%2E+Learning+to+Explain%3A+An+Information%2DTheoretic+Perspective+on+Model+Interpretation%2E+In+International+Conference+on+Machine+Learning%2C+p)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
7. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2019. [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://md.research.elamur.ai/free/en/papers/deyoung-2020-eraser-benchmark-evaluate-rationalized-nlp/). arXiv preprint arXiv:1911.03429.
8. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of Neural Models Make Interpretations Difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719–3728, Brussels, Belgium. Associatio… [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+Alvin+Grissom+II%2C+Mohit+Iyyer%2C+Pedro+Rodriguez%2C+and+Jordan+Boyd%2DGraber%2E+2018%2E+Pathologies+of+Neural+Models+Make+Interpretations+Difficult%2E+In+Proceedings+of+the+2018+Conference)
9. Amirata Ghorbani, Abubakar Abid, and James Zou. 2019. Interpretation of Neural Networks is Fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3681–3688. [find](https://scholar.google.com/scholar?q=Amirata+Ghorbani%2C+Abubakar+Abid%2C+and+James+Zou%2E+2019%2E+Interpretation+of+Neural+Networks+is+Fragile%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+33%2C+pages+3681%E2%80%933688%2E)
10. Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. 2018. Generating counterfactual explanations with natural language. In ICML Workshop on Human Interpretability in Machine Learning, pages 95–98. [find](https://scholar.google.com/scholar?q=Lisa+Anne+Hendricks%2C+Ronghang+Hu%2C+Trevor+Darrell%2C+and+Zeynep+Akata%2E+2018%2E+Generating+counterfactual+explanations+with+natural+language%2E+In+ICML+Workshop+on+Human+Interpretability+in+Machine+Learning%2C+)
11. Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018. Multimodal Explanations: Justifying Decisions and Pointing to the Evidence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8779– 8788. [find](https://scholar.google.com/scholar?q=Dong+Huk+Park%2C+Lisa+Anne+Hendricks%2C+Zeynep+Akata%2C+Anna+Rohrbach%2C+Bernt+Schiele%2C+Trevor+Darrell%2C+and+Marcus+Rohrbach%2E+2018%2E+Multimodal+Explanations%3A+Justifying+Decisions+and+Pointing+to+the+Evidence%2E+I)
12. Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. 2018. Textual Explanations for Self-Driving Vehicles. In Proceedings of the European conference on computer vision (ECCV), pages 563–578. [find](https://scholar.google.com/scholar?q=Jinkyu+Kim%2C+Anna+Rohrbach%2C+Trevor+Darrell%2C+John+Canny%2C+and+Zeynep+Akata%2E+2018%2E+Textual+Explanations+for+Self%2DDriving+Vehicles%2E+In+Proceedings+of+the+European+conference+on+computer+vision+%28ECCV%29%2C+page)
13. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2019%2E+ALBERT%3A+A+Lite+BERT+for+Self%2Dsupervised+Learning+of+Language+Representations%2E+In+International+Confe)
14. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461.
15. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, Vancouver… [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Program+Induction+by+Rationale+Generation%3A+Learning+to+Solve+and+Explain+Algebraic+Word+Problems%2E+In+Proceedings+of+the+55th+Annual+Meetin)
16. Zachary C Lipton. 2016. The Mythos of Model Interpretability. arXiv preprint arXiv:1606.03490. — [arXiv:1606.03490](https://arxiv.org/abs/1606.03490) [find](https://scholar.google.com/scholar?q=Zachary+C+Lipton%2E+2016%2E+The+Mythos+of+Model+Interpretability%2E+arXiv+preprint+arXiv%3A1606%2E03490%2E+%E2%80%94+%5BarXiv%3A1606%2E03490%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1606%2E03490%29)
17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+Robustly+Optimized+BERT+Pretraining+Approach%2E+a)
18. Scott M Lundberg and Su-In Lee. 2017. [A Unified Approach to Interpreting Model Predictions](https://md.research.elamur.ai/free/en/papers/lundberg-2017-unified-approach-interpreting-model-predictions/). In Advances in Neural Information Processing Systems, pages 4765–4774.
19. Camburu Oana-Maria, Shillingford Brendan, Minervini Pasquale, Lukasiewicz Thomas, and Blunsom Phil. 2019a. Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations. arXiv preprint arXiv:1910.03065. [find](https://scholar.google.com/scholar?q=Camburu+Oana%2DMaria%2C+Shillingford+Brendan%2C+Minervini+Pasquale%2C+Lukasiewicz+Thomas%2C+and+Blunsom+Phil%2E+2019a%2E+Make+Up+Your+Mind%21+Adversarial+Generation+of+Inconsistent+Natural+Language+Explanations%2E+arXi)
20. Camburu Oana-Maria, Giunchiglia Eleonora, Foerster Jakob, Lukasiewicz Thomas, and Blunsom Phil. 2019b. Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods. arXiv preprint arXiv:1910.02065. — [arXiv:1910.02065](https://arxiv.org/abs/1910.02065) [find](https://scholar.google.com/scholar?q=Camburu+Oana%2DMaria%2C+Giunchiglia+Eleonora%2C+Foerster+Jakob%2C+Lukasiewicz+Thomas%2C+and+Blunsom+Phil%2E+2019b%2E+Can+I+Trust+the+Explainer%3F+Verifying+Post%2Dhoc+Explanatory+Methods%2E+arXiv+preprint+arXiv%3A1910%2E0206)
21. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8). [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+Models+are+Unsupervised+Multitask+Learners%2E+OpenAI+Blog%2C+1%288%29%2E)
22. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683.
23. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. [Explain yourself! leveraging language models for commonsense reasoning](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4932–4942, Florence, Italy. Association for Computat…
24. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. [“Why Should I Trust You?”: Explaining the Predictions of Any Classifier](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/). In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144. ACM.
25. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s [Transformers: State-of-the-art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). ArXiv, abs/1910.03771.
26. Jialin Wu, Zeyuan Hu, and Raymond Mooney. 2019. Generating question relevant captions to aid visual question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3585–3594, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jialin+Wu%2C+Zeyuan+Hu%2C+and+Raymond+Mooney%2E+2019%2E+Generating+question+relevant+captions+to+aid+visual+question+answering%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+L)
27. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in neural information processing systems, pages 5754–5764.
28. Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2019. Semantics-aware BERT for Language Understanding. arXiv preprint arXiv:1909.02209. — [arXiv:1909.02209](https://arxiv.org/abs/1909.02209) [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Yuwei+Wu%2C+Hai+Zhao%2C+Zuchao+Li%2C+Shuailiang+Zhang%2C+Xi+Zhou%2C+and+Xiang+Zhou%2E+2019%2E+Semantics%2Daware+BERT+for+Language+Understanding%2E+arXiv+preprint+arXiv%3A1909%2E02209%2E+%E2%80%94+%5BarXiv%3A1909%2E02209%5D%28)

*Updated 24 October 2026*