---
title: "Neuron-level Interpretation of Deep NLP Models: A Survey (2022)"
description: "They define a neuron as one output dimension of any network component. 79 references, cited by 45 papers here."
authors: ["Hassan Sajjad", "Nadir Durrani", "Fahim Dalvi"]
year: 2022
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2022.tacl-1.74/"
doi: "10.1162/tacl_a_00519"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sajjad-2022-neuron-level-interpretation-deep-nlp/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sajjad-2022-neuron-level-interpretation-deep-nlp/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Neuron-level Interpretation of Deep NLP Models: A Survey

# Neuron-level Interpretation of Deep NLP Models: A Survey

**Authors:** [Hassan Sajjad](https://md.research.elamur.ai/free/en/authors/hassan-sajjad/), [Nadir Durrani](https://md.research.elamur.ai/free/en/authors/nadir-durrani/) and [Fahim Dalvi](https://md.research.elamur.ai/free/en/authors/fahim-dalvi/) · **Year:** 2022 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2022](https://md.research.elamur.ai/free/en/venues/tacl/2022/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.tacl-1.74/) · [PDF](https://aclanthology.org/2022.tacl-1.74.pdf) · [DOI 10.1162/tacl\_a\_00519](https://doi.org/10.1162/tacl_a_00519) · [OpenAlex](https://openalex.org/W3196986263)

**Citations:** 39 citations (OpenAlex, 2026-10-14) · [45](https://md.research.elamur.ai/free/en/papers/sajjad-2022-neuron-level-interpretation-deep-nlp/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
This survey addresses a gap in work on interpreting deep NLP models. Most earlier analysis studied whole representations to measure how much linguistic knowledge they hold. The authors instead review studies that examine individual neurons. They ask which concepts single units or small groups encode, and how localized or redundant that knowledge is. They define a neuron as one output dimension of any network component. Neuron analysis is framed around three goals:
- ranking neurons for a given concept
- finding the concepts a given neuron captures
- grouping neurons that hold similar knowledge

Existing techniques fall into five families: visualization, corpus-based, probing-based, causation-based and miscellaneous. The authors compare these families on a matrix of seven attributes, such as scope, scalability, need for human involvement, supervision and link to predictions. The survey also covers evaluation practice, findings, and applications such as model control, domain adaptation and compositional explanations. Its main conclusions are that methods often select different neurons and that the field has no shared benchmarks. Causal relevance to predictions is also rarely tested. This matters for practical uses such as debiasing, distillation and domain transfer.

## Key points
- The paper appeared in volume 10 of the TACL journal (2022), pp. 1285–1303. Example scale: in BERT base, a layer block outputs 768 neurons and an attention head outputs 64.
- The survey distinguishes "focused" neurons, which carry a single concept, from "group" neurons, which encode a concept jointly. Example concept annotations include POS, SEM, Chunk and CCG tags.
- Visualization is described as subjective, hard to scale, and weak on polysemous or grouped neurons. Corpus-based methods split into Concept Search (e.g., Kádár et al., 2017; Na et al., 2019) and Neuron Search (e.g., Mu and Andreas, 2020).
- Gu et al. (2021) is cited as an application to domain adaptation. Their three-stage method ranks neurons, prunes and retrains with a teacher–student setup, then regrows the network while freezing the salient units. This reduces catastrophic forgetting.
- Open directions include unsupervised discovery of latent concepts (e.g., BERT ConceptNet by Dalvi et al., 2022) and systematic comparison of methods (Antverg and Belinkov, 2022). Another direction is evolutionary algorithms for selecting neuron groups.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hamad bin Khalifa University](https://md.research.elamur.ai/free/en/universities/hamad-bin-khalifa-university/) — University, Doha, Qatar · 71 works · 12,942 citations
- [Dalhousie University](https://md.research.elamur.ai/free/en/universities/dalhousie-university/) — University, Halifax, Canada · 6 works · 440 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Hassan Sajjad, Nadir Durrani and Fahim Dalvi. 2022. Neuron-level Interpretation of Deep NLP Models: A Survey. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{sajjad2022neuron,
  title = {{Neuron-level Interpretation of Deep NLP Models: A Survey}},
  author = {Hassan Sajjad, Nadir Durrani and Fahim Dalvi},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00519},
  url = {https://doi.org/10.1162/tacl_a_00519}
}
```

- [← previous Finding Skill Neurons in Pre-trained Transformer-based Language Models](https://md.research.elamur.ai/free/en/papers/wang-2022-finding-skill-neurons-pre-trained/)
- [next → Scaling Laws and Interpretability of Learning from Repeated Data](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

79 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. [Fine-grained analysis of sentence embeddings using auxiliary prediction tasks](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/). arXiv preprint arXiv:1608.04207.
2. J. Alammar. 2020. Interfaces for explaining transformer language models. [find](https://scholar.google.com/scholar?q=J%2E+Alammar%2E+2020%2E+Interfaces+for+explaining+transformer+language+models%2E)
3. J. Alammar. 2021. Ecco: An open source library for the explainability of transformer language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 24… [find](https://scholar.google.com/scholar?q=J%2E+Alammar%2E+2021%2E+Ecco%3A+An+open+source+library+for+the+explainability+of+transformer+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the)
4. Rana Ali Amjad, Kairen Liu, and Bernhard C. Geiger. 2018. Understanding neural networks and individual neuron importance via information-ordered cumulative ablation. [find](https://scholar.google.com/scholar?q=Rana+Ali+Amjad%2C+Kairen+Liu%2C+and+Bernhard+C%2E+Geiger%2E+2018%2E+Understanding+neural+networks+and+individual+neuron+importance+via+information%2Dordered+cumulative+ablation%2E)
5. Omer Antverg and Yonatan Belinkov. 2022. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Omer+Antverg+and+Yonatan+Belinkov%2E+2022%2E+On+the+pitfalls+of+analyzing+individual+neurons+in+language+models%2E+In+International+Conference+on+Learning+Representations%2E)
6. Omer Antverg, Eyal Ben-David, and Yonatan Belinkov. 2022. Idani: Inference-time domain adaptation via neuron-level interventions. https://doi.org/10.18653/v1/2022 .deeplo-1.3 [↗](https://doi.org/10.18653/v1/2022)
7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473.
8. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Anthony+Bau%2C+Yonatan+Belinkov%2C+Hassan+Sajjad%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+and+James+Glass%2E+2019%2E+Identifying+and+controlling+important+neurons+in+neural+machine+translation%2E+In+International+Conferenc)
9. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017a. [What do neural machine translation models learn about morphology?](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/) In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), Vancouver. Association for Computational Linguistic…
10. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020a. On the linguistic representational power of neural machine translation models. Computational Linguistics, 45(1):1–57. https://doi.org/10.1162/coli_a_00367 [↗](https://doi.org/10.1162/coli_a_00367)
11. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020b. On the linguistic representational power of neural machine translation models. Computational Linguistics, 46(1):1–52. https://doi.org/10.1162/coli_a_00367 [↗](https://doi.org/10.1162/coli_a_00367)
12. Yonatan Belinkov, Lluı́s Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017b. [Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks](https://md.research.elamur.ai/free/en/papers/belinkov-2018-evaluating-layers-representation-neural-machine/). In Proceedings of the Eighth International Joint Conference on Natural Language Proces…
13. Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. 2012. [Unsupervised feature learning and deep learning: A review and new perspectives](https://md.research.elamur.ai/free/en/papers/bengio-2012-representation-learning-review-new-perspectives/). CoRR, abs/1206.5538.
14. Maxim Binshtok, Ronen I. Brafman, Solomon Eyal Shimony, Ajay Martin, and Crag Boutilier. 2007. Computing optimal subsets. In Proceedings of the Twenty Second AAAI Conference on Artificial Intelligence (AAAI, Oral presentation). [find](https://scholar.google.com/scholar?q=Maxim+Binshtok%2C+Ronen+I%2E+Brafman%2C+Solomon+Eyal+Shimony%2C+Ajay+Martin%2C+and+Crag+Boutilier%2E+2007%2E+Computing+optimal+subsets%2E+In+Proceedings+of+the+Twenty+Second+AAAI+Conference+on+Artificial+Intelligence)
15. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). [https://doi.or…](https://doi.or…) [↗](https://doi.or…)
16. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2021. Knowledge neurons in pre-trained transformers. CoRR, abs/2104.08696. [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Li+Dong%2C+Yaru+Hao%2C+Zhifang+Sui%2C+and+Furu+Wei%2E+2021%2E+Knowledge+neurons+in+pre%2Dtrained+transformers%2E+CoRR%2C+abs%2F2104%2E08696%2E)
17. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, D. Anthony Bau, and James Glass. 2019. What is one grain of sand in the desert? Analyzing individual neurons in deep nlp models. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI, Oral presentation). https… — [doi:10.1609/aaai.v33i01.33016309](https://doi.org/10.1609/aaai.v33i01.33016309) [find](https://scholar.google.com/scholar?q=Fahim+Dalvi%2C+Nadir+Durrani%2C+Hassan+Sajjad%2C+Yonatan+Belinkov%2C+D%2E+Anthony+Bau%2C+and+James+Glass%2E+2019%2E+What+is+one+grain+of+sand+in+the+desert%3F+Analyzing+individual+neurons+in+deep+nlp+models%2E+In+Proceed)
18. Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. 2022. Discovering latent concepts learned in BERT. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Fahim+Dalvi%2C+Abdul+Rafae+Khan%2C+Firoj+Alam%2C+Nadir+Durrani%2C+Jia+Xu%2C+and+Hassan+Sajjad%2E+2022%2E+Discovering+latent+concepts+learned+in+BERT%2E+In+International+Conference+on+Learning+Representations%2E)
19. Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020. Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP-2020), pages 4908–4926, Online. [find](https://scholar.google.com/scholar?q=Fahim+Dalvi%2C+Hassan+Sajjad%2C+Nadir+Durrani%2C+and+Yonatan+Belinkov%2E+2020%2E+Analyzing+redundancy+in+pretrained+transformer+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Lang)
20. Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. 2020. How do decisions emerge across layers in neural models? Interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3243–3255, Online… [find](https://scholar.google.com/scholar?q=Nicola+De+Cao%2C+Michael+Sejr+Schlichtkrull%2C+Wilker+Aziz%2C+and+Ivan+Titov%2E+2020%2E+How+do+decisions+emerge+across+layers+in+neural+models%3F+Interpretation+with+differentiable+masking%2E+In+Proceedings+of+the+)
21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
22. Kedar Dhamdhere, Ashish Agarwal, and Mukund Sundararajan. 2020. The shapley taylor interaction index. [find](https://scholar.google.com/scholar?q=Kedar+Dhamdhere%2C+Ashish+Agarwal%2C+and+Mukund+Sundararajan%2E+2020%2E+The+shapley+taylor+interaction+index%2E)
23. Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. 2018. How important is a neuron? CoRR, abs/1805.12233. — [arXiv:1805.12233](https://arxiv.org/abs/1805.12233) [find](https://scholar.google.com/scholar?q=Kedar+Dhamdhere%2C+Mukund+Sundararajan%2C+and+Qiqi+Yan%2E+2018%2E+How+important+is+a+neuron%3F+CoRR%2C+abs%2F1805%2E12233%2E+%E2%80%94+%5BarXiv%3A1805%2E12233%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1805%2E12233%29)
24. Nadir Durrani, Fahim Dalvi, and Hassan Sajjad. 2022. Linguistic correlation analysis: Discovering salient neurons in deep NLP models. [find](https://scholar.google.com/scholar?q=Nadir+Durrani%2C+Fahim+Dalvi%2C+and+Hassan+Sajjad%2E+2022%2E+Linguistic+correlation+analysis%3A+Discovering+salient+neurons+in+deep+NLP+models%2E)
25. Nadir Durrani, Hassan Sajjad, and Fahim Dalvi. 2021. How transfer learning impacts linguistic knowledge in deep NLP models? In Findings of the Association for Computational Linguistics: ACL 2021, pages 4947–4957, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021 .fi… [↗](https://doi.org/10.18653/v1/2021)
26. Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. 2020. Analyzing individual neurons in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4865–4880, Online. Association for Computational Linguistics. h… [find](https://scholar.google.com/scholar?q=Nadir+Durrani%2C+Hassan+Sajjad%2C+Fahim+Dalvi%2C+and+Yonatan+Belinkov%2E+2020%2E+Analyzing+individual+neurons+in+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natura)
27. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. [Amnesic probing: Behavioral explanation with amnesic counterfactuals](https://md.research.elamur.ai/free/en/papers/elazar-2021-amnesic-probing-behavioral-explanation-amnesic/). Transactions of the Association for Computational Linguistics, 9:160–175. https://doi.org/10.1162/tacl a 00359 [↗](https://doi.org/10.1162/tacl)
28. Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visualizing higher-layer features of a deep network. Technical Report 1341, University of Montreal. Also presented at the ICML 2009 Workshop on Learning Feature Hierarchies, Montreal, Canada. [find](https://scholar.google.com/scholar?q=Dumitru+Erhan%2C+Yoshua+Bengio%2C+Aaron+Courville%2C+and+Pascal+Vincent%2E+2009%2E+Visualizing+higher%2Dlayer+features+of+a+deep+network%2E+Technical+Report+1341%2C+University+of+Montreal%2E+Also+presented+at+the+ICML+)
29. Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. 2015. [Sparse overcomplete word vector representations](https://md.research.elamur.ai/free/en/papers/faruqui-2015-sparse-overcomplete-word-vector-representations/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (V…
30. Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021. [CausaLM: Causal model explanation through counterfactual language models](https://md.research.elamur.ai/free/en/papers/feder-2021-causalm-causal-model-explanation-through/). Computational Linguistics, 47(2):333–386. https://doi.org/10.1162/coli a 00404 [↗](https://doi.org/10.1162/coli)
31. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719–3728, Brussels, Belgium. Associatio… [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+Alvin+Grissom+II%2C+Mohit+Iyyer%2C+Pedro+Rodriguez%2C+and+Jordan+Boyd%2DGraber%2E+2018%2E+Pathologies+of+neural+models+make+interpretations+difficult%2E+In+Proceedings+of+the+2018+Conference)
32. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+2019%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+In+International+Conference+on+Learning+Representations%2E)
33. Alona Fyshe, Leila Wehbe, Partha P. Talukdar, Brian Murphy, and Tom M. Mitchell. 2015. A compositional and interpretable semantic space. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 32–41, Den… [find](https://scholar.google.com/scholar?q=Alona+Fyshe%2C+Leila+Wehbe%2C+Partha+P%2E+Talukdar%2C+Brian+Murphy%2C+and+Tom+M%2E+Mitchell%2E+2015%2E+A+compositional+and+interpretable+semantic+space%2E+In+Proceedings+of+the+2015+Conference+of+the+North+American+Cha)
34. Fréderic Godin, Kris Demuynck, Joni Dambre, Wesley De Neve, and Thomas Demeester. 2018. Explaining character-aware neural networks for word-level prediction: Do they discover linguistic rules? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3275–3284… [find](https://scholar.google.com/scholar?q=Fre%CC%81deric+Godin%2C+Kris+Demuynck%2C+Joni+Dambre%2C+Wesley+De+Neve%2C+and+Thomas+Demeester%2E+2018%2E+Explaining+character%2Daware+neural+networks+for+word%2Dlevel+prediction%3A+Do+they+discover+linguistic+rules%3F+In+Pro)
35. Shuhao Gu, Yang Feng, and Wanying Xie. 2021. Pruning-then-expanding model for domain adaptation of neural machine translation. [find](https://scholar.google.com/scholar?q=Shuhao+Gu%2C+Yang+Feng%2C+and+Wanying+Xie%2E+2021%2E+Pruning%2Dthen%2Dexpanding+model+for+domain+adaptation+of+neural+machine+translation%2E)
36. Lucas Torroba Hennigen, Adina Williams, and Ryan Cotterell. 2020. Intrinsic probing through dimension selection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 197–216, Online. Association for Computational Linguistics. [https://doi.org](https://doi.org) /10.18… [↗](https://doi.org)
37. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, Chi… [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Percy+Liang%2E+2019%2E+Designing+and+interpreting+probes+with+control+tasks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Internati)
38. Dieuwke Hupkes. 2020. Hierarchy and interpretability in neural models of language processing. Ph.D. thesis. University of Amsterdam. [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2E+2020%2E+Hierarchy+and+interpretability+in+neural+models+of+language+processing%2E+Ph%2ED%2E+thesis%2E+University+of+Amsterdam%2E)
39. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2C+Sara+Veldhoen%2C+and+Willem+Zuidema%2E+2018%2E+Visualisation+and+%E2%80%98diagnostic+classifiers%E2%80%99+reveal+how+recurrent+and+recursive+neural+networks+process+hierarchical+structure%2E)
40. Bevan Jones, Jacob Andreas, Daniel Bauer, Karl Moritz Hermann, and Kevin Knight. 2012. [Semantics-based machine translation with hyperedge replacement grammars](https://md.research.elamur.ai/free/en/papers/jones-2012-semantics-based-machine-translation-hyperedge/). In Proceedings of COLING 2012, pages 1359–1376, Mumbai, India.
41. The COLING 2012 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2012+Organizing+Committee%2E)
42. Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. 2017. [Representation of linguistic form and function in recurrent neural networks](https://md.research.elamur.ai/free/en/papers/kadar-2017-representation-linguistic-form-function-recurrent/). Computational Linguistics, 43(4):761–780. https://doi.org/10.1162/COLI a 00300 [↗](https://doi.org/10.1162/COLI)
43. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. [Visualizing and understanding recurrent networks](https://md.research.elamur.ai/free/en/papers/karpathy-2015-visualizing-understanding-recurrent-networks/). arXiv preprint arXiv:1506.02078.
44. Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. [The emergence of number and syntax units in LSTM language models](https://md.research.elamur.ai/free/en/papers/lakretz-2019-emergence-number-syntax-units-lstm/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human La…
45. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016a. [Visualizing and understanding neural models in NLP](https://md.research.elamur.ai/free/en/papers/li-2016-visualizing-understanding-neural-models-nlp/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681–691, San Diego, California. As…
46. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b. Understanding neural networks through representation erasure. CoRR, abs/1612.08220. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+2016b%2E+Understanding+neural+networks+through+representation+erasure%2E+CoRR%2C+abs%2F1612%2E08220%2E)
47. Zachary C. Lipton. 2016. The mythos of model interpretability. In ICML Workshop on Human Interpretability in Machine Learning (WHI). [find](https://scholar.google.com/scholar?q=Zachary+C%2E+Lipton%2E+2016%2E+The+mythos+of+model+interpretability%2E+In+ICML+Workshop+on+Human+Interpretability+in+Machine+Learning+%28WHI%29%2E)
48. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. [Linguistic knowledge and transferability of contextual representations](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech…
49. Scott M. Lundberg and Su-In Lee. 2017. [A unified approach to interpreting model predictions](https://md.research.elamur.ai/free/en/papers/lundberg-2017-unified-approach-interpreting-model-predictions/). In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R.
50. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Garnett%2C+editors%2C+Advances+in+Neural+Information+Processing+Systems+30%2C+pages+4765%E2%80%934774%2E+Curran+Associates%2C+Inc%2E)
51. Ryan Cotterell. 2020. [Information-theoretic probing for linguistic structure](https://md.research.elamur.ai/free/en/papers/pimentel-2020-information-theoretic-probing-linguistic-structure/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609–4622, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main .420 [↗](https://doi.org/10.18653/v1/2020.acl-main)
52. Ana Marasović. 2018. NLP’s generalization problem, and how researchers are tackling it. The Gradient. [find](https://scholar.google.com/scholar?q=Ana+Marasovic%CC%81%2E+2018%2E+NLP%E2%80%99s+generalization+problem%2C+and+how+researchers+are+tackling+it%2E+The+Gradient%2E)
53. Leland Mclnnes, John Healy, and James Melville. 2020. [UMAP: Uniform manifold approximation and projection for dimension reduction](https://md.research.elamur.ai/free/en/papers/mcinnes-2018-umap-uniform-manifold-approximation-projection/).
54. Richard Meyes, Constantin Waubert de Puiseau, Andres Posada-Moreno, and Tobias Meisen. 2020. Under the hood of neural networks: Characterizing learned representations by functional neuron populations and network ablations. CoRR, abs/2004.01254. — [arXiv:2004.01254](https://arxiv.org/abs/2004.01254) [find](https://scholar.google.com/scholar?q=Richard+Meyes%2C+Constantin+Waubert+de+Puiseau%2C+Andres+Posada%2DMoreno%2C+and+Tobias+Meisen%2E+2020%2E+Under+the+hood+of+neural+networks%3A+Characterizing+learned+representations+by+functional+neuron+populations+)
55. Julian Michael, Jan A. Botha, and Ian Tenney. 2020. Asking without telling: Exploring latent ontologies in contextual representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6792–6812, Online. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Julian+Michael%2C+Jan+A%2E+Botha%2C+and+Ian+Tenney%2E+2020%2E+Asking+without+telling%3A+Exploring+latent+ontologies+in+contextual+representations%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Nat)
56. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). In Proceedings of the ICLR Workshop, Scottsdale, AZ, USA.
57. Jesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. CoRR, abs/2006.14032. — [arXiv:2006.14032](https://arxiv.org/abs/2006.14032) [find](https://scholar.google.com/scholar?q=Jesse+Mu+and+Jacob+Andreas%2E+2020%2E+Compositional+explanations+of+neurons%2E+CoRR%2C+abs%2F2006%2E14032%2E+%E2%80%94+%5BarXiv%3A2006%2E14032%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E14032%29)
58. W. James Murdoch, Peter J. Liu, and Bin Yu. 2018. Beyond word importance: Contextual decomposition to extract interactions from lstms. [find](https://scholar.google.com/scholar?q=W%2E+James+Murdoch%2C+Peter+J%2E+Liu%2C+and+Bin+Yu%2E+2018%2E+Beyond+word+importance%3A+Contextual+decomposition+to+extract+interactions+from+lstms%2E)
59. Seil Na, Yo Joong Choe, Dong-Hyun Lee, and Gunhee Kim. 2019. Discovery of natural language concepts in individual units of CNNs. CoRR, abs/1902.07249. — [arXiv:1902.07249](https://arxiv.org/abs/1902.07249) [find](https://scholar.google.com/scholar?q=Seil+Na%2C+Yo+Joong+Choe%2C+Dong%2DHyun+Lee%2C+and+Gunhee+Kim%2E+2019%2E+Discovery+of+natural+language+concepts+in+individual+units+of+CNNs%2E+CoRR%2C+abs%2F1902%2E07249%2E+%E2%80%94+%5BarXiv%3A1902%2E07249%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1902%2E0)
60. Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The building blocks of interpretability. Distill. [https://distill.pub/2018/building-blocks](https://distill.pub/2018/building-blocks). https://doi.org/10.23915/distill .00010 — [doi:10.23915/distill](https://doi.org/10.23915/distill) [↗](https://distill.pub/2018/building-blocks)
61. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Nina Poerner, Benjamin Roth, and Hinrich Schütze. 2018. Interpretable textual neuron representations for NLP. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NL… [find](https://scholar.google.com/scholar?q=Tiago+Pimentel%2C+Josef+Valvoda%2C+Rowan+Hall+Maudslay%2C+Ran+Zmigrod%2C+Adina+Williams%2C+and+Nina+Poerner%2C+Benjamin+Roth%2C+and+Hinrich+Schu%CC%88tze%2E+2018%2E+Interpretable+textual+neuron+representations+for+NLP%2E+In+P)
62. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%3A9%2E)
63. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017, SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Ne… [find](https://scholar.google.com/scholar?q=Maithra+Raghu%2C+Justin+Gilmer%2C+Jason+Yosinski%2C+and+Jascha+Sohl%2DDickstein%2E+2017%2C+SVCCA%3A+Singular+vector+canonical+correlation+analysis+for+deep+learning+dynamics+and+interpretability%2E+In+I%2E+Guyon%2C+U%2E+V%2E)
64. Hassan Sajjad, Firoj Alam, Fahim Dalvi, and Nadir Durrani. 2021. Effect of post-processing on contextualized word representations. CoRR, abs/2104.07456. [find](https://scholar.google.com/scholar?q=Hassan+Sajjad%2C+Firoj+Alam%2C+Fahim+Dalvi%2C+and+Nadir+Durrani%2E+2021%2E+Effect+of+post%2Dprocessing+on+contextualized+word+representations%2E+CoRR%2C+abs%2F2104%2E07456%2E)
65. Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Firoj Alam, Abdul Khan, and Jia Xu. 2022. Analyzing encoded concepts in transformer language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 308… [find](https://scholar.google.com/scholar?q=Hassan+Sajjad%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+Firoj+Alam%2C+Abdul+Khan%2C+and+Jia+Xu%2E+2022%2E+Analyzing+encoded+concepts+in+transformer+language+models%2E+In+Proceedings+of+the+2022+Conference+of+the+North+Ameri)
66. Esther Seyffarth, Younes Samih, Laura Kallmeyer, and Hassan Sajjad. 2021. Implicit representations of event properties within contextual language models: Searching for ‘‘causativity neurons’’. In International Conference on Computational Semantics (IWCS). transformation in mechanical translation. In… [find](https://scholar.google.com/scholar?q=Esther+Seyffarth%2C+Younes+Samih%2C+Laura+Kallmeyer%2C+and+Hassan+Sajjad%2E+2021%2E+Implicit+representations+of+event+properties+within+contextual+language+models%3A+Searching+for+%E2%80%98%E2%80%98causativity+neurons%E2%80%99%E2%80%99%2E+In+Inte)
67. Karolina Stanczak, Lucas Torroba Hennigen, Adina Williams, Ryan Cotterell, and Isabelle Augenstein. 2022. A latent-variable model for intrinsic probing. CoRR, abs/2201.08214. — [arXiv:2201.08214](https://arxiv.org/abs/2201.08214) [find](https://scholar.google.com/scholar?q=Karolina+Stanczak%2C+Lucas+Torroba+Hennigen%2C+Adina+Williams%2C+Ryan+Cotterell%2C+and+Isabelle+Augenstein%2E+2022%2E+A+latent%2Dvariable+model+for+intrinsic+probing%2E+CoRR%2C+abs%2F2201%2E08214%2E+%E2%80%94+%5BarXiv%3A2201%2E08214%5D%28http)
68. Elena Voita and Ivan Titov. 2020. [Information-theoretic probing with minimum description length](https://md.research.elamur.ai/free/en/papers/voita-2020-information-theoretic-probing-minimum-description/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
69. Wolfgang G. Stock. 2010. Concepts and semantic relations in information science. Journal of the American Society for Information Science and Technology, 61(10):1951–1969. https://doi.org/10.1002/asi.21382 — [doi:10.1002/asi.21382](https://doi.org/10.1002/asi.21382) [↗](https://doi.org/10.1002/asi.21382)
70. Xavier Suau, Luca Zappella, and Nicholas Apostoloff. 2020. Finding experts in transformer models. CoRR, abs/2005.07647. — [arXiv:2005.07647](https://arxiv.org/abs/2005.07647) [find](https://scholar.google.com/scholar?q=Xavier+Suau%2C+Luca+Zappella%2C+and+Nicholas+Apostoloff%2E+2020%2E+Finding+experts+in+transformer+models%2E+CoRR%2C+abs%2F2005%2E07647%2E+%E2%80%94+%5BarXiv%3A2005%2E07647%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2005%2E07647%29)
71. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. [find](https://scholar.google.com/scholar?q=Mukund+Sundararajan%2C+Ankur+Taly%2C+and+Qiqi+Yan%2E+2017%2E+Axiomatic+attribution+for+deep+networks%2E)
72. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. CoRR, abs/1409.3215. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+CoRR%2C+abs%2F1409%2E3215%2E)
73. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653 /v1/P19-14…
74. Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. arXiv preprint arXiv:1803.03585. https://doi.org/10 .18653/v1/D18-1503 [find](https://scholar.google.com/scholar?q=Ke+Tran%2C+Arianna+Bisazza%2C+and+Christof+Monz%2E+2018%2E+The+importance+of+being+recurrent+for+modeling+hierarchical+structure%2E+arXiv+preprint+arXiv%3A1803%2E03585%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E18653%2Fv1%2FD18%2D1503)
75. Mehrdad Valipour, En-Shiun Annie Lee, Jaime R. Jamacaro, and Carolina Bessega. 2019. Unsupervised transfer learning via BERT neuron selection. CoRR, abs/1912.05308. — [arXiv:1912.05308](https://arxiv.org/abs/1912.05308) [find](https://scholar.google.com/scholar?q=Mehrdad+Valipour%2C+En%2DShiun+Annie+Lee%2C+Jaime+R%2E+Jamacaro%2C+and+Carolina+Bessega%2E+2019%2E+Unsupervised+transfer+learning+via+BERT+neuron+selection%2E+CoRR%2C+abs%2F1912%2E05308%2E+%E2%80%94+%5BarXiv%3A1912%2E05308%5D%28https%3A%2F%2Farxiv%2E)
76. Bernard Vauquois. 1968. A survey of formal grammars and algorithms for recognition and John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2020. Similarity analysis of contextual word representation models. In Proceedings of the 58th Annual Meeting of the Associati… [find](https://scholar.google.com/scholar?q=Bernard+Vauquois%2E+1968%2E+A+survey+of+formal+grammars+and+algorithms+for+recognition+and+John+Wu%2C+Yonatan+Belinkov%2C+Hassan+Sajjad%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+and+James+Glass%2E+2020%2E+Similarity+analysis+)
77. Ji Xin, Jimmy Lin, and Yaoliang Yu. 2019. What part of the neural network does this? Understanding LSTMs by measuring and dissecting neurons. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… [find](https://scholar.google.com/scholar?q=Ji+Xin%2C+Jimmy+Lin%2C+and+Yaoliang+Yu%2E+2019%2E+What+part+of+the+neural+network+does+this%3F+Understanding+LSTMs+by+measuring+and+dissecting+neurons%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods)
78. Kelly Zhang and Samuel Bowman. 2018. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 359–361, Brussels, Belgium. Assoc… [find](https://scholar.google.com/scholar?q=Kelly+Zhang+and+Samuel+Bowman%2E+2018%2E+Language+modeling+teaches+you+more+than+translation+does%3A+Lessons+learned+through+auxiliary+syntactic+task+analysis%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+Blac)
79. Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling. 2017. Visualizing deep neural network decisions: Prediction difference analysis. CoRR, abs/1702.04595. *Table 3: Comparison of neuron analysis methods based on various attributes. The exhaustive list of citations for each method are pro… — [arXiv:1702.04595](https://arxiv.org/abs/1702.04595) [find](https://scholar.google.com/scholar?q=Luisa+M%2E+Zintgraf%2C+Taco+S%2E+Cohen%2C+Tameem+Adel%2C+and+Max+Welling%2E+2017%2E+Visualizing+deep+neural+network+decisions%3A+Prediction+difference+analysis%2E+CoRR%2C+abs%2F1702%2E04595%2E+%2ATable+3%3A+Comparison+of+neuron+an)

*Updated 24 October 2026*