---
title: "Identifying and Controlling Important Neurons in Neural Machine"
description: "Earlier probing work showed that NMT encoders hold morphological, syntactic and semantic… 33 references, cited by 91 papers here."
authors: ["Anthony Bau", "Yonatan Belinkov", "Hassan Sajjad", "Nadir Durrani", "Fahim Dalvi", "James Glass"]
year: 2018
venue: "arXiv:1811.01157"
source: "https://arxiv.org/abs/1811.01157"
doi: "10.48550/arxiv.1811.01157"
arxiv: "1811.01157"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bau-2018-identifying-controlling-important-neurons-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bau-2018-identifying-controlling-important-neurons-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Identifying and Controlling Important Neurons in Neural Machine Translation

# Identifying and Controlling Important Neurons in Neural Machine Translation

**Authors:** Anthony Bau, [Yonatan Belinkov](https://md.research.elamur.ai/free/en/authors/yonatan-belinkov/), [Hassan Sajjad](https://md.research.elamur.ai/free/en/authors/hassan-sajjad/), [Nadir Durrani](https://md.research.elamur.ai/free/en/authors/nadir-durrani/), [Fahim Dalvi](https://md.research.elamur.ai/free/en/authors/fahim-dalvi/) and [James Glass](https://md.research.elamur.ai/free/en/authors/james-glass/) · **Year:** 2018 · **Venue:** arXiv:1811.01157 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1811.01157) · [PDF](https://arxiv.org/pdf/1811.01157) · [DOI 10.48550/arxiv.1811.01157](https://doi.org/10.48550/arxiv.1811.01157) · [arXiv 1811.01157](https://arxiv.org/abs/1811.01157) · [OpenAlex](https://openalex.org/W2899032424)

**Citations:** 30 citations (OpenAlex, 2026-10-14) · [91](https://md.research.elamur.ai/free/en/papers/bau-2018-identifying-controlling-important-neurons-neural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Earlier probing work showed that NMT encoders hold morphological, syntactic and semantic knowledge. However, it examined whole vectors and depended on labeled data. This paper asks three questions: do single neurons matter for translation, are they interpretable, and can they be used to steer the output? The authors rank top-layer encoder neurons without supervision. Their premise is that independently trained models converge on similar important units, so they compare neurons across models using MaxCorr, MinCorr, linear regression (LinReg) and SVCCA. The experiments use 18 LSTM attention models with charCNN inputs, trained on the UN parallel corpus. Zeroing out top-ranked neurons at test time damages translation far more than removing low-ranked ones, which confirms that shared neurons carry key information. Visualizations and inexpensive supervised checks reveal neurons that track position, punctuation, noun phrases, plurality and Spanish noun gender. Shifting the activations of chosen neurons gives partial control over tense, number and gender in translations. The results suggest that some NMT knowledge is localized rather than fully distributed. The methods are task-independent and could help reduce bias.

## Key points
- **Data and models:** The models translate English into Arabic, Chinese, French, Russian and Spanish, plus an English-English auto-encoder, all trained on the UN corpus (Ziemski et al., 2016). There are 3 models per target, each trained on a different 500K-sentence slice, for 18 models in total. Each is a 500-dimensional 2-layer LSTM with attention.
- **Ranking methods:**
  - MaxCorr takes a neuron's single best Pearson match in any other model.
  - MinCorr takes the weakest of its best matches across the other models.
  - LinReg uses the mean squared error when predicting the neuron from another model's full representation.
  - SVCCA keeps PCA components covering 99% of the variance and ranks directions rather than individual neurons.
- **Verification:** Neurons are erased from the top or bottom of each ranking before decoding. Supervised checks use GMM classifiers and conditional variance computed over Spacy annotations.
- **Interpretable units (English-Spanish model):** One neuron detects plural nouns (F1 0.87), and another segments noun phrases (IOB accuracy 0.80). Others respond to parentheses, conjunctions, dates and Spanish noun gender. Many highly ranked units mainly encode word position.
- **Control:** Adjusting neuron activations changes tense, number and gender with varying success. For example, a Spanish verb moves from past (apoyó) to imperfect (apoyaba) to present (apoya). Chinese needed much larger shifts, which fits the fact that it rarely marks tense.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,105 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 149 works

## Cite

Cite as: Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi and James Glass. 2018. Identifying and Controlling Important Neurons in Neural Machine Translation. arXiv:1811.01157.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bau2018identifying,
  title = {{Identifying and Controlling Important Neurons in Neural Machine Translation}},
  author = {Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi and James Glass},
  year = {2018},
  doi = {10.48550/arxiv.1811.01157},
  eprint = {1811.01157},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.01157}
}
```

- [← previous Analyzing Compositionality-Sensitivity of NLI Models](https://md.research.elamur.ai/free/en/papers/nie-2018-analyzing-compositionality-sensitivity-nli-models/)
- [next → Parsing Tweets into Universal Dependencies](https://md.research.elamur.ai/free/en/papers/liu-2018-parsing-tweets-into-universal-dependencies/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/). arXiv preprint arXiv:1608.04207, 2016.
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473, 2014.
3. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition, 2017. [find](https://scholar.google.com/scholar?q=David+Bau%2C+Bolei+Zhou%2C+Aditya+Khosla%2C+Aude+Oliva%2C+and+Antonio+Torralba%2E+Network+dissection%3A+Quantifying+interpretability+of+deep+visual+representations%2E+In+Computer+Vision+and+Pattern+Recognition%2C+201)
4. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. [What do Neural Machine Translation Models Learn about Morphology?](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/) In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 861–872. Association for Computation…
5. La parte interesada Cuestion interesada Gran útil sing. sing. sing. α Translation Gen -1 -0.5, 0, 0.5 Temas relativos a la información Cuestiones relativas a la información ms. fm. Translation Tense past/present past/present past/impf./present past/present untensed/present Gino Brunner, Yuyi Wang… — [arXiv:1801.06024](https://arxiv.org/abs/1801.06024) [find](https://scholar.google.com/scholar?q=La+parte+interesada+Cuestion+interesada+Gran+u%CC%81til+sing%2E+sing%2E+sing%2E+%CE%B1+Translation+Gen+%2D1+%2D0%2E5%2C+0%2C+0%2E5+Temas+relativos+a+la+informacio%CC%81n+Cuestiones+relativas+a+la+informacio%CC%81n+ms%2E+fm%2E+Translation+Tens)
6. Chris Dyer, Victor Chahuneau, and Noah A. Smith. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 644–648. Association for Computatio… [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Victor+Chahuneau%2C+and+Noah+A%2E+Smith%2E+A+Simple%2C+Fast%2C+and+Effective+Reparameterization+of+IBM+Model+2%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Association)
7. Jeffrey L. Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7(2-3):195–225, 1991. [find](https://scholar.google.com/scholar?q=Jeffrey+L%2E+Elman%2E+Distributed+representations%2C+simple+recurrent+networks%2C+and+grammatical+structure%2E+Machine+learning%2C+7%282%2D3%29%3A195%E2%80%93225%2C+1991%2E)
8. J. Ganesh, Manish Gupta, and Vasudeva Varma. Interpretation of Semantic Tweet Representations. In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, ASONAM ’17, pp. 95–102, New York, NY, USA, 2017. ACM. ISBN 978-1-4503-4993-2. doi: 10.1… — [doi:10.1145/3110025.3110083](https://doi.org/10.1145/3110025.3110083) [find](https://scholar.google.com/scholar?q=J%2E+Ganesh%2C+Manish+Gupta%2C+and+Vasudeva+Varma%2E+Interpretation+of+Semantic+Tweet+Representations%2E+In+Proceedings+of+the+2017+IEEE%2FACM+International+Conference+on+Advances+in+Social+Networks+Analysis+and+)
9. Ross W. Gayler and Simon D. Levy. Compositional connectionism in cognitive science ii: the localist/distributed dimension. Connection Science, 23(2):85–89, 2011. doi: 10.1080/09540091.2011.587505. URL https://doi.org/10.1080/09540091.2011.587505. — [doi:10.1080/09540091.2011.587505](https://doi.org/10.1080/09540091.2011.587505) [↗](https://doi.org/10.1080/09540091.2011.587505)
10. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. [Convolutional Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1243–…
11. Felix Hill, Kyunghyun Cho, Sébastien Jean, and Yoshua Bengio. The representational geometry of word meanings acquired by neural machine translation models. Machine Translation, 31(1):3– 18, Jun 2017. ISSN 1573-0573. doi: 10.1007/s10590-017-9194-2. URL https://doi.org/10.1007/s10590-017-9194-2. [↗](https://doi.org/10.1007/s10590-017-9194-2)
12. Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. [Representation of linguistic form and function in recurrent neural networks](https://md.research.elamur.ai/free/en/papers/kadar-2017-representation-linguistic-form-function-recurrent/). arXiv preprint arXiv:1602.08952, 2016.
13. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. [Visualizing and understanding recurrent networks](https://md.research.elamur.ai/free/en/papers/karpathy-2015-visualizing-understanding-recurrent-networks/). arXiv preprint arXiv:1506.02078, 2015.
14. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. [Character-aware Neural Language Models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/). arXiv preprint arXiv:1508.06615, 2015.
15. Arne Köhn. What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2067–2073, Lisbon, Portugal, September 2015. Association for Computational Linguistics. URL [http://aclweb.org/…](http://aclweb.org/…) [↗](http://aclweb.org/…)
16. Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding Neural Networks through Representation Erasure. arXiv preprint arXiv:1612.08220, 2016a. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+Understanding+Neural+Networks+through+Representation+Erasure%2E+arXiv+preprint+arXiv%3A1612%2E08220%2C+2016a%2E)
17. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent Learning: Do different neural networks learn the same representations? In International Conference for Learning Representations (ICLR), 2016b. [find](https://scholar.google.com/scholar?q=Yixuan+Li%2C+Jason+Yosinski%2C+Jeff+Clune%2C+Hod+Lipson%2C+and+John+Hopcroft%2E+Convergent+Learning%3A+Do+different+neural+networks+learn+the+same+representations%3F+In+International+Conference+for+Learning+Represe)
18. Zachary C Lipton. The Mythos of Model Interpretability. In ICML Workshop on Human Interpretability in Machine Learning (WHI), 2016. [find](https://scholar.google.com/scholar?q=Zachary+C+Lipton%2E+The+Mythos+of+Model+Interpretability%2E+In+ICML+Workshop+on+Human+Interpretability+in+Machine+Learning+%28WHI%29%2C+2016%2E)
19. Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations, 2018. URL [https://openreview.net/forum?id=r1iuQjxCZ](https://openreview.net/forum?id=r1iuQjxCZ). [↗](https://openreview.net/forum?id=r1iuQjxCZ)
20. Peng Qian, Xipeng Qiu, and Xuanjing Huang. Analyzing Linguistic Knowledge in Sequential Model of Sentence. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 826–835, Austin, Texas, November 2016a. Association for Computational Linguistics. URL https://acl… [find](https://scholar.google.com/scholar?q=Peng+Qian%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+Analyzing+Linguistic+Knowledge+in+Sequential+Model+of+Sentence%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pp)
21. Peng Qian, Xipeng Qiu, and Xuanjing Huang. Investigating Language Universal and Specific Properties in Word Embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1478–1488, Berlin, Germany, August 2016b. Association for Co… [find](https://scholar.google.com/scholar?q=Peng+Qian%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+Investigating+Language+Universal+and+Specific+Properties+in+Word+Embeddings%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+L)
22. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. [Learning to generate reviews and discovering sentiment](https://md.research.elamur.ai/free/en/papers/radford-2017-learning-generate-reviews-discovering-sentiment/). arXiv preprint arXiv:1704.01444, 2017.
23. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Inf… [find](https://scholar.google.com/scholar?q=Maithra+Raghu%2C+Justin+Gilmer%2C+Jason+Yosinski%2C+and+Jascha+Sohl%2DDickstein%2E+SVCCA%3A+Singular+Vector+Canonical+Correlation+Analysis+for+Deep+Learning+Dynamics+and+Interpretability%2E+In+I%2E+Guyon%2C+U%2E+V%2E+Luxbu)
24. Rico Sennrich, Barry Haddow, and Alexandra Birch. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725. Association for Computational Linguistics, 2016. doi: 10.1…
25. Xing Shi, Kevin Knight, and Deniz Yuret. Why Neural Translations are the Right Length. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2278–2282. Association for Computational Linguistics, 2016a. doi: 10.18653/v1/D16-1248. URL [http://www.aclweb.org/anth…](http://www.aclweb.org/anth…) [↗](http://www.aclweb.org/anth…)
26. Xing Shi, Inkit Padhi, and Kevin Knight. [Does String-Based Neural MT Learn Source Syntax?](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/) In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1526–1534, Austin, Texas, November 2016b. Association for Computational Linguistics. URL [https://aclweb.org/antholo…](https://aclweb.org/antholo…) [↗](https://aclweb.org/antholo…)
27. Jinsong Su, Shan Wu, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. Variational Recurrent Neural Machine Translation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), 2018. [find](https://scholar.google.com/scholar?q=Jinsong+Su%2C+Shan+Wu%2C+Deyi+Xiong%2C+Yaojie+Lu%2C+Xianpei+Han%2C+and+Biao+Zhang%2E+Variational+Recurrent+Neural+Machine+Translation%2E+In+Proceedings+of+the+Thirty%2DSecond+AAAI+Conference+on+Artificial+Intelligenc)
28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. [Attention is All you Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Sys…
29. Matthew D Zeiler and Rob Fergus. [Visualizing and understanding convolutional networks](https://md.research.elamur.ai/free/en/papers/zeiler-2013-visualizing-understanding-convolutional-networks/). In European conference on computer vision, pp. 818–833. Springer, 2014.
30. Biao Zhang, Deyi Xiong, jinsong su, Hong Duan, and Min Zhang. [Variational Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2016-variational-neural-machine-translation/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 521–530. Association for Computational Linguistics, 2016. doi: 10.18653/v1/D16-1050. URL [http://www.acl…](http://www.acl…) [↗](http://www.acl…)
31. B. Zhou, A. Khosla, Lapedriza. A., A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. CVPR, 2016. [find](https://scholar.google.com/scholar?q=B%2E+Zhou%2C+A%2E+Khosla%2C+Lapedriza%2E+A%2E%2C+A%2E+Oliva%2C+and+A%2E+Torralba%2E+Learning+Deep+Features+for+Discriminative+Localization%2E+CVPR%2C+2016%2E)
32. Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the Importance of Individual Units in CNNs via Ablation. arXiv preprint arXiv:1806.02891, 2018. — [arXiv:1806.02891](https://arxiv.org/abs/1806.02891) [find](https://scholar.google.com/scholar?q=Bolei+Zhou%2C+Yiyou+Sun%2C+David+Bau%2C+and+Antonio+Torralba%2E+Revisiting+the+Importance+of+Individual+Units+in+CNNs+via+Ablation%2E+arXiv+preprint+arXiv%3A1806%2E02891%2C+2018%2E+%E2%80%94+%5BarXiv%3A1806%2E02891%5D%28https%3A%2F%2Farxiv%2Eor)
33. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. [The United Nations Parallel Corpus v1.0](https://md.research.elamur.ai/free/en/papers/ziemski-2016-united-nations-parallel-corpus-v1/). In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperid…

*Updated 2 November 2026*