---
title: "The emergence of number and syntax units in LSTM language"
description: "Lakretz et al. study the internal workings of a pretrained Wikipedia LSTM (from Gulordava et al.… 25 references, cited by 83 papers here."
authors: ["Yair Lakretz", "German Kruszewski", "Theo Desbordes", "Dieuwke Hupkes", "Stanislas Dehaene", "Marco Baroni"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1002/"
doi: "10.18653/v1/n19-1002"
arxiv: "1903.07435"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lakretz-2019-emergence-number-syntax-units-lstm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lakretz-2019-emergence-number-syntax-units-lstm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / The emergence of number and syntax units in LSTM language models

# The emergence of number and syntax units in LSTM language models

**Authors:** [Yair Lakretz](https://md.research.elamur.ai/free/en/authors/yair-lakretz/), [German Kruszewski](https://md.research.elamur.ai/free/en/authors/german-kruszewski/), [Theo Desbordes](https://md.research.elamur.ai/free/en/authors/theo-desbordes/), [Dieuwke Hupkes](https://md.research.elamur.ai/free/en/authors/dieuwke-hupkes/), Stanislas Dehaene and [Marco Baroni](https://md.research.elamur.ai/free/en/authors/marco-baroni/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2019](https://md.research.elamur.ai/free/en/venues/naacl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1002/) · [PDF](https://aclanthology.org/N19-1002.pdf) · [DOI 10.18653/v1/n19-1002](https://doi.org/10.18653/v1/n19-1002) · [arXiv 1903.07435](https://arxiv.org/abs/1903.07435) · [OpenAlex](https://openalex.org/W2922523190)

**Citations:** 111 citations (OpenAlex, 2026-10-14) · [83](https://md.research.elamur.ai/free/en/papers/lakretz-2019-emergence-number-syntax-units-lstm/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
LSTM language models handle long-distance subject–verb number agreement well, but it was unclear whether they rely on real syntactic structure or on shallow cues. Lakretz et al. study the internal workings of a pretrained Wikipedia LSTM (from Gulordava et al., 2018) at the level of individual neurons. They use the Linzen et al. (2016) agreement test set, a family of synthetic number-agreement tasks with controlled structures and congruent or incongruent intervening nouns, and a dataset annotated with syntactic depth. Their tools are single-unit ablation, analysis of unit activity, and inspection of recurrent weights.

They find that two dedicated units carry the subject's number over long distances, one for singular and one for plural. A separate unit that tracks syntactic structure gates these number units, signalling when to store number and when to release it. A more distributed code handles agreement only when subject and verb are close. These results suggest that an LSTM trained on raw text can learn structure-sensitive processing. The study also yields predictions that can be tested in neuroscience.

## Key points
- **Model:** the Gulordava et al. (2018) LSTM, with a 650-dimensional embedding layer, two 650-unit hidden layers, and a 50,000-word vocabulary. It was trained on Wikipedia with no fine-tuning for agreement.
- **Synthetic tasks:** the NA-tasks are Simple, Adv, 2Adv, CoAdv, NamePP, NounPP and NounPPAdv, with 600 sentences per condition. The syntactic depth set contains 4,033 positions drawn from 1,303 sentences. The data are released on GitHub (FAIRNS).
- **Number units:** ablating unit 776 (plural) or unit 988 (singular) brought accuracy close to chance in incongruent conditions, but only for subjects of the matching number.
- **Syntax unit:** unit 1150 has unusually strong weights to the number units' gates. These are positive to the forget gates (z = 8.1 and 11.2) and negative to the input gates (z = −16.2 and −7.2), which acts as a "remember" signal.
- **Two mechanisms:** a distributed short-range encoding of number also exists, but it fails to carry number across embedded phrases.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations
- [University of Amsterdam](https://md.research.elamur.ai/free/en/universities/university-of-amsterdam/) — University, Amsterdam, Netherlands · 89 works · 81,337 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [📐 Syntax & grammar formalisms](https://md.research.elamur.ai/free/en/topics/syntax-grammar/) · 161 works

## Cite

Cite as: Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lakretz2019emergence,
  title = {{The emergence of number and syntax units in LSTM language models}},
  author = {Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene and Marco Baroni},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1002},
  eprint = {1903.07435},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1002}
}
```

- [← previous Compound Probabilistic Context-Free Grammars for Grammar Induction](https://md.research.elamur.ai/free/en/papers/kim-2019-compound-probabilistic-context-free-grammars/)
- [next → Densely Connected Graph Convolutional Networks for Graph-to-Sequence Learning](https://md.research.elamur.ai/free/en/papers/guo-2019-densely-connected-graph-convolutional-networks/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. [Fine-grained analysis of sentence embeddings using auxiliary prediction tasks](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/). In Proceedings of ICLR Conference Track, Toulon, France. Published online: [https://openreview.net/group?](https://openreview.net/group?) id=ICLR.cc/2017/conference. [↗](https://openreview.net/group?)
2. Guillaume Alain and Yoshua Bengio. 2017. Understanding intermediate layers using linear classifier probes. In Proceedings of ICLR Conference Track, Toulon, France. [find](https://scholar.google.com/scholar?q=Guillaume+Alain+and+Yoshua+Bengio%2E+2017%2E+Understanding+intermediate+layers+using+linear+classifier+probes%2E+In+Proceedings+of+ICLR+Conference+Track%2C+Toulon%2C+France%2E)
3. Jean-Philippe Bernardy and Shalom Lappin. 2017. Using deep neural networks to learn syntactic agreement. Linguistic Issues in Language Technology, 15(2):1–15. [find](https://scholar.google.com/scholar?q=Jean%2DPhilippe+Bernardy+and+Shalom+Lappin%2E+2017%2E+Using+deep+neural+networks+to+learn+syntactic+agreement%2E+Linguistic+Issues+in+Language+Technology%2C+15%282%29%3A1%E2%80%9315%2E)
4. Jeffrey Bowers. 2009. On the biological plausibility of grandmother cells: Implications for neural network theories in psychology and neuroscience. Psychological Review, 116(1):220–251. agreement. Poster presented at the Learning Language in Humans and in Machines conference, online at: [https://osf.…](https://osf.…) [↗](https://osf.…)
5. Shammur Chowdhury and Roberto Zamparelli. 2018. RNN simulations of grammaticality judgments on long-distance dependencies. In Proceedings of COLING, pages 133–144, Santa Fe, NM. [find](https://scholar.google.com/scholar?q=Shammur+Chowdhury+and+Roberto+Zamparelli%2E+2018%2E+RNN+simulations+of+grammaticality+judgments+on+long%2Ddistance+dependencies%2E+In+Proceedings+of+COLING%2C+pages+133%E2%80%93144%2C+Santa+Fe%2C+NM%2E)
6. Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018b. [LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better](https://md.research.elamur.ai/free/en/papers/kuncoro-2018-lstms-can-learn-syntax-sensitive/). In Proceedings of ACL, pages 1426–1436, Melbourne, Australia.
7. Lieke Gelderloos and Grzegorz Chrupała. 2016. From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1309–1319. [find](https://scholar.google.com/scholar?q=Lieke+Gelderloos+and+Grzegorz+Chrupa%C5%82a%2E+2016%2E+From+phonemes+to+images%3A+levels+of+representation+in+a+recurrent+neural+model+of+visually%2Dgrounded+language+learning%2E+In+Proceedings+of+COLING+2016%2C+the+2)
8. Esther Kutter, Jan Bostroem, Christian Elger, Florian Mormann, and Andreas Nieder. 2018. Single neurons in the human brain encode numbers. Neuron, 100(3):753–761. [find](https://scholar.google.com/scholar?q=Esther+Kutter%2C+Jan+Bostroem%2C+Christian+Elger%2C+Florian+Mormann%2C+and+Andreas+Nieder%2E+2018%2E+Single+neurons+in+the+human+brain+encode+numbers%2E+Neuron%2C+100%283%29%3A753%E2%80%93761%2E)
9. Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the EMNLP BlackboxNLP Workshop, pages 240–248, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Mario+Giulianelli%2C+Jack+Harding%2C+Florian+Mohnert%2C+Dieuwke+Hupkes%2C+and+Willem+Zuidema%2E+2018%2E+Under+the+hood%3A+Using+diagnostic+classifiers+to+investigate+and+improve+how+language+models+track+agreement+)
10. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. [Visualizing and understanding neural models in NLP](https://md.research.elamur.ai/free/en/papers/li-2016-visualizing-understanding-neural-models-nlp/). In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 681–691.
11. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. [Colorless green recurrent networks dream hierarchically](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/). In Proceedings of NAACL, pages 1195–1205, New Orleans, LA.
12. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. [Assessing the ability of LSTMs to learn syntax-sensitive dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). Transactions of the Association for Computational Linguistics, 4:521– 535.
13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2E)
14. Tal Linzen and Brian Leonard. 2018. Distinct patterns of syntactic agreement errors in recurrent networks and humans. In Proceedings of CogSci, pages 692– 697, Austin, TX. [find](https://scholar.google.com/scholar?q=Tal+Linzen+and+Brian+Leonard%2E+2018%2E+Distinct+patterns+of+syntactic+agreement+errors+in+recurrent+networks+and+humans%2E+In+Proceedings+of+CogSci%2C+pages+692%E2%80%93+697%2C+Austin%2C+TX%2E)
15. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and ’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907–926. [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2C+Sara+Veldhoen%2C+and+Willem+Zuidema%2E+2018%2E+Visualisation+and+%E2%80%99diagnostic+classifiers%E2%80%99+reveal+how+recurrent+and+recursive+neural+networks+process+hierarchical+structure%2E+Journal+of+Artifi)
16. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1192%E2%80%931202%2E)
17. Jaap Jumelet and Dieuwke Hupkes. 2018. Do language models understand anything? on the ability of lstms to understand negative polarity items. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 222–231. [find](https://scholar.google.com/scholar?q=Jaap+Jumelet+and+Dieuwke+Hupkes%2E+2018%2E+Do+language+models+understand+anything%3F+on+the+ability+of+lstms+to+understand+negative+polarity+items%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Ana)
18. Matthew Nelson, Imen El Karoui, Kristof Giber, Xiaofang Yang, Laurent Cohen, Hilda Koopman, Sydney Cash, Lionel Naccache, John Hale, Christophe Pallier, and Stanislas Dehaene. 2017. Neurophysiological dynamics of phrase-structure building during sentence processing. Proceedings of the National Acade… [find](https://scholar.google.com/scholar?q=Matthew+Nelson%2C+Imen+El+Karoui%2C+Kristof+Giber%2C+Xiaofang+Yang%2C+Laurent+Cohen%2C+Hilda+Koopman%2C+Sydney+Cash%2C+Lionel+Naccache%2C+John+Hale%2C+Christophe+Pallier%2C+and+Stanislas+Dehaene%2E+2017%2E+Neurophysiological)
19. Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2016. [Visualizing and understanding recurrent networks](https://md.research.elamur.ai/free/en/papers/karpathy-2015-visualizing-understanding-recurrent-networks/). In Proceedings of ICLR Workshop Track, San Juan, Puerto Rico. Published online: [https://openreview.net/group?](https://openreview.net/group?) id=ICLR.cc/2016/workshop. [↗](https://openreview.net/group?)
20. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. [Learning to generate reviews and discovering sentiment](https://md.research.elamur.ai/free/en/papers/radford-2017-learning-generate-reviews-discovering-sentiment/). https://arxiv.org/abs/1704. 01444. [↗](https://arxiv.org/abs/1704)
21. Yova Kementchedjhieva and Adam Lopez. 2018. ‘In-dicatements’ that character language models learn English morpho-syntactic units and regularities. In Proceedings of the EMNLP Workshop on analyzing and interpreting neural networks for NLP, Brussels, Belgium. In press. [find](https://scholar.google.com/scholar?q=Yova+Kementchedjhieva+and+Adam+Lopez%2E+2018%2E+%E2%80%98In%2Ddicatements%E2%80%99+that+character+language+models+learn+English+morpho%2Dsyntactic+units+and+regularities%2E+In+Proceedings+of+the+EMNLP+Workshop+on+analyzing+and)
22. Zhiyuan Tang, Ying Shi, Dong Wang, Yang Feng, and Shiyue Zhang. 2017. Memory visualization for gated recurrent neural networks in speech recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 2736–2740. IEEE. [find](https://scholar.google.com/scholar?q=Zhiyuan+Tang%2C+Ying+Shi%2C+Dong+Wang%2C+Yang+Feng%2C+and+Shiyue+Zhang%2E+2017%2E+Memory+visualization+for+gated+recurrent+neural+networks+in+speech+recognition%2E+In+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP)
23. Jean-Rémi King and Stanislas Dehaene. 2014. Characterizing the dynamics of mental representations: The temporal generalization method. Trends in Cognitive Sciences, 18(4):203–210. [find](https://scholar.google.com/scholar?q=Jean%2DRe%CC%81mi+King+and+Stanislas+Dehaene%2E+2014%2E+Characterizing+the+dynamics+of+mental+representations%3A+The+temporal+generalization+method%2E+Trends+in+Cognitive+Sciences%2C+18%284%29%3A203%E2%80%93210%2E)
24. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. [What do RNN language models learn about filler–gap dependencies?](https://md.research.elamur.ai/free/en/papers/wilcox-2018-rnn-language-models-learn-about/) In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221.
25. Adhiguna Kuncoro, Chris Dyer, John Hale, and Phil Blunsom. 2018a. The perils of natural behavioral tests for unnatural models: The case of number [find](https://scholar.google.com/scholar?q=Adhiguna+Kuncoro%2C+Chris+Dyer%2C+John+Hale%2C+and+Phil+Blunsom%2E+2018a%2E+The+perils+of+natural+behavioral+tests+for+unnatural+models%3A+The+case+of+number)

*Updated 24 October 2026*