---
title: "Understanding Learning Dynamics Of Language Models with"
description: "Saphra and Lopez examine this for language models. 23 references, cited by 73 papers here."
authors: ["Naomi Saphra", "Adam Lopez"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1329/"
doi: "10.18653/v1/n19-1329"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/saphra-2019-understanding-learning-dynamics-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/saphra-2019-understanding-learning-dynamics-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / Understanding Learning Dynamics Of Language Models with SVCCA

# Understanding Learning Dynamics Of Language Models with SVCCA

**Authors:** Naomi Saphra and [Adam Lopez](https://md.research.elamur.ai/free/en/authors/adam-lopez/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2019](https://md.research.elamur.ai/free/en/venues/naacl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1329/) · [PDF](https://aclanthology.org/N19-1329.pdf) · [DOI 10.18653/v1/n19-1329](https://doi.org/10.18653/v1/n19-1329) · [OpenAlex](https://openalex.org/W2950976310)

**Citations:** 31 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/saphra-2019-understanding-learning-dynamics-language-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

Earlier work showed that neural networks implicitly capture linguistic categories, but not how those encodings develop during training. Saphra and Lopez examine this for language models. They train a two-layer LSTM LM on lowercased English Wikipedia and save it after every epoch. At each snapshot, they compare its embedding and recurrent layers with the internal states of separately trained LSTM taggers. The taggers cover four kinds of labels: silver UDP POS, PTB POS from the Groningen Meaning Bank, SEM tags, and the topic of the source Wikipedia article. The comparison uses SVCCA, which first trims noisy dimensions with SVD and then measures how well the two representations correlate in a shared subspace found by CCA. This avoids training a new probe for every layer and epoch.

The LM acquires POS-like structure early in training, while topic information keeps developing later. Over time, recurrent layers grow more similar to the taggers, and embedding layers grow less similar. These findings could guide learning algorithms that make better use of linguistic structure.

## Key points

- **LM training data:** more than 227 million tokens with a 20K-type vocabulary, from English Wikipedia (70/10/20 split). Training ran for 50 epochs with SGD, and the learning rate was quartered whenever validation loss rose.
- **SVCCA settings:** the SVD step keeps enough dimensions to cover 99% of the variance. The empirical ceiling, measured against an identically built LM with a different initialization, peaks around 0.64.
- **Tag sets:** UDP POS (spacy) and topic labels come from the 100 longest Wikipedia articles. SEM tags have 66 fine classes grouped into 13 coarse ones. Representations are compared on a shared 191K-token test corpus.
- **Convergence order:** in this shallow model, the first recurrent layer settles slightly after the second. This contrasts with Raghu et al. (2017), and the authors attribute the difference to their much larger training data.
- **Efficiency and future work:** the SVCCA experiments finished in hours, whereas diagnostic-classifier probing took days. The authors suggest future work on gradient starvation and on analysing ELMo.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Naomi Saphra and Adam Lopez. 2019. Understanding Learning Dynamics Of Language Models with SVCCA. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{saphra2019understanding,
  title = {{Understanding Learning Dynamics Of Language Models with SVCCA}},
  author = {Naomi Saphra and Adam Lopez},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1329},
  url = {https://doi.org/10.18653/v1/n19-1329}
}
```

- [← previous Simple and Effective Noisy Channel Modeling for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/yee-2019-simple-effective-noisy-channel-modeling/)
- [next → Exact Hard Monotonic Attention for Character-Level Transduction](https://md.research.elamur.ai/free/en/papers/wu-2019-exact-hard-monotonic-attention-character/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

23 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Afra Alishahi, Marie Barking, and Grzegorz Chrupała. 2017. Encoding of phonology in a recurrent neural model of grounded speech. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 368–378. [find](https://scholar.google.com/scholar?q=Afra+Alishahi%2C+Marie+Barking%2C+and+Grzegorz+Chrupa%C5%82a%2E+2017%2E+Encoding+of+phonology+in+a+recurrent+neural+model+of+grounded+speech%2E+In+Proceedings+of+the+21st+Conference+on+Computational+Natural+Language)
2. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. [What do Neural Machine Translation Models Learn about Morphology?](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/) In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872, Vancouver, Canada.…
3. Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James R. Glass. 2018. [Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks](https://md.research.elamur.ai/free/en/papers/belinkov-2018-evaluating-layers-representation-neural-machine/). CoRR, abs/1801.07772.
4. Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic Tagging with Deep Residual Networks. In COLING. [find](https://scholar.google.com/scholar?q=Johannes+Bjerva%2C+Barbara+Plank%2C+and+Johan+Bos%2E+2016%2E+Semantic+Tagging+with+Deep+Residual+Networks%2E+In+COLING%2E)
5. Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. [Deep RNNs Encode Soft Hierarchical Syntax](https://md.research.elamur.ai/free/en/papers/blevins-2018-deep-rnns-encode-soft-hierarchical/). arXiv:1805.04218 [cs]. ArXiv: 1805.04218.
6. Johan Bos, Valerio Basile, Kilian Evang, Noortje J Venhuizen, and Johannes Bjerva. 2017. The groningen meaning bank. In Handbook of linguistic annotation, pages 463–496. Springer. [find](https://scholar.google.com/scholar?q=Johan+Bos%2C+Valerio+Basile%2C+Kilian+Evang%2C+Noortje+J+Venhuizen%2C+and+Johannes+Bjerva%2E+2017%2E+The+groningen+meaning+bank%2E+In+Handbook+of+linguistic+annotation%2C+pages+463%E2%80%93496%2E+Springer%2E)
7. Ondřej Cífka and Ondřej Bojar. 2018. Are bleu and meaning representation in opposition? arXiv preprint arXiv:1805.06536. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+C%C3%ADfka+and+Ondr%CC%8Cej+Bojar%2E+2018%2E+Are+bleu+and+meaning+representation+in+opposition%3F+arXiv+preprint+arXiv%3A1805%2E06536%2E)
8. Remi Tachet des Combes, Mohammad Pezeshki, Samira Shabanian, Aaron Courville, and Yoshua Bengio. 2018. On the Learning Dynamics of Deep Neural Networks. arXiv:1809.06848 [cs, stat]. ArXiv: 1809.06848. — [arXiv:1809.06848](https://arxiv.org/abs/1809.06848) [find](https://scholar.google.com/scholar?q=Remi+Tachet+des+Combes%2C+Mohammad+Pezeshki%2C+Samira+Shabanian%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2018%2E+On+the+Learning+Dynamics+of+Deep+Neural+Networks%2E+arXiv%3A1809%2E06848+%5Bcs%2C+stat%5D%2E+ArXiv%3A+1809%2E06848%2E)
9. Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/). arXiv:1805.01070 [cs]. ArXiv: 1805.01070.
10. Jonathan Frankle and Michael Carbin. 2018. The Lottery Ticket Hypothesis: Training Pruned Neural Networks. arXiv:1803.03635 [cs]. ArXiv: 1803.03635. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+2018%2E+The+Lottery+Ticket+Hypothesis%3A+Training+Pruned+Neural+Networks%2E+arXiv%3A1803%2E03635+%5Bcs%5D%2E+ArXiv%3A+1803%2E03635%2E)
11. Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information. arXiv:1808.08079 [cs]. ArXiv: 1808.08079. [find](https://scholar.google.com/scholar?q=Mario+Giulianelli%2C+Jack+Harding%2C+Florian+Mohnert%2C+Dieuwke+Hupkes%2C+and+Willem+Zuidema%2E+2018%2E+Under+the+Hood%3A+Using+Diagnostic+Classifiers+to+Investigate+and+Improve+how+Language+Models+Track+Agreement+)
12. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and’diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907–926. [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2C+Sara+Veldhoen%2C+and+Willem+Zuidema%2E+2018%2E+Visualisation+and%E2%80%99diagnostic+classifiers%E2%80%99+reveal+how+recurrent+and+recursive+neural+networks+process+hierarchical+structure%2E+Journal+of+Artific)
13. Yova Kementchedjhieva and Adam Lopez. 2018. Indicatements that character language models learn English morpho-syntactic units and regularities. In Proc. of Workshop on Analyzing and interpreting neural networks for NLP. [find](https://scholar.google.com/scholar?q=Yova+Kementchedjhieva+and+Adam+Lopez%2E+2018%2E+Indicatements+that+character+language+models+learn+English+morpho%2Dsyntactic+units+and+regularities%2E+In+Proc%2E+of+Workshop+on+Analyzing+and+interpreting+neura)
14. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context. arXiv:1805.04623 [cs]. ArXiv: 1805.04623. [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+He+He%2C+Peng+Qi%2C+and+Dan+Jurafsky%2E+2018%2E+Sharp+Nearby%2C+Fuzzy+Far+Away%3A+How+Neural+Language+Models+Use+Context%2E+arXiv%3A1805%2E04623+%5Bcs%5D%2E+ArXiv%3A+1805%2E04623%2E)
15. Christo Kirov and Ryan Cotterell. 2018. [Recurrent neural networks in linguistic theory: Revisiting pinker and prince (1988) and the past tense debate](https://md.research.elamur.ai/free/en/papers/kirov-2018-recurrent-neural-networks-linguistic-theory/). arXiv preprint arXiv:1807.04783.
16. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). arXiv:1611.01368 [cs]. ArXiv: 1611.01368.
17. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). arXiv preprint arXiv:1802.05365.
18. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. arXiv:1706.05806 [cs, stat]. ArXiv: 1706.05806. — [arXiv:1706.05806](https://arxiv.org/abs/1706.05806) [find](https://scholar.google.com/scholar?q=Maithra+Raghu%2C+Justin+Gilmer%2C+Jason+Yosinski%2C+and+Jascha+Sohl%2DDickstein%2E+2017%2E+SVCCA%3A+Singular+Vector+Canonical+Correlation+Analysis+for+Deep+Learning+Dynamics+and+Interpretability%2E+arXiv%3A1706%2E05806+%5B)
19. Ravid Shwartz-Ziv and Naftali Tishby. 2017. Opening the Black Box of Deep Neural Networks via Information. arXiv:1703.00810 [cs]. ArXiv: 1703.00810. — [arXiv:1703.00810](https://arxiv.org/abs/1703.00810) [find](https://scholar.google.com/scholar?q=Ravid+Shwartz%2DZiv+and+Naftali+Tishby%2E+2017%2E+Opening+the+Black+Box+of+Deep+Neural+Networks+via+Information%2E+arXiv%3A1703%2E00810+%5Bcs%5D%2E+ArXiv%3A+1703%2E00810%2E+%E2%80%94+%5BarXiv%3A1703%2E00810%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1703%2E008)
20. Eva Vanmassenhove, Jinhua Du, and Andy Way. 2017. Investigating ‘aspect’in nmt and smt: Translating the english simple past and present perfect. Computational Linguistics in the Netherlands Journal, 7:109–128. [find](https://scholar.google.com/scholar?q=Eva+Vanmassenhove%2C+Jinhua+Du%2C+and+Andy+Way%2E+2017%2E+Investigating+%E2%80%98aspect%E2%80%99in+nmt+and+smt%3A+Translating+the+english+simple+past+and+present+perfect%2E+Computational+Linguistics+in+the+Netherlands+Journal%2C+7)
21. John Wieting and Douwe Kiela. 2019. No training required: Exploring random encoders for sentence classification. arXiv preprint arXiv:1901.10444. — [arXiv:1901.10444](https://arxiv.org/abs/1901.10444) [find](https://scholar.google.com/scholar?q=John+Wieting+and+Douwe+Kiela%2E+2019%2E+No+training+required%3A+Exploring+random+encoders+for+sentence+classification%2E+arXiv+preprint+arXiv%3A1901%2E10444%2E+%E2%80%94+%5BarXiv%3A1901%2E10444%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1901%2E10444%29)
22. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530. [find](https://scholar.google.com/scholar?q=Chiyuan+Zhang%2C+Samy+Bengio%2C+Moritz+Hardt%2C+Benjamin+Recht%2C+and+Oriol+Vinyals%2E+2016%2E+Understanding+deep+learning+requires+rethinking+generalization%2E+arXiv+preprint+arXiv%3A1611%2E03530%2E)
23. Kelly W. Zhang and Samuel R. Bowman. 2018. Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis. arXiv:1809.10040 [cs]. ArXiv: 1809.10040. [find](https://scholar.google.com/scholar?q=Kelly+W%2E+Zhang+and+Samuel+R%2E+Bowman%2E+2018%2E+Language+Modeling+Teaches+You+More+Syntax+than+Translation+Does%3A+Lessons+Learned+Through+Auxiliary+Task+Analysis%2E+arXiv%3A1809%2E10040+%5Bcs%5D%2E+ArXiv%3A+1809%2E10040%2E)

*Updated 24 October 2026*