---
title: "The Bottom-up Evolution of Representations in — Voita 2019"
description: "Earlier probing work showed that Transformer layers encode different information depending on the training objective, but it rarely explained why."
authors: ["Elena Voita", "Rico Sennrich", "Ivan Titov"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1448/"
doi: "10.18653/v1/d19-1448"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/voita-2019-bottom-up-evolution-representations-transformer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/voita-2019-bottom-up-evolution-representations-transformer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives

# The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives

**Authors:** [Elena Voita](https://md.research.elamur.ai/free/en/authors/elena-voita/), [Rico Sennrich](https://md.research.elamur.ai/free/en/authors/rico-sennrich/), [Ivan Titov](https://md.research.elamur.ai/free/en/authors/ivan-titov/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1448/) · [PDF](https://aclanthology.org/D19-1448.pdf) · [DOI 10.18653/v1/d19-1448](https://doi.org/10.18653/v1/d19-1448) · [OpenAlex](https://openalex.org/W2970820321)

**Citations:** 127 citations (openalex-snapshot · 2026-10-14) · [95](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Earlier probing work showed that Transformer layers encode different information depending on the training objective, but it rarely explained why. The authors frame this with the information bottleneck idea: each token representation has to discard input details that are irrelevant and collect what is needed to predict the output, while also supplying information to other tokens. They keep data, architecture and initialization fixed and train Transformers with three objectives: machine translation (analyzing the encoder), left-to-right LM and masked LM, all on WMT data. For each layer, they estimate mutual information with the input token and with the output label. They use canonical correlation analysis to measure how much representations change and influence one another. Nearest-neighbour similarity shows which properties are kept: word type, position, neighbouring tokens and CCG supertags. In LM layers, information about the past fades as predictions about the next token form. MLM first encodes context, partly losing token identity, then rebuilds that identity at the top. MT encoders change less and keep most word-type information. These patterns suggest why MLM tends to beat LM as a pretraining objective.

## Key points
- Data: WMT 2017 English-German (5.8m sentence pairs) and WMT 2014 English-French (40.8m). From English-French, 2.8m pairs are held out and their source side is used for analysis, while the remaining 38m are used for training. Training-set sizes of 2.5m/5.8m (En-De) and 2.5m/5.8m/38m (En-Fr) and several random initializations all give consistent results. Models follow the Transformer base setup.
- MLM follows the Devlin et al. (2018) scheme. 15% of tokens are selected for prediction: 80% of these are replaced with [MASK], 10% with a random token, and 10% are left unchanged.
- MLM shows two distinct phases, "context encoding" and "token prediction". The importance of neighbouring tokens rises until the second layer and then declines.
- Positional information is gradually dropped in LM and MLM. In the MT encoder, ordering by position is still visible at the last layer.
- CCG supertags come from the Yoshikawa et al. (2017) depccg tagger (ELMO version). LM first gathers the left (past) part of the tag and later discards it while building the right (future) part. The way MLM's CCG importance falls at higher layers matches the per-layer trends reported by Tenney et al. (2019a).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

95 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 89 not yet.

- [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/dankers-2022-can-transformer-be-too-compositional/)
- [Analyzing Transformers in Embedding Space](https://md.research.elamur.ai/free/en/papers/dar-2023-analyzing-transformers-embedding-space/)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://md.research.elamur.ai/free/en/papers/elhoushi-2024-layerskip-enabling-early-exit-inference/)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://md.research.elamur.ai/free/en/papers/geva-2022-transformer-feed-forward-layers-build/)
- [What Happens To BERT Embeddings During Fine-tuning?](https://md.research.elamur.ai/free/en/papers/merchant-2020-what-happens-bert-embeddings-during/)
- [Information-Theoretic Probing with Minimum Description Length](https://md.research.elamur.ai/free/en/papers/voita-2020-information-theoretic-probing-minimum-description/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{voita2019bottom,
  title = {{The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives}},
  author = {Voita, Elena and Sennrich, Rico and Titov, Ivan},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1448},
  url = {https://doi.org/10.18653/v1/d19-1448}
}
```

- [← previous The BEA-2019 Shared Task on Grammatical Error Correction](https://md.research.elamur.ai/free/en/papers/bryant-2019-bea-2019-shared-task-grammatical/)
- [next → The CommitmentBank: Investigating projection in naturally occurring discourse](https://md.research.elamur.ai/free/en/papers/enprag-2019-marneffe-commitmentbank-investigating-projection-naturally/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

34 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations, New Orleans. [find](https://scholar.google.com/scholar?q=Anthony+Bau%2C+Yonatan+Belinkov%2C+Hassan+Sajjad%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+and+James+Glass%2E+2019%2E+Identifying+and+controlling+important+neurons+in+neural+machine+translation%2E+In+International+Conferenc)
2. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017a. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872. Association for Co… [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+Hassan+Sajjad%2C+and+James+Glass%2E+2017a%2E+What+do+neural+machine+translation+models+learn+about+morphology%3F+In+Proceedings+of+the+55th+Annual+Meeting+of+the+) [our card](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/)
3. Yonatan Belinkov, Lluís Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017b. Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. In Proceedings of the Eighth International Joint Conference on Natural Language Processi… [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2C+Llu%C3%ADs+M%C3%A0rquez%2C+Hassan+Sajjad%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+and+James+Glass%2E+2017b%2E+Evaluating+layers+of+representation+in+neural+machine+translation+on+part%2Dof%2Dspeech+and+semantic+tag) [our card](https://md.research.elamur.ai/free/en/papers/belinkov-2018-evaluating-layers-representation-neural-machine/)
4. Arianna Bisazza and Clara Tump. 2018. The lazy encoder: A fine-grained analysis of the role of morphology in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2871–2876, Brussels, Belgium. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Arianna+Bisazza+and+Clara+Tump%2E+2018%2E+The+lazy+encoder%3A+A+fine%2Dgrained+analysis+of+the+role+of+morphology+in+neural+machine+translation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+N)
5. Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14–19, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Terra+Blevins%2C+Omer+Levy%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+RNNs+encode+soft+hierarchical+syntax%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+S)
6. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (wmt18). In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 272–… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Philipp+Koehn%2C+and+Christof+Monz%2E+2018%2E+Findings+of+the+2018+conference+on+machine+translation+%28wmt18%29%2E+In+) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/)
7. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, and Stephan Vogel. 2017. Understanding and improving morphological learning in the neural machine translation decoder. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), page… [find](https://scholar.google.com/scholar?q=Fahim+Dalvi%2C+Nadir+Durrani%2C+Hassan+Sajjad%2C+Yonatan+Belinkov%2C+and+Stephan+Vogel%2E+2017%2E+Understanding+and+improving+morphological+learning+in+the+neural+machine+translation+decoder%2E+In+Proceedings+of+th)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
9. Felix Hill, Kyunghyun Cho, Sébastien Jean, and Y Bengio. 2017. The representational geometry of word meanings acquired by neural machine translation models. Machine Translation, 31. [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Kyunghyun+Cho%2C+S%C3%A9bastien+Jean%2C+and+Y+Bengio%2E+2017%2E+The+representational+geometry+of+word+meanings+acquired+by+neural+machine+translation+models%2E+Machine+Translation%2C+31%2E)
10. Harold Hotelling. 1936. Relations between two sets of variates. Biometrika, 28:321–337. [find](https://scholar.google.com/scholar?q=Harold+Hotelling%2E+1936%2E+Relations+between+two+sets+of+variates%2E+Biometrika%2C+28%3A321%E2%80%93337%2E)
11. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+arXiv+preprint+arXiv%3A1901%2E07291%2E)
12. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltim… [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+J%2E+Bethard%2C+and+David+Mc%2DClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Proceedings+of+52nd+Annual+) [our card](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/)
13. Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In NeurIPS, Montreal, Canada. [find](https://scholar.google.com/scholar?q=Ari+S%2E+Morcos%2C+Maithra+Raghu%2C+and+Samy+Bengio%2E+2018%2E+Insights+on+representational+similarity+in+neural+networks+with+canonical+correlation%2E+In+NeurIPS%2C+Montreal%2C+Canada%2E)
14. Jan Niehues, Ronaldo Cattoni, Sebastian Stüker, Mauro Cettolo, Marco Turchi, and Marcello Federico. 2018. [find](https://scholar.google.com/scholar?q=Jan+Niehues%2C+Ronaldo+Cattoni%2C+Sebastian+St%C3%BCker%2C+Mauro+Cettolo%2C+Marco+Turchi%2C+and+Marcello+Federico%2E+2018%2E)
15. The IWSLT 2018 Evaluation Campaign. In Proceedings of the 15th International Workshop on Spoken Language Translation, pages 118–123, Bruges, Belgium. [find](https://scholar.google.com/scholar?q=The+IWSLT+2018+Evaluation+Campaign%2E+In+Proceedings+of+the+15th+International+Workshop+on+Spoken+Language+Translation%2C+pages+118%E2%80%93123%2C+Bruges%2C+Belgium%2E)
16. Liam Paninski. 2003. Estimation of entropy and mutual information. Neural computation, 15(6):1191– 1253. [find](https://scholar.google.com/scholar?q=Liam+Paninski%2E+2003%2E+Estimation+of+entropy+and+mutual+information%2E+Neural+computation%2C+15%286%29%3A1191%E2%80%93+1253%2E)
17. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
18. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
19. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297, Brussels, Belgium. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Alessandro+Raganato+and+J%C3%B6rg+Tiedemann%2E+2018%2E+An+analysis+of+encoder+representations+in+transformer%2Dbased+machine+translation%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+and+Inte) [our card](https://md.research.elamur.ai/free/en/papers/raganato-2018-analysis-encoder-representations-transformer-based/)
20. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In NeurIPS, Los Angeles. [find](https://scholar.google.com/scholar?q=Maithra+Raghu%2C+Justin+Gilmer%2C+Jason+Yosinski%2C+and+Jascha+Sohl%2DDickstein%2E+2017%2E+Svcca%3A+Singular+vector+canonical+correlation+analysis+for+deep+learning+dynamics+and+interpretability%2E+In+NeurIPS%2C+Los+An)
21. Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. 2018. Assessing generative models via precision and recall. In 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5797–5808, Florence, Italy. Association for Comput… [find](https://scholar.google.com/scholar?q=Mehdi+S%2E+M%2E+Sajjadi%2C+Olivier+Bachem%2C+Mario+Lucic%2C+Olivier+Bousquet%2C+and+Sylvain+Gelly%2E+2018%2E+Assessing+generative+models+via+precision+and+recall%2E+In+57th+Annual+Meeting+of+the+Association+for+Computa)
22. Advances in Neural Information Processing Systems 31, pages 5228–5237. [find](https://scholar.google.com/scholar?q=Advances+in+Neural+Information+Processing+Systems+31%2C+pages+5228%E2%80%935237%2E)
23. Naomi Saphra and Adam Lopez. 2019. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3257–3267, Minn… [find](https://scholar.google.com/scholar?q=Naomi+Saphra+and+Adam+Lopez%2E+2019%2E+Understanding+learning+dynamics+of+language+models+with+SVCCA%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computation)
24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Ling)
25. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural mt learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526– 1534. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xing+Shi%2C+Inkit+Padhi%2C+and+Kevin+Knight%2E+2016%2E+Does+string%2Dbased+neural+mt+learn+source+syntax%3F+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1526%E2%80%93+1) [our card](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/)
26. Masashi Yoshikawa, Hiroshi Noji, and Yuji Matsumoto. 2017. A* CCG parsing with a supertag and dependency factored model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 277–287, Vancouver, Canada. Association for Computational… [find](https://scholar.google.com/scholar?q=Masashi+Yoshikawa%2C+Hiroshi+Noji%2C+and+Yuji+Matsumoto%2E+2017%2E+A%2A+CCG+parsing+with+a+supertag+and+dependency+factored+model%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+)
27. Kelly Zhang and Samuel Bowman. 2018. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 359–361, Brussels, Belgium. Assoc… [find](https://scholar.google.com/scholar?q=Kelly+Zhang+and+Samuel+Bowman%2E+2018%2E+Language+modeling+teaches+you+more+than+translation+does%3A+Lessons+learned+through+auxiliary+syntactic+task+analysis%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+Blac)
28. Ravid Shwartz-Ziv and Naftali Tishby. 2017. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810. — [arXiv:1703.00810](https://arxiv.org/abs/1703.00810) [find](https://scholar.google.com/scholar?q=Ravid+Shwartz%2DZiv+and+Naftali+Tishby%2E+2017%2E+Opening+the+black+box+of+deep+neural+networks+via+information%2E+arXiv+preprint+arXiv%3A1703%2E00810%2E+%E2%80%94+%5BarXiv%3A1703%2E00810%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1703%2E00810%29)
29. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019a%2E+BERT+rediscovers+the+classical+NLP+pipeline%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4593%E2%80%93) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)
30. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning R… [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Patrick+Xia%2C+Berlin+Chen%2C+Alex+Wang%2C+Adam+Poliak%2C+R+Thomas+McCoy%2C+Najoung+Kim%2C+Benjamin+Van+Durme%2C+Samuel+R+Bowman%2C+Dipanjan+Das%2C+et+al%2E+2019b%2E+What+do+you+learn+from+context%3F+probing+for+) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/)
31. Naftali Tishby, Fernando C Pereira, and William Bialek. 1999. The information bottleneck method. In Proceedings of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368–377. [find](https://scholar.google.com/scholar?q=Naftali+Tishby%2C+Fernando+C+Pereira%2C+and+William+Bialek%2E+1999%2E+The+information+bottleneck+method%2E+In+Proceedings+of+the+37%2Dth+Annual+Allerton+Conference+on+Communication%2C+Control+and+Computing%2C+pages+3)
32. Naftali Tishby and Noga Zaslavsky. 2015. Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), pages 1– 5. [find](https://scholar.google.com/scholar?q=Naftali+Tishby+and+Noga+Zaslavsky%2E+2015%2E+Deep+learning+and+the+information+bottleneck+principle%2E+2015+IEEE+Information+Theory+Workshop+%28ITW%29%2C+pages+1%E2%80%93+5%2E)
33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008, Los Angeles. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi)
34. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+David+Talbot%2C+Fedor+Moiseev%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2019%2E+Analyzing+multi%2Dhead+self%2Dattention%3A+Specialized+heads+do+the+heavy+lifting%2C+the+rest+can+be+pruned%2E+In+Proceedings+of+th) [our card](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/)