--- title: "What Does BERT Learn about the Structure of — Jawahar 2019" description: "This paper asks which parts of English linguistic structure BERT actually encodes, and at which depth of the network." authors: ["Ganesh Jawahar", "Benoît Sagot", "Djamé Seddah"] year: 2019 venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P19-1356/" doi: "10.18653/v1/p19-1356" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / What Does BERT Learn about the Structure of Language? # What Does BERT Learn about the Structure of Language? **Authors:** [Ganesh Jawahar](https://md.research.elamur.ai/free/en/authors/ganesh-jawahar/), [Benoît Sagot](https://md.research.elamur.ai/free/en/authors/benoit-sagot/), [Djamé Seddah](https://md.research.elamur.ai/free/en/authors/djame-seddah/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P19-1356/) · [PDF](https://aclanthology.org/P19-1356.pdf) · [DOI 10.18653/v1/p19-1356](https://doi.org/10.18653/v1/p19-1356) · [OpenAlex](https://openalex.org/W2948947170) **Citations:** 1219 citations (openalex-snapshot · 2026-10-14) · [568](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This paper asks which parts of English linguistic structure BERT actually encodes, and at which depth of the network. The authors probe each of the 12 layers of the pretrained bert-base-uncased model in four ways. First, they cluster span representations of CoNLL 2000 chunks. Second, they run the ten SentEval probing tasks of Conneau et al. (2018) and compare against an untrained BERT baseline. Third, they test subject-verb agreement on the Linzen et al. (2016) stimuli while controlling for the number of attractor nouns. Fourth, they fit Tensor Product Decomposition Networks (TPDN) with different role schemes to approximate BERT's representations on SNLI. Phrase-level information is strongest in the early layers and fades higher up. Lower layers favour surface features, middle layers syntactic ones, and upper layers semantic ones. Agreement across longer distances needs deeper layers. A tree-based role scheme fits BERT's representations best, so even a purely attention-based model ends up with implicitly hierarchical, compositional structure. This helps explain BERT's success and points to its limits. ## Key points - Phrasal syntax: k-means clustering (k = 10) of span embeddings gives an NMI of 0.38 at layer 1, which drops to about 0.16–0.19 in layers 8–12. - SentEval probing shows a layered hierarchy. SentLen peaks at layer 3 (96.2), and an untrained BERT beats the trained model on it in higher layers. TopConst peaks at layer 7 (84.1), and BShift and CoordInv peak at layer 9, both above earlier published results. - Subject-verb agreement: with no attractors, accuracy reaches about 95.7 (layer 8). With 3–4 attractors, the best scores are around 49–50 and appear only at layer 9. - TPDN: a tree-based role scheme gives the lowest MSE against BERT at most layers, ahead of left-to-right, right-to-left, bag-of-words, bidirectional and random-tree schemes. - A dependency tree built from the self-attention weights of layer 2, head 11 using Chu-Liu-Edmonds recovers determiner–noun and subject–verb links. The code is published at github.com/ganeshjawahar/interpret_bert. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 568 papers of our library cite it (of the 140673 whose bibliographies we read); 36 of them have a page here, 532 not yet. - [BEReshiT: an Ancient Hebrew Model based on DictaBERT](https://md.research.elamur.ai/free/en/papers/aclbulk-2026-stoupak-bereshit-ancient-hebrew-model-based/) - [Syntax-augmented Multilingual BERT for Cross-lingual Transfer](https://md.research.elamur.ai/free/en/papers/ahmad-2021-syntax-augmented-multilingual-bert-cross/) - [Anaphoric Zero Pronoun Identification: A Multilingual Approach](https://md.research.elamur.ai/free/en/papers/aloraini-2020-anaphoric-zero-pronoun-identification-multilingual/) - [Cross-lingual Zero Pronoun Resolution](https://md.research.elamur.ai/free/en/papers/aloraini-2020-cross-lingual-zero-pronoun-resolution/) - [Neural Coreference Resolution for Arabic](https://md.research.elamur.ai/free/en/papers/aloraini-2020-neural-coreference-resolution-arabic/) - [Comparing Native and Learner Englishes Using a Large Pre-trained Language Model](https://md.research.elamur.ai/free/en/papers/aoyama-2022-comparing-native-learner-englishes-large/) - [Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/) - [BERTje: A Dutch BERT Model](https://md.research.elamur.ai/free/en/papers/de-2019-vries-bertje-dutch-bert-model/) - [Paragraph-level Simplification of Medical Texts](https://md.research.elamur.ai/free/en/papers/devaraj-2021-paragraph-level-simplification-medical-texts/) - [Joint Universal Syntactic and Semantic Parsing](https://md.research.elamur.ai/free/en/papers/eskin-2021-joint-universal-syntactic-semantic-parsing/) - [Fine-grained Information Status Classification Using Discourse Context-Aware BERT](https://md.research.elamur.ai/free/en/papers/hou-2020-fine-grained-information-status-classification/) - [Transformer-Patcher: One Mistake worth One Neuron](https://md.research.elamur.ai/free/en/papers/huang-2023-transformer-patcher-one-mistake-worth/) - [Probing Contextual Language Models for Common Ground with Visual Representations](https://md.research.elamur.ai/free/en/papers/ilharco-2021-probing-contextual-language-models-common/) - [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/) - [How Can We Know What Language Models Know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) - [Attention is Not Only a Weight: Analyzing Transformers with Vector Norms](https://md.research.elamur.ai/free/en/papers/kobayashi-2020-attention-is-not-only-weight/) - [Top-down Discourse Parsing via Sequence Labelling](https://md.research.elamur.ai/free/en/papers/koto-2021-top-down-discourse-parsing-sequence/) - [Revealing the Dark Secrets of BERT](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/) - [HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition](https://md.research.elamur.ai/free/en/papers/langstream-2021-chriqui-hebert-hebemo-hebrew-bert-model/) - [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://md.research.elamur.ai/free/en/papers/lex-2021-hofmann-superbizarre-derivational/) - [Coreference-Aware Dialogue Summarization](https://md.research.elamur.ai/free/en/papers/liu-2021-coreference-aware-dialogue-summarization/) - [POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection](https://md.research.elamur.ai/free/en/papers/liu-2022-politics-pretraining-same-story-article/) - [Compositional Meaning Representations in LLMs: a Critical Review of Probing Studies](https://md.research.elamur.ai/free/en/papers/marro-2026-compositional-meaning-representations-llms-critical/) - [CamemBERT: a Tasty French Language Model](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/) - [Automatic Prediction of Prominence and Boundary Strength from Text](https://md.research.elamur.ai/free/en/papers/mas-2026-automatic-prediction-prominence-boundary-strength/) - [Rethinking the Idiomaticity Decomposability Hypothesis: Evidence from Distributional Learning](https://md.research.elamur.ai/free/en/papers/mi-2026-rethinking-idiomaticity-decomposability-hypothesis-evidence/) - [PhoBERT: Pre-trained language models for Vietnamese](https://md.research.elamur.ai/free/en/papers/nguyen-2020-phobert-pre-trained-language-models/) - [Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/) - [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/) - [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/) - [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/) - [How Are Idioms Processed Inside Transformer Language Models?](https://md.research.elamur.ai/free/en/papers/tian-2023-how-are-idioms-processed-inside/) - [Probing Pretrained Language Models for Lexical Semantics](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/) - [UniTE: Unified Translation Evaluation](https://md.research.elamur.ai/free/en/papers/wan-2022-unite-unified-translation-evaluation/) - [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://md.research.elamur.ai/free/en/papers/ye-2022-zerogen-efficient-zero-shot-learning/) - [Explainability for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{jawahar2019what, title = {{What Does BERT Learn about the Structure of Language?}}, author = {Jawahar, Ganesh and Sagot, Benoît and Seddah, Djamé}, booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, year = {2019}, doi = {10.18653/v1/p19-1356}, url = {https://doi.org/10.18653/v1/p19-1356} } ``` - [← previous What do you learn from context? Probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/) - [next → What makes a good conversation? How controllable attributes affect human judgments](https://md.research.elamur.ai/free/en/papers/see-2019-what-makes-good-conversation-how/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) ## References 24 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yossi+Adi%2C+Einat+Kermany%2C+Yonatan+Belinkov%2C+Ofer+Lavi%2C+and+Yoav+Goldberg%2E+2017%2E+Fine%2Dgrained+Analysis+of+Sentence+Embeddings+Using+Auxiliary+Prediction+Tasks%2E+International+Conference+on+Learning+Repr) [our card](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/) 2. Yonatan Belinkov and James Glass. 2019. Analysis Methods in Neural Language Processing: A Survey. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+James+Glass%2E+2019%2E+Analysis+Methods+in+Neural+Language+Processing%3A+A+Survey%2E+Transactions+of+the+Association+for+Computational+Linguistics%2E) [our card](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/) 3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica) 4. Yoeng-Jin Chu and Tseng-Hong Liu. 1967. On the shortest arborescence of a directed graph. In Science Sinica, pages 1396–1400. [find](https://scholar.google.com/scholar?q=Yoeng%2DJin+Chu+and+Tseng%2DHong+Liu%2E+1967%2E+On+the+shortest+arborescence+of+a+directed+graph%2E+In+Science+Sinica%2C+pages+1396%E2%80%931400%2E) 5. Alexis Conneau and Douwe Kiela. 2018. SentEval: An Evaluation Toolkit for Universal Sentence Representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). European Language Resource Association. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Douwe+Kiela%2E+2018%2E+SentEval%3A+An+Evaluation+Toolkit+for+Universal+Sentence+Representations%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Evaluati) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/) 6. Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single \\$&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Germa%CC%81n+Kruszewski%2C+Guillaume+Lample%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Marco+Baroni%2E+2018%2E+What+you+can+cram+into+a+single+%5C%5C%24%26%21%5C%23%2A+vector%3A+Probing+sentence+embeddings+for+linguistic+properties%2E+In) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/) 7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Computing Research Repository, arXiv:1810.04805. Version 1. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding%2E+Computing+Research+Repository%2C+arXiv%3A1810%2E0480) 8. Yoav Goldberg. 2019. Assessing BERT’s Syntactic Abilities. Computing Research Repository, arXiv:1901.05287. Version 1. [find](https://scholar.google.com/scholar?q=Yoav+Goldberg%2E+2019%2E+Assessing+BERT%E2%80%99s+Syntactic+Abilities%2E+Computing+Research+Repository%2C+arXiv%3A1901%2E05287%2E+Version+1%2E) [our card](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/) 9. Networks Process Hierarchical Structure. J. Artif. Int. Res., 61(1):907–926. [find](https://scholar.google.com/scholar?q=Networks+Process+Hierarchical+Structure%2E+J%2E+Artif%2E+Int%2E+Res%2E%2C+61%281%29%3A907%E2%80%93926%2E) 10. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, pages 423– 430, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dan+Klein+and+Christopher+D%2E+Manning%2E+2003%2E+Accurate+unlexicalized+parsing%2E+In+Proceedings+of+the+41st+Annual+Meeting+on+Association+for+Computational+Linguistics+%2D+Volume+1%2C+ACL+%E2%80%9903%2C+pages+423%E2%80%93+430%2C+) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2003-klein-accurate-unlexicalized-parsing/) 11. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2C+Emmanuel+Dupoux%2C+and+Yoav+Goldberg%2E+2016%2E+Assessing+the+Ability+of+LSTMs+to+Learn+Syntax%2DSensitive+Dependencies%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+4%3A521%E2%80%93+535%2E) [our card](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/) 12. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A. Smith. 2019. Linguistic Knowledge and Transferability of Contextual Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol… [find](https://scholar.google.com/scholar?q=Nelson+F%2E+Liu%2C+Matt+Gardner%2C+Yonatan+Belinkov%2C+Matthew+Peters%2C+and+Noah+A%2E+Smith%2E+2019%2E+Linguistic+Knowledge+and+Transferability+of+Contextual+Representations%2E+In+Proceedings+of+the+2019+Conference+of) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/) 13. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov):2579–2605. [find](https://scholar.google.com/scholar?q=Laurens+van+der+Maaten+and+Geoffrey+Hinton%2E+2008%2E+Visualizing+data+using+t%2DSNE%2E+Journal+of+machine+learning+research%2C+9%28Nov%29%3A2579%E2%80%932605%2E) 14. R. Thomas McCoy, Tal Linzen, Ewan Dunbar, and Paul Smolensky. 2019. RNNs Implicitly Implement Tensor Product Representations. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=R%2E+Thomas+McCoy%2C+Tal+Linzen%2C+Ewan+Dunbar%2C+and+Paul+Smolensky%2E+2019%2E+RNNs+Implicitly+Implement+Tensor+Product+Representations%2E+International+Conference+on+Learning+Representations%2E) 15. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532– 1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D%2E+Manning%2E+2014%2E+GloVe%3A+Global+Vectors+for+Word+Representation%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+1532%E2%80%93+1543%2E) 16. Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting Contextual Word Embeddings: Architecture and Representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Luke+Zettlemoyer%2C+and+Wen%2Dtau+Yih%2E+2018%2E+Dissecting+Contextual+Word+Embeddings%3A+Architecture+and+Representation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Method) [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/) 17. John Hewitt and Christopher D. Manning. 2019. A Structural Probe for Finding Syntax in Word Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Christopher+D%2E+Manning%2E+2019%2E+A+Structural+Probe+for+Finding+Syntax+in+Word+Representations%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/) 18. Alessandro Raganato and Jörg Tiedemann. 2018. An Analysis of Encoder Representations in Transformer-Based Machine Translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alessandro+Raganato+and+Jo%CC%88rg+Tiedemann%2E+2018%2E+An+Analysis+of+Encoder+Representations+in+Transformer%2DBased+Machine+Translation%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+and+Int) 19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E) 20. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 Shared Task Chunking. In Fourth Conference on Computational Natural Language Learning, CoNLL 2000, and the Second Learning Language in Logic Workshop, LLL 2000, Held in cooperation with ICGI-2000, Lisbon, Portugal, Sept… [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang+and+Sabine+Buchholz%2E+2000%2E+Introduction+to+the+CoNLL%2D2000+Shared+Task+Chunking%2E+In+Fourth+Conference+on+Computational+Natural+Language+Learning%2C+CoNLL+2000%2C+and+the+Second+Learn) [our card](https://md.research.elamur.ai/free/en/papers/sang-2000-introduction-conll-2000-shared-task/) 21. Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and ‘Diagnostic Classifiers’ Reveal How Recurrent and Recursive Neural Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 20… [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2C+Sara+Veldhoen%2C+and+Willem+Zuidema%2E+2018%2E+Visualisation+and+%E2%80%98Diagnostic+Classifiers%E2%80%99+Reveal+How+Recurrent+and+Recursive+Neural+Ian+Tenney%2C+Patrick+Xia%2C+Berlin+Chen%2C+Alex+Wang%2C+Adam+Poli) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/) 22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017,… [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+All+you+Need%2E+In+Advances+in+Neural+Information+Process) 23. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355.… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E+GLUE%3A+A+Multi%2DTask+Benchmark+and+Analysis+Platform+for+Natural+Language+Understanding%2E+In+Proceedings+of+the) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) 24. Kelly W. Zhang and Samuel R. Bowman. 2018. Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis. Computing Research Repository, arXiv:1809.10040. Version 2. [find](https://scholar.google.com/scholar?q=Kelly+W%2E+Zhang+and+Samuel+R%2E+Bowman%2E+2018%2E+Language+Modeling+Teaches+You+More+Syntax+than+Translation+Does%3A+Lessons+Learned+Through+Auxiliary+Task+Analysis%2E+Computing+Research+Repository%2C+arXiv%3A1809%2E1)