--- title: "Assessing BERT's Syntactic Abilities — Goldberg 2019" description: "This paper tests whether BERT handles English syntactic dependencies that earlier studies had probed in LSTMs." authors: ["Yoav Goldberg"] year: 2019 venue: "arXiv:1901.05287" source: "https://arxiv.org/abs/1901.05287" doi: "10.48550/arxiv.1901.05287" arxiv: "1901.05287" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / Assessing BERT's Syntactic Abilities # Assessing BERT's Syntactic Abilities **Authors:** [Yoav Goldberg](https://md.research.elamur.ai/free/en/authors/yoav-goldberg/) · **Year:** 2019 · **Venue:** arXiv:1901.05287 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1901.05287) · [PDF](https://arxiv.org/pdf/1901.05287) · [DOI 10.48550/arxiv.1901.05287](https://doi.org/10.48550/arxiv.1901.05287) · [arXiv 1901.05287](https://arxiv.org/abs/1901.05287) · [OpenAlex](https://openalex.org/W2910243263) **Citations:** 248 citations (openalex-snapshot · 2026-10-14) · [261](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This paper tests whether BERT handles English syntactic dependencies that earlier studies had probed in LSTMs. BERT is a Transformer built only on self-attention, with absolute position embeddings and no recurrent state, so it was not obvious that it would. Goldberg reuses three existing stimulus sets: - naturally occurring Wikipedia agreement sentences from Linzen et al. (2016); - nonce sentences from Gulordava et al. (2018), where content words are swapped for random words with the same part of speech and inflection, so the model can't rely on meaning; - controlled minimal pairs from Marvin and Linzen (2018) for subject-verb agreement and reflexive anaphora. Because BERT reads context in both directions, the focus verb is replaced with [MASK] and the rest of the sentence stays visible. The model counts as correct when it scores the right verb form above the wrong one. The pre-trained bert-base-uncased and bert-large-uncased models are used with no fine-tuning. Both get high accuracy on every test and usually beat the published LSTM numbers. The setups differ, so this comparison is only a rough guide. Still, the result goes against the expectation that attention-only models struggle with hierarchical structure, and it helped prompt later probing of syntax in pretrained Transformers. ## Key points - **Linzen et al. (2016) sentences:** accuracy stays at 0.96–0.97 for both models with one to four agreement attractors. There are 24,031 sentences with one attractor and 254 with four. - **Gulordava et al. (2018) nonce set (383 pairs):** BERT Base scores 0.83 and BERT Large 0.80. The LSTM result reported in the original work is 74.1 ± 1.6. - **Marvin and Linzen (2018) pairs, where BERT wins:** BERT is far ahead on long-distance agreement. For example, across an object relative clause BERT Base scores 0.89 and the LSTM 0.50. - **Marvin and Linzen (2018) pairs, where the LSTM wins:** the LSTM is better only on sentential complements and short VP coordination. Goldberg attributes this to the LSTM overfitting those constructions. BERT Base often outperforms BERT Large. - **Discarded stimuli:** three kinds were removed. - Negative polarity item (NPI) pairs, because they differ in two words. - is/are cases in the naturalistic sets, because copular sentences give extra agreement cues. - Verbs that are not single wordpieces in BERT's vocabulary. This removed 23,368 of 152,300 Marvin and Linzen pairs, 680 Linzen sentences and 28 Gulordava pairs. Code is at github.com/yoavg/bert-syntax. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 261 papers of our library cite it (of the 140673 whose bibliographies we read); 38 of them have a page here, 223 not yet. - [Unsupervised Domain Clusters in Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/aharoni-2020-unsupervised-domain-clusters-pretrained-language/) - [The Unreasonable Effectiveness of Transformer Language Models in Grammatical Error Correction](https://md.research.elamur.ai/free/en/papers/alikaniotis-2019-unreasonable-effectiveness-transformer-language-models/) - [Anaphoric Zero Pronoun Identification: A Multilingual Approach](https://md.research.elamur.ai/free/en/papers/aloraini-2020-anaphoric-zero-pronoun-identification-multilingual/) - [On the Cross-lingual Transferability of Monolingual Representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/) - [Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/) - [PIQA: Reasoning about Physical Commonsense in Natural Language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/) - [Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?](https://md.research.elamur.ai/free/en/papers/eacl-2023-shmidman-do-pretrained-contextual-language-models/) - [Probing for idiomaticity in vector space models](https://md.research.elamur.ai/free/en/papers/garcia-2021-probing-idiomaticity-vector-space-models/) - [Assessing the Agreement Competence of Large Language Models](https://md.research.elamur.ai/free/en/papers/garcia-2025-assessing-agreement-competence-large-language/) - [Collocational bootstrapping: A hypothesis about the learning of subject-verb agreement in humans and neural networks](https://md.research.elamur.ai/free/en/papers/hobbs-2026-collocational-bootstrapping-hypothesis-about-learning/) - [A closer look at the performance of neural language models on reflexive anaphor licensing](https://md.research.elamur.ai/free/en/papers/hu-2020-closer-look-performance-neural-language/) - [What Does BERT Learn about the Structure of Language?](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/) - [How Can We Know What Language Models Know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) - [ARAFA: An LLM-Generated Arabic Fact-Checking Dataset](https://md.research.elamur.ai/free/en/papers/khalil-2026-arafa-llm-generated-arabic-fact/) - [Adapting BERT to Implicit Discourse Relation Classification with a Focus on Discourse Connectives](https://md.research.elamur.ai/free/en/papers/kishimoto-2020-adapting-bert-implicit-discourse-relation/) - [Attention is Not Only a Weight: Analyzing Transformers with Vector Norms](https://md.research.elamur.ai/free/en/papers/kobayashi-2020-attention-is-not-only-weight/) - [Revealing the Dark Secrets of BERT](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/) - [Can language models capture syntactic associations without surface cues? A case study of reflexive anaphor licensing in English control constructions](https://md.research.elamur.ai/free/en/papers/lee-2022-can-language-models-capture-syntactic/) - [Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model](https://md.research.elamur.ai/free/en/papers/lex-2023-weissweiler-chatgpt-wugs-morphology/) - [Open Sesame: Getting inside BERT's Linguistic Knowledge](https://md.research.elamur.ai/free/en/papers/lin-2019-open-sesame-getting-inside-bert/) - [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/) - [What Happens To BERT Embeddings During Fine-tuning?](https://md.research.elamur.ai/free/en/papers/merchant-2020-what-happens-bert-embeddings-during/) - [Syntactic Data Augmentation Increases Robustness to Inference Heuristics](https://md.research.elamur.ai/free/en/papers/min-2020-syntactic-data-augmentation-increases-robustness/) - [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/) - [Language Models as Knowledge Bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) - [Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/) - [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/) - [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://md.research.elamur.ai/free/en/papers/ravfogel-2020-null-it-out-guarding-protected/) - [Probing Natural Language Inference Models through Semantic Fragments](https://md.research.elamur.ai/free/en/papers/richardson-2019-probing-natural-language-inference-models/) - [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://md.research.elamur.ai/free/en/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/) - [oLMpics -- On what Language Model Pre-training Captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/) - [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://md.research.elamur.ai/free/en/papers/ud-2019-kondratyuk-75-languages-1-model-parsing/) - [Analyzing the Structure of Attention in a Transformer Language Model](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/) - [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/) - [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://md.research.elamur.ai/free/en/papers/wu-2020-perturbed-masking-parameter-free-probing/) - [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://md.research.elamur.ai/free/en/papers/ye-2022-zerogen-efficient-zero-shot-learning/) - [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://md.research.elamur.ai/free/en/papers/yin-2020-tabert-pretraining-joint-understanding-textual/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{goldberg2019assessing, title = {{Assessing BERT's Syntactic Abilities}}, author = {Goldberg, Yoav}, year = {2019}, doi = {10.48550/arxiv.1901.05287}, eprint = {1901.05287}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1901.05287} } ``` - [← previous Aspect-Level Sentiment Analysis Via Convolution over Dependency Tree](https://md.research.elamur.ai/free/en/papers/sun-2019-aspect-level-sentiment-analysis-convolution/) - [next → Attention Guided Graph Convolutional Networks for Relation Extraction](https://md.research.elamur.ai/free/en/papers/guo-2019-attention-guided-graph-convolutional-networks/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/) ## References 6 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jacob Devlin, Ming-Wei Chang, Ken-Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ton Lee, and Kristina Toutanova. 2018. Ł ukasz Kaiser, and Illia Polosukhin. 2017. BERT: pre-training of deep bidirectional transformers for language Attention understanding. is… [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Ken%2DJakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+ton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2) 2. BERT Large LSTM (M&L) Humans (M&L) \# Pairs (\# M&L Pairs) 1.00 0.86 0.86 0.97 0.85 0.85 0.85 0.81 0.99 0.82 0.94 0.99 0.90 0.61 0.57 0.56 0.50 0.52 0.84 0.71 0.96 0.93 0.82 0.82 0.85 0.88 0.85 0.82 0.78 0.79 120 (140) 1440 (1680) 720 (840) 400 (400) 19440 (22400) 9600 (11200) 19680 (22400) 19680 (224… [find](https://scholar.google.com/scholar?q=BERT+Large+LSTM+%28M%26L%29+Humans+%28M%26L%29+%5C%23+Pairs+%28%5C%23+M%26L+Pairs%29+1%2E00+0%2E86+0%2E86+0%2E97+0%2E85+0%2E85+0%2E85+0%2E81+0%2E99+0%2E82+0%2E94+0%2E99+0%2E90+0%2E61+0%2E57+0%2E56+0%2E50+0%2E52+0%2E84+0%2E71+0%2E96+0%2E93+0%2E82+0%2E82+0%2E85+0%2E88+0%2E85+0%2E82+0) 3. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencie Transactions of the Association for Computational Linguistics, 4:521–535. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2C+Emmanuel+Dupoux%2C+and+Yoav+Goldberg%2E+2016%2E+Assessing+the+ability+of+LSTMs+to+learn+syntax%2Dsensitive+dependencie+Transactions+of+the+Association+for+Computational+Linguistics%2C+4%3A521%E2%80%93535%2E) 4. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1192%E2%80%931202%2E) 5. Gongbo Tang, Mathias Mller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? a targeted evaluation of neural machine translatio In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4263–4272. [find](https://scholar.google.com/scholar?q=Gongbo+Tang%2C+Mathias+Mller%2C+Annette+Rios%2C+and+Rico+Sennrich%2E+2018%2E+Why+self%2Dattention%3F+a+targeted+evaluation+of+neural+machine+translatio+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+) 6. Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731–4736. [find](https://scholar.google.com/scholar?q=Ke+Tran%2C+Arianna+Bisazza%2C+and+Christof+Monz%2E+2018%2E+The+importance+of+being+recurrent+for+modeling+hierarchical+structure+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+)