--- title: "What BERT Is Not — Ettinger 2020" description: "Language-model pre-training clearly helps downstream NLP tasks, but it remains unclear which linguistic abilities models such as BERT actually acquire." authors: ["Allyson Ettinger"] year: 2020 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2020.tacl-1.3/" doi: "10.1162/tacl_a_00298" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models # What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models **Authors:** [Allyson Ettinger](https://md.research.elamur.ai/free/en/authors/allyson-ettinger/) · **Year:** 2020 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.tacl-1.3/) · [PDF](https://aclanthology.org/2020.tacl-1.3.pdf) · [DOI 10.1162/tacl\_a\_00298](https://doi.org/10.1162/tacl_a_00298) · [OpenAlex](https://openalex.org/W2965647350) **Citations:** 510 citations (openalex-snapshot · 2026-10-14) · [333](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Language-model pre-training clearly helps downstream NLP tasks, but it remains unclear which linguistic abilities models such as BERT actually acquire. Ettinger addresses this with a diagnostic suite adapted from psycholinguistic experiments. The selected experiments are ones where human cloze judgments and N400 brain responses diverge. The tests probe models through word prediction in context, so no fine-tuning or representation probing is needed. They cover areas beyond syntax: commonsense/pragmatic inference (CPRAG-102), semantic roles and event knowledge (ROLE-88), and category membership with negation (NEG-136). Each test measures both prediction accuracy and whether good completions receive higher probability than bad ones. Applied to BERT BASE and BERT LARGE, the suite yields a mixed picture: - BERT retrieves hypernyms very well. - It mostly separates good completions from same-category alternatives and from role reversals, though with much smaller margins than humans. - It struggles with difficult inferences and role-based event prediction. - It largely ignores negation. These findings suggest that a purely predictive training objective may not teach meaning-dependent phenomena like truth reversal. ## Key points - Three diagnostic sets are released with code (github.com/aetting/lm-diagnostics): CPRAG-102, ROLE-88 and NEG-136. NEG-136 has two variants, NEG-136-SIMP and NEG-136-NAT. - On CPRAG-102, both BERT models place the best completion among their top 5 predictions in roughly half of the items. Perturbation tests show the models depend on word order and full context, but some correct answers seem to exploit shortcuts. - On ROLE-88, BERT LARGE matches the top human prediction in only 37.5% of items. Both models usually still rank good completions above role reversals. - On NEG-136-SIMP, BERT favours the true completion in 100% of affirmative sentences but in 0% of negated ones, defaulting to the subject's category. - With more natural negated sentences (NEG-136-NAT), true statements are preferred in 75% of items for BERT BASE and 87.5% for BERT LARGE. Less natural items reach only 37.5%, a gap the author attributes mainly to training-data frequency. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 333 papers of our library cite it (of the 140673 whose bibliographies we read); 28 of them have a page here, 305 not yet. - [Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models](https://md.research.elamur.ai/free/en/papers/askari-2025-are-babylms-deaf-gricean-maxims/) - [Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings](https://md.research.elamur.ai/free/en/papers/bommasani-2020-interpreting-pretrained-contextualized-representations-reductions/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?](https://md.research.elamur.ai/free/en/papers/eacl-2023-shmidman-do-pretrained-contextual-language-models/) - [Probing for idiomaticity in vector space models](https://md.research.elamur.ai/free/en/papers/garcia-2021-probing-idiomaticity-vector-space-models/) - [Context Matters: A Pragmatic Study of PLMs' Negation Understanding](https://md.research.elamur.ai/free/en/papers/gubelmann-2022-context-matters-pragmatic-study-plms/) - [A Fine-Grained Analysis of BERTScore](https://md.research.elamur.ai/free/en/papers/hanna-2021-fine-grained-analysis-bertscore/) - [Are Pretrained Language Models Symbolic Reasoners over Knowledge?](https://md.research.elamur.ai/free/en/papers/kassner-2020-are-pretrained-language-models-symbolic/) - [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://md.research.elamur.ai/free/en/papers/kassner-2020-negated-misprimed-probes-pretrained-language/) - [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://md.research.elamur.ai/free/en/papers/lex-2021-hofmann-superbizarre-derivational/) - [How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://md.research.elamur.ai/free/en/papers/linzen-2020-how-can-we-accelerate-progress/) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) - [On Quick Kisses and How to Make Them Count: A Study on Event Construal in Light Verb Constructions with BERT](https://md.research.elamur.ai/free/en/papers/liu-2023-quick-kisses-how-make-them/) - [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://md.research.elamur.ai/free/en/papers/miletic-2024-semantics-multiword-expressions-transformer-based/) - [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/) - [HONEST: Measuring Hurtful Sentence Completion in Language Models](https://md.research.elamur.ai/free/en/papers/nozza-2021-honest-measuring-hurtful-sentence-completion/) - [NOPE: A Corpus of Naturally-Occurring Presuppositions in English](https://md.research.elamur.ai/free/en/papers/parrish-2021-nope-corpus-naturally-occurring-presuppositions/) - [True Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/) - [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/) - [HateCheck: Functional Tests for Hate Speech Detection Models](https://md.research.elamur.ai/free/en/papers/rottger-2021-hatecheck-functional-tests-hate-speech/) - [Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/) - [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/) - [When Your Language Model Cannot Even Do Determiners Right: Probing for Anti-Presuppositions and the Maximize Presupposition! Principle](https://md.research.elamur.ai/free/en/papers/sieker-2023-when-your-language-model-cannot/) - [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://md.research.elamur.ai/free/en/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/) - [Can a Remedy Find a Researcher? Exploring the Development of Semantic Knowledge in Italian BabyLMs](https://md.research.elamur.ai/free/en/papers/suozzi-2026-can-remedy-find-researcher-exploring/) - [oLMpics -- On what Language Model Pre-training Captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/) - [Evaluating Text Generation from Discourse Representation Structures](https://md.research.elamur.ai/free/en/papers/wang-2021-evaluating-text-generation-discourse-representation/) - [SemEval-2022 Task 3: PreTENS — Evaluating Neural Networks on Presuppositional Semantic Knowledge](https://md.research.elamur.ai/free/en/papers/zamparelli-2022-semeval-2022-task-3-pretens/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{ettinger2020what, title = {{What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models}}, author = {Ettinger, Allyson}, journal = {Transactions of the Association for Computational Linguistics}, year = {2020}, doi = {10.1162/tacl\_a\_00298}, url = {https://doi.org/10.1162/tacl_a_00298} } ``` - [← previous Weight Poisoning Attacks on Pretrained Models](https://md.research.elamur.ai/free/en/papers/kurita-2020-weight-poisoning-attacks-pretrained-models/) - [next → What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://md.research.elamur.ai/free/en/papers/nie-2020-what-can-we-learn-collective/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) ## References 2 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yossi+Adi%2C+Einat+Kermany%2C+Yonatan+Belinkov%2C+Ofer+Lavi%2C+and+Yoav+Goldberg%2E+2016%2E+Fine%2Dgrained+analysis+of+sentence+embeddings+using+auxiliary+prediction+tasks%2E+International+Conference+on+Learning+Repr) [our card](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/) 2. Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proce… [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Mona+Diab%2C+Daniel+Cer%2C+and+Aitor+Gonzalez%2DAgirre%2E+2012%2E+SemEval%2D2012+task+6%3A+A+pilot+on+semantic+textual+similarity%2E+In+Proceedings+of+the+First+Joint+Conference+on+Lexical+and+Computati) [our card](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/)