--- title: "A Primer in BERTology — Rogers 2020" description: "BERT quickly became a default baseline across NLP, but it was still unclear why it worked so well, and that made principled improvements to the…" authors: ["Anna Rogers", "Olga Kovaleva", "Anna Rumshisky"] year: 2020 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2020.tacl-1.54/" doi: "10.1162/tacl_a_00349" arxiv: "2002.12327" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / A Primer in BERTology: What We Know About How BERT Works # A Primer in BERTology: What We Know About How BERT Works **Authors:** [Anna Rogers](https://md.research.elamur.ai/free/en/authors/anna-rogers/), [Olga Kovaleva](https://md.research.elamur.ai/free/en/authors/olga-kovaleva/), [Anna Rumshisky](https://md.research.elamur.ai/free/en/authors/anna-rumshisky/) · **Year:** 2020 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.tacl-1.54/) · [PDF](https://aclanthology.org/2020.tacl-1.54.pdf) · [DOI 10.1162/tacl\_a\_00349](https://doi.org/10.1162/tacl_a_00349) · [arXiv 2002.12327](https://arxiv.org/abs/2002.12327) · [OpenAlex](https://openalex.org/W3006881356) **Citations:** 1527 citations (openalex-snapshot · 2026-10-14) · [600](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary BERT quickly became a default baseline across NLP, but it was still unclear why it worked so well, and that made principled improvements to the architecture hard. This TACL 2020 survey brings together findings from more than 150 analysis papers. It briefly recaps the model: stacked Transformer encoder layers, wordpiece input, pre-training with MLM and NSP, then task-specific fine-tuning. It then sorts the evidence on what syntactic, semantic and world knowledge BERT learns, how researchers detect it (cloze-style MLM probes, attention-weight analysis, probing classifiers) and where in the network that knowledge seems to sit. Later sections cover proposed changes to training objectives, architecture and fine-tuning, the evidence that the model is overparameterized, and compression methods. The overall finding is that BERT stores a lot of linguistic and factual information but often fails to use it robustly. It relies on shallow heuristics and struggles with reasoning, numbers and negation. The paper ends with a research agenda, which makes it a useful starting point for anyone analysing or adapting BERT-style models, including for Ukrainian NLP. ## Key points - Syntactic structure can be recovered from BERT's token embeddings: Hewitt and Manning learned linear transformations that reconstructed dependencies on Penn Treebank data. Full parse trees could not be extracted from individual attention heads, though. - BERT is fragile in ways that undercut its apparent linguistic skill. Its outputs barely change under shuffled word order or truncated input (Ettinger, 2019), and swapping names in a coreference task changes 85% of its predictions. - Through cloze queries, vanilla BERT matches knowledge-base methods for some relation types (Petroni et al., 2019, LAMA). However, it cannot reason over those facts, and some of its apparent world knowledge comes from stereotypical associations. - Embeddings become more context-specific in later layers and fill a narrow, anisotropic cone in vector space. Many heads and layers can be disabled with little loss, and "winning" lottery-ticket subnetworks exist. - The suggested research directions include harder benchmarks that need verbal reasoning, behavioural stress tests such as Checklist (Ribeiro et al., 2020), and caution when using pruning or ablation as analysis tools, because knowledge may be spread across the network or duplicated. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 600 papers of our library cite it (of the 140673 whose bibliographies we read); 34 of them have a page here, 566 not yet. - [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/) - [Comparing Native and Learner Englishes Using a Large Pre-trained Language Model](https://md.research.elamur.ai/free/en/papers/aoyama-2022-comparing-native-learner-englishes-large/) - [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://md.research.elamur.ai/free/en/papers/arora-2023-probing-pre-trained-language-models/) - [Probing Classifiers: Promises, Shortcomings, and Advances](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/) - [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://md.research.elamur.ai/free/en/papers/bianchi-2021-pre-training-is-hot-topic/) - [Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings](https://md.research.elamur.ai/free/en/papers/bommasani-2020-interpreting-pretrained-contextualized-representations-reductions/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-paragraph-level-rationale-extraction-through/) - [Analyzing Transformers in Embedding Space](https://md.research.elamur.ai/free/en/papers/dar-2023-analyzing-transformers-embedding-space/) - [Context Matters: A Pragmatic Study of PLMs' Negation Understanding](https://md.research.elamur.ai/free/en/papers/gubelmann-2022-context-matters-pragmatic-study-plms/) - [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/) - [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://md.research.elamur.ai/free/en/papers/hedderich-2021-survey-recent-approaches-natural-language/) - [Probing Contextual Language Models for Common Ground with Visual Representations](https://md.research.elamur.ai/free/en/papers/ilharco-2021-probing-contextual-language-models-common/) - [Attention is Not Only a Weight: Analyzing Transformers with Vector Norms](https://md.research.elamur.ai/free/en/papers/kobayashi-2020-attention-is-not-only-weight/) - [Pre-training via Paraphrasing](https://md.research.elamur.ai/free/en/papers/lewis-2020-pre-training-paraphrasing/) - [What Happens To BERT Embeddings During Fine-tuning?](https://md.research.elamur.ai/free/en/papers/merchant-2020-what-happens-bert-embeddings-during/) - [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://md.research.elamur.ai/free/en/papers/miletic-2024-semantics-multiword-expressions-transformer-based/) - [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/) - [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://md.research.elamur.ai/free/en/papers/park-2023-linear-representation-hypothesis-geometry-large/) - [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/) - [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/) - [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://md.research.elamur.ai/free/en/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/) - [Prompting Instruction-tuned LLMs for Semantic Similarity Values](https://md.research.elamur.ai/free/en/papers/snelder-2026-prompting-instruction-tuned-llms-semantic/) - [Entropy-aware Masking for Masked Language Modeling](https://md.research.elamur.ai/free/en/papers/srinivasagan-2026-entropy-aware-masking-masked-language/) - [Extracting Latent Steering Vectors from Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/subramani-2022-extracting-latent-steering-vectors-pretrained/) - [Explore Political Discourse with Transformers. Emergent Paradigmatic and Syntagmatic Representations.](https://md.research.elamur.ai/free/en/papers/vanni-2026-explore-political-discourse-transformers-emergent/) - [Probing Pretrained Language Models for Lexical Semantics](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/) - [UniTE: Unified Translation Evaluation](https://md.research.elamur.ai/free/en/papers/wan-2022-unite-unified-translation-evaluation/) - [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://md.research.elamur.ai/free/en/papers/wang-2023-pandalm-automatic-evaluation-benchmark-llm/) - [NegNLI-BR: A Brazilian Portuguese Benchmark for Negation in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/westhelle-2026-negnli-br-brazilian-portuguese-benchmark/) - [Knowledge Conflicts for LLMs: A Survey](https://md.research.elamur.ai/free/en/papers/xu-2024-knowledge-conflicts-llms-survey/) - [Explainability for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/) - [Factual Probing Is MASK: Learning vs. Learning to Recall](https://md.research.elamur.ai/free/en/papers/zhong-2021-factual-probing-is-mask-learning/) - [A Survey on Model Compression for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2024-survey-model-compression-large-language/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{rogers2020primer, title = {{A Primer in BERTology: What We Know About How BERT Works}}, author = {Rogers, Anna and Kovaleva, Olga and Rumshisky, Anna}, journal = {Transactions of the Association for Computational Linguistics}, year = {2020}, doi = {10.1162/tacl\_a\_00349}, eprint = {2002.12327}, archivePrefix = {arXiv}, url = {https://doi.org/10.1162/tacl_a_00349} } ``` - [← previous A preference semantics for imperatives](https://md.research.elamur.ai/free/en/papers/starr-2020-preference-semantics-imperatives/) - [next → A Report on the 2020 Sarcasm Detection Shared Task](https://md.research.elamur.ai/free/en/papers/ghosh-2020-report-2020-sarcasm-detection-shared/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Edward Guo. 2019. Knowledge Distillation from Internal Representations. arXiv preprint arXiv:1910.03723. — [arXiv:1910.03723](https://arxiv.org/abs/1910.03723) [find](https://scholar.google.com/scholar?q=Gustavo+Aguilar%2C+Yuan+Ling%2C+Yu+Zhang%2C+Benjamin+Yao%2C+Xing+Fan%2C+and+Edward+Guo%2E+2019%2E+Knowledge+Distillation+from+Internal+Representations%2E+arXiv+preprint+arXiv%3A1910%2E03723%2E+%E2%80%94+%5BarXiv%3A1910%2E03723%5D%28https%3A%2F%2F) 2. Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled Contextualized Embeddings for Named Entity Recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p… [find](https://scholar.google.com/scholar?q=Alan+Akbik%2C+Tanja+Bergmann%2C+and+Roland+Vollgraf%2E+2019%2E+Pooled+Contextualized+Embeddings+for+Named+Entity+Recognition%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Associa) [our card](https://md.research.elamur.ai/free/en/papers/akbik-2019-pooled-contextualized-embeddings-named-entity/)