---
title: "Revealing the Dark Secrets of BERT — Kovaleva 2019"
description: "BERT tops many NLP benchmarks, but it is unclear how it achieves this."
authors: ["Olga Kovaleva", "Alexey Romanov", "Anna Rogers", "Anna Rumshisky"]
year: 2019
venue: "arXiv:1908.08593"
source: "https://arxiv.org/abs/1908.08593"
doi: "10.48550/arxiv.1908.08593"
arxiv: "1908.08593"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / Revealing the Dark Secrets of BERT

# Revealing the Dark Secrets of BERT

**Authors:** [Olga Kovaleva](https://md.research.elamur.ai/free/en/authors/olga-kovaleva/), [Alexey Romanov](https://md.research.elamur.ai/free/en/authors/alexey-romanov/), [Anna Rogers](https://md.research.elamur.ai/free/en/authors/anna-rogers/), [Anna Rumshisky](https://md.research.elamur.ai/free/en/authors/anna-rumshisky/) · **Year:** 2019 · **Venue:** arXiv:1908.08593 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1908.08593) · [PDF](https://arxiv.org/pdf/1908.08593) · [DOI 10.48550/arxiv.1908.08593](https://doi.org/10.48550/arxiv.1908.08593) · [arXiv 1908.08593](https://arxiv.org/abs/1908.08593) · [OpenAlex](https://openalex.org/W2969624041)

**Citations:** 495 citations (OpenAlex, 2026-10-14) · [242](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary

BERT tops many NLP benchmarks, but it is unclear how it achieves this. Kovaleva et al. look at self-attention, the core component, to see what individual heads encode. They use bert-base-uncased (12 layers, 12 heads per layer, 110M parameters), both pre-trained and fine-tuned on seven GLUE tasks: MRPC, STS-B, SST-2, QQP, RTE, QNLI and MNLI. For each input they extract every head's attention matrix. They then sort these matrices into recurring visual types using manual labels and a CNN classifier. They also test whether some heads track FrameNet relations between predicates and core frame elements. Finally, they "switch off" heads by giving every token equal attention and measure how task scores change.

Only a few attention patterns appear, and they repeat across heads and tasks, which points to heavy redundancy. Removing single heads, or even whole layers, often raises accuracy rather than lowering it. This makes the work an early argument for pruning BERT and searching for leaner sub-architectures.

## Key points

- **Five attention types:** Vertical (focus on [CLS]/[SEP]), Diagonal (neighbouring tokens), Vertical+Diagonal, Block (within one sentence of a pair) and Heterogeneous.
  - About 400 hand-labelled maps trained an 8-layer CNN, which reached F1 0.86.
  - At most 32% (MRPC) to 61% (QQP) of heads fall into the potentially informative Heterogeneous class.
- **How heads are disabled:** each token's attention weights in a head are replaced with the uniform value 1/L, where L is sequence length.
- **Effect of disabling one head:** the best gain per task ranges from 0.1% (STS-B) to 1.2% (MRPC). For MRPC and RTE, removing a random head improves results on average.
- **Effect of disabling a whole layer:** removing layer 1 boosts RTE by 3.2% absolute. QNLI and MNLI lose up to 0.2%.
- **FrameNet heads:** two of the 144 heads appear to align with FrameNet relations, but neither matters for GLUE performance.
  - STS-B and RTE instead depend on two heads (layer 4 head 1, layer 2 head 12).
  - These heads emphasise words that occur in both input sentences.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Institutions

- [University of Massachusetts Lowell](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-lowell/) — University, Lowell, United States · 5 works · 3 446 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kovaleva2019revealing,
  title = {{Revealing the Dark Secrets of BERT}},
  author = {Kovaleva, Olga and Romanov, Alexey and Rogers, Anna and Rumshisky, Anna},
  year = {2019},
  doi = {10.48550/arxiv.1908.08593},
  eprint = {1908.08593},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.08593}
}
```

- [← previous Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://md.research.elamur.ai/free/en/papers/fabbri-2019-multi-news-large-scale-multi/)
- [next → Analysis Methods in Neural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

24 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019. Rethinking complex neural network architectures for document classification. In Proceedings of NAACL 2019: Conference of the North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ashutosh+Adhikari%2C+Achyudh+Ram%2C+Raphael+Tang%2C+and+Jimmy+Lin%2E+2019%2E+Rethinking+complex+neural+network+architectures+for+document+classification%2E+In+Proceedings+of+NAACL+2019%3A+Conference+of+the+North+Am)
2. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 86–90. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Collin+F+Baker%2C+Charles+J+Fillmore%2C+and+John+B+Lowe%2E+1998%2E+The+berkeley+framenet+project%2E+In+Proceedings+of+the+17th+international+conference+on+Computational+linguistics%2DVolume+1%2C+pages+86%E2%80%9390%2E+Associ) [our card](https://md.research.elamur.ai/free/en/papers/lex-1998-baker-berkeley-framenet/)
3. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+Inigo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+Semeval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+Proceedings+of+th) [our card](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
5. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). [find](https://scholar.google.com/scholar?q=William+B+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
6. Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+2018%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+arXiv+preprint+arXiv%3A1803%2E03635%2E)
7. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. arXiv preprint arXiv:1901.05287. [find](https://scholar.google.com/scholar?q=Yoav+Goldberg%2E+2019%2E+Assessing+bert%E2%80%99s+syntactic+abilities%2E+arXiv+preprint+arXiv%3A1901%2E05287%2E) [our card](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/)
8. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy. [find](https://scholar.google.com/scholar?q=Ganesh+Jawahar%2C+Beno%C4%B1%CC%82t+Sagot%2C+and+Djame%CC%81+Seddah%2E+2019%2E+What+does+BERT+learn+about+the+structure+of+language%3F+In+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+Florence%2C+I) [our card](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/)
9. Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A Smith. 2019. Linguistic knowledge and transferability of contextual representations. arXiv preprint arXiv:1903.08855. [find](https://scholar.google.com/scholar?q=Nelson+F+Liu%2C+Matt+Gardner%2C+Yonatan+Belinkov%2C+Matthew+Peters%2C+and+Noah+A+Smith%2E+2019%2E+Linguistic+knowledge+and+transferability+of+contextual+representations%2E+arXiv+preprint+arXiv%3A1903%2E08855%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/)
10. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? arXiv preprint arXiv:1905.10650. [find](https://scholar.google.com/scholar?q=Paul+Michel%2C+Omer+Levy%2C+and+Graham+Neubig%2E+2019%2E+Are+sixteen+heads+really+better+than+one%3F+arXiv+preprint+arXiv%3A1905%2E10650%2E) [our card](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/)
11. Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Time+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+with+unsupervised+learning%2E+Technical+report%2C+Technical+report%2C+OpenAI%2E)
12. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+arXiv+preprint+arXiv%3A1606%2E05250%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
13. Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference in the clinical domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1586–1596. [find](https://scholar.google.com/scholar?q=Alexey+Romanov+and+Chaitanya+Shivade%2E+2018%2E+Lessons+from+natural+language+inference+in+the+clinical+domain%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+p) [our card](https://md.research.elamur.ai/free/en/papers/romanov-2018-lessons-natural-language-inference-clinical/)
14. Carson T. Schutze. 1996. The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. University of Chicago Press. [find](https://scholar.google.com/scholar?q=Carson+T%2E+Schutze%2E+1996%2E+The+Empirical+Base+of+Linguistics%3A+Grammaticality+Judgments+and+Linguistic+Methodology%2E+University+of+Chicago+Press%2E)
15. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+)
16. Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? a targeted evaluation of neural machine translation architectures. arXiv preprint arXiv:1808.08946. [find](https://scholar.google.com/scholar?q=Gongbo+Tang%2C+Mathias+Mu%CC%88ller%2C+Annette+Rios%2C+and+Rico+Sennrich%2E+2018%2E+Why+self%2Dattention%3F+a+targeted+evaluation+of+neural+machine+translation+architectures%2E+arXiv+preprint+arXiv%3A1808%2E08946%2E)
17. Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. arXiv preprint arXiv:1803.03585. [find](https://scholar.google.com/scholar?q=Ke+Tran%2C+Arianna+Bisazza%2C+and+Christof+Monz%2E+2018%2E+The+importance+of+being+recurrent+for+modeling+hierarchical+structure%2E+arXiv+preprint+arXiv%3A1803%2E03585%2E)
18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processi) [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
19. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418. [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+David+Talbot%2C+Fedor+Moiseev%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2019%2E+Analyzing+multi%2Dhead+self%2Dattention%3A+Specialized+heads+do+the+heavy+lifting%2C+the+rest+can+be+pruned%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/)
20. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. arXiv:1905.00537 [cs]. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+SuperGLUE%3A+A+Stickier+Benchmark+for+General%2DPurpose+Language+Understan) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
21. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amapreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2018%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
22. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
23. Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.10430. [find](https://scholar.google.com/scholar?q=Felix+Wu%2C+Angela+Fan%2C+Alexei+Baevski%2C+Yann+N+Dauphin%2C+and+Michael+Auli%2E+2019%2E+Pay+less+attention+with+lightweight+and+dynamic+convolutions%2E+arXiv+preprint+arXiv%3A1901%2E10430%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-pay-less-attention-lightweight-dynamic/)
24. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320–3328. [find](https://scholar.google.com/scholar?q=Jason+Yosinski%2C+Jeff+Clune%2C+Yoshua+Bengio%2C+and+Hod+Lipson%2E+2014%2E+How+transferable+are+features+in+deep+neural+networks%3F+In+Advances+in+neural+information+processing+systems%2C+pages+3320%E2%80%933328%2E)