---
title: "Visualizing and Understanding the Effectiveness of BERT (2019)"
description: "Hao et al. study this using BERT-large. 35 references, cited by 55 papers here. Among them: 166."
authors: ["Yaru Hao", "Li Dong", "Furu Wei", "Ke Xu"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1424/"
doi: "10.18653/v1/d19-1424"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hao-2019-visualizing-understanding-effectiveness-bert/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hao-2019-visualizing-understanding-effectiveness-bert/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Visualizing and Understanding the Effectiveness of BERT

# Visualizing and Understanding the Effectiveness of BERT

**Authors:** [Yaru Hao](https://md.research.elamur.ai/free/en/authors/yaru-hao/), [Li Dong](https://md.research.elamur.ai/free/en/authors/li-dong/), [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) and [Ke Xu](https://md.research.elamur.ai/free/en/authors/ke-xu/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1424/) · [PDF](https://aclanthology.org/D19-1424.pdf) · [DOI 10.18653/v1/d19-1424](https://doi.org/10.18653/v1/d19-1424) · [OpenAlex](https://openalex.org/W2971033911)

**Citations:** 166 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/hao-2019-visualizing-understanding-effectiveness-bert/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Pre-training followed by fine-tuning works well across NLP tasks, but it has not been clear why it trains more easily and generalizes better. Hao et al. study this using BERT-large. They compare fine-tuning the pre-trained weights with training the same architecture from random initialization on four GLUE datasets: MNLI, RTE, SST-2 and MRPC.

They use three visualizations:
- **1D loss curves**, interpolated between the starting and final parameters.
- **2D loss surfaces**, whose second axis is the fine-tuning direction from a different dataset, rescaled to the same norm.
- **Optimization trajectories**, with each epoch projected onto that 2D plane.

Pre-trained initialization gives wider, flatter optima and smoother descent paths, so convergence is faster. Fine-tuning also resists overfitting even though BERT is heavily over-parameterized. The training loss surface closely tracks the generalization error surface, which helps explain BERT's strong results on unseen data. A layer-wise analysis shows that layers near the input change little during fine-tuning and carry transferable knowledge, while upper layers absorb task-specific information. The findings point toward fine-tuning methods that deliberately seek flat minima.

## Key points
- **Setup:** BERT-large with the cased tokenizer and the Adam optimizer, fine-tuned for 3–5 epochs. Training from scratch took longer, running 8 epochs on SST-2 and 16 on each of the remaining datasets. Data splits follow GLUE (Wang et al., 2019), and accuracy is the metric.
- **Loss landscapes:** both interpolation coefficients range over [−4, 4], with 40 sampled points per axis. Task-specific layers are frozen, so only the BERT encoder parameters are perturbed.
- **Fine-tuned baselines:** dev accuracy is 86.54 on MNLI, 75.45 on RTE, 94.04 on SST-2 and 90.20 on MRPC.
- **Layer rollback (low layers):** resetting layers 0–7 to their pre-trained values costs little and slightly improves MNLI (+0.19).
- **Layer rollback (high layers):** resetting layers 16–23 is damaging, for example −53.88 on MNLI and −34.75 on SST-2.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 240 works · 123,537 citations
- [Beihang University](https://md.research.elamur.ai/free/en/universities/beihang-university/) — University, Beijing, China · 27 works · 8,089 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Yaru Hao, Li Dong, Furu Wei and Ke Xu. 2019. Visualizing and Understanding the Effectiveness of BERT. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{hao2019visualizing,
  title = {{Visualizing and Understanding the Effectiveness of BERT}},
  author = {Yaru Hao, Li Dong, Furu Wei and Ke Xu},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1424},
  url = {https://doi.org/10.18653/v1/d19-1424}
}
```

- [← previous Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://md.research.elamur.ai/free/en/papers/wieting-2019-beyond-bleu-training-neural-machine/)
- [next → Head-Driven Phrase Structure Grammar Parsing on Penn Treebank](https://md.research.elamur.ai/free/en/papers/zhou-2019-head-driven-phrase-structure-grammar/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

35 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019. Cloze-driven pretraining of self-attention networks. arXiv preprint arXiv:1903.07785. [find](https://scholar.google.com/scholar?q=Alexei+Baevski%2C+Sergey+Edunov%2C+Yinhan+Liu%2C+Luke+Zettlemoyer%2C+and+Michael+Auli%2E+2019%2E+Cloze%2Ddriven+pretraining+of+self%2Dattention+networks%2E+arXiv+preprint+arXiv%3A1903%2E07785%2E)
2. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. [find](https://scholar.google.com/scholar?q=Roy+Bar%2DHaim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+and+Danilo+Giampiccolo%2E+2006%2E+The+second+PASCAL+recognising+textual+entailment+challenge%2E+In+Proceedings+of+the+Second+PASCAL+Challenges+Workshop+on+Re)
3. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. In In Proc Text Analysis Conference (TAC09. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+Danilo+Giampiccolo%2C+and+Bernardo+Magnini%2E+2009%2E+The+fifth+PASCAL+recognizing+textual+entailment+challenge%2E+In+In+Proc+Text+Analysis+Conference+%28TAC09%2E)
4. Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. 2017. Entropy-SGD: Biasing gradient descent into wide valleys. [find](https://scholar.google.com/scholar?q=Pratik+Chaudhari%2C+Anna+Choromanska%2C+Stefano+Soatto%2C+Yann+LeCun%2C+Carlo+Baldassi%2C+Christian+Borgs%2C+Jennifer+Chayes%2C+Levent+Sagun%2C+and+Riccardo+Zecchina%2E+2017%2E+Entropy%2DSGD%3A+Biasing+gradient+descent+into+)
5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW’05,… [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Proceedings+of+the+First+International+Conference+on+Machine+Learning+Challenges%3A+Evaluati)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805.
7. William B Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
8. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. [Unified language model pre-training for natural language understanding and generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). arXiv preprint arXiv:1905.03197.
9. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1–9, Prague. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+and+Bill+Dolan%2E+2007%2E+The+third+PASCAL+recognizing+textual+entailment+challenge%2E+In+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Pa)
10. Yoav Goldberg. 2019. [Assessing BERT’s syntactic abilities](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/). CoRR, abs/1901.05287.
11. Ian J. Goodfellow and Oriol Vinyals. 2015. Qualitatively characterizing neural network optimization problems. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Ian+J%2E+Goodfellow+and+Oriol+Vinyals%2E+2015%2E+Qualitatively+characterizing+neural+network+optimization+problems%2E+In+3rd+International+Conference+on+Learning+Representations%2C+ICLR+2015%2C+San+Diego%2C+CA%2C+USA)
12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Flat minima. Neural Comput., 9(1):1–42. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Flat+minima%2E+Neural+Comput%2E%2C+9%281%29%3A1%E2%80%9342%2E)
13. Jeremy Howard and Sebastian Ruder. 2018a. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018a%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volu)
14. Jeremy Howard and Sebastian Ruder. 2018b. Universal language model fine-tuning for text classification. In ACL. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018b%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+ACL%2E+Association+for+Computational+Linguistics%2E)
15. Daniel Jiwoong Im, Michael Tao, and Kristin Branson. 2016. An empirical analysis of deep network loss surfaces. CoRR, abs/1612.04010. — [arXiv:1612.04010](https://arxiv.org/abs/1612.04010) [find](https://scholar.google.com/scholar?q=Daniel+Jiwoong+Im%2C+Michael+Tao%2C+and+Kristin+Branson%2E+2016%2E+An+empirical+analysis+of+deep+network+loss+surfaces%2E+CoRR%2C+abs%2F1612%2E04010%2E+%E2%80%94+%5BarXiv%3A1612%2E04010%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1612%2E04010%29)
16. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. [Averaging weights leads to wider optima and better generalization](https://md.research.elamur.ai/free/en/papers/izmailov-2018-averaging-weights-leads-wider-optima/). arXiv preprint arXiv:1803.05407.
17. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2016. On large-batch training for deep learning: Generalization gap and sharp minima. CoRR, abs/1609.04836. [find](https://scholar.google.com/scholar?q=Nitish+Shirish+Keskar%2C+Dheevatsa+Mudigere%2C+Jorge+Nocedal%2C+Mikhail+Smelyanskiy%2C+and+Ping+Tak+Peter+Tang%2E+2016%2E+On+large%2Dbatch+training+for+deep+learning%3A+Generalization+gap+and+sharp+minima%2E+CoRR%2C+abs%2F)
18. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, San Diego, CA. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+3rd+International+Conference+on+Learning+Representations%2C+San+Diego%2C+CA%2E)
19. Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. [LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better](https://md.research.elamur.ai/free/en/papers/kuncoro-2018-lstms-can-learn-syntax-sensitive/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
20. Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Hao+Li%2C+Zheng+Xu%2C+Gavin+Taylor%2C+Christoph+Studer%2C+and+Tom+Goldstein%2E+2018%2E+Visualizing+the+loss+landscape+of+neural+nets%2E+In+Neural+Information+Processing+Systems%2E)
21. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. [Assessing the ability of LSTMs to learn syntax-sensitive dependencies](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/). CoRR, abs/1611.01368.
22. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A. Smith. 2019a. [Linguistic knowledge and transferability of contextual representations](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/). CoRR, abs/1903.08855.
23. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b. [Multi-task deep neural networks for natural language understanding](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/). CoRR, abs/1901.11504.
24. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. [Learned in translation: Contextualized word vectors](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/). In Advances in Neural Information Processing Systems, pages 6297–6308.
25. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
27. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In International Conference on Learning Representations.
28. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
29. Chiyuan Zhang, Samy Bengio, and Yoram Singer. 2019. Are all layers created equal? arXiv preprint arXiv:1902.01996. — [arXiv:1902.01996](https://arxiv.org/abs/1902.01996) [find](https://scholar.google.com/scholar?q=Chiyuan+Zhang%2C+Samy+Bengio%2C+and+Yoram+Singer%2E+2019%2E+Are+all+layers+created+equal%3F+arXiv+preprint+arXiv%3A1902%2E01996%2E+%E2%80%94+%5BarXiv%3A1902%2E01996%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1902%2E01996%29)
30. Matthew Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? Adapting pretrained representations to diverse tasks. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Sebastian+Ruder%2C+and+Noah+A%2E+Smith%2E+2019%2E+To+tune+or+not+to+tune%3F+Adapting+pretrained+representations+to+diverse+tasks%2E)
31. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
32. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
33. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In)
34. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. [BERT rediscovers the classical NLP pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
35. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. [What do you learn from context? Probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/). In International Conference…

*Updated 24 October 2026*