---
title: "A Multiscale Visualization of Attention in the — Vig 2019"
description: "Transformer models such as BERT and GPT-2 rely entirely on attention, which can make them easier to inspect."
authors: ["Jesse Vig"]
year: 2019
venue: "arXiv:1906.05714"
source: "https://arxiv.org/abs/1906.05714"
doi: "10.48550/arxiv.1906.05714"
arxiv: "1906.05714"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vig-2019-multiscale-visualization-attention-transformer-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vig-2019-multiscale-visualization-attention-transformer-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / A Multiscale Visualization of Attention in the Transformer Model

# A Multiscale Visualization of Attention in the Transformer Model

**Authors:** [Jesse Vig](https://md.research.elamur.ai/free/en/authors/jesse-vig/) · **Year:** 2019 · **Venue:** arXiv:1906.05714 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1906.05714) · [PDF](https://arxiv.org/pdf/1906.05714) · [DOI 10.48550/arxiv.1906.05714](https://doi.org/10.48550/arxiv.1906.05714) · [arXiv 1906.05714](https://arxiv.org/abs/1906.05714) · [OpenAlex](https://openalex.org/W2949202705)

**Citations:** 555 citations (openalex-snapshot · 2026-10-14) · [153](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Transformer models such as BERT and GPT-2 rely entirely on attention, which can make them easier to inspect. But the number of layer/head combinations (384 per input for BERT-Large) makes raw attention hard to read. Vig extends the multi-head attention visualizer of Jones (2017) into an open-source tool, bertviz, that shows attention at three levels of granularity. The attention-head view draws token-to-token links for selected heads in one layer. The model view arranges thumbnails of every layer and head in a grid. The neuron view breaks down how query and key vector components combine into attention scores. The tool was adapted from the original encoder-decoder setting to encoder-only BERT and decoder-only GPT-2. Three case studies show what it can do: revealing gender-dependent coreference in GPT-2, finding BERT heads that link two sentences (useful for paraphrase detection), and tracing a distance-decaying attention pattern to a few neurons. The work offers a practical interpretability resource and a starting point for targeted model interventions.

## Key points
- The tool is available at github.com/jessevig/bertviz. The demos use the GPT-2 small model and BERT-base (uncased).
- The neuron view shows several columns for a selected token:
  - its query vector,
  - the key vectors of the other tokens,
  - their element-wise product,
  - the dot product,
  - the resulting softmax attention.
- Bias case: with greedy decoding, GPT-2 resolves "She" to the nurse and "He" to the doctor in otherwise identical prompts. One attention head mirrors this gendered pattern.
- In the BERT model view, heads that connect Sentence A and Sentence B have a recognizable cross-hatch shape (e.g., layer 3, head 0).
- In GPT-2 layer 1, head 10, attention fades smoothly with distance, largely independent of content. A small set of neuron positions drives this effect, suggesting that the decay rate could be adjusted to control context length.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

153 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 146 not yet.

- [Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing](https://md.research.elamur.ai/free/en/papers/lin-2020-bridging-textual-tabular-data-cross/)
- [GPT Understands, Too](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://md.research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/)
- [Explore Political Discourse with Transformers. Emergent Paradigmatic and Syntagmatic Representations.](https://md.research.elamur.ai/free/en/papers/vanni-2026-explore-political-discourse-transformers-emergent/)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/)
- [Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders](https://md.research.elamur.ai/free/en/papers/wang-2020-two-are-better-than-one/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{vig2019multiscale,
  title = {{A Multiscale Visualization of Attention in the Transformer Model}},
  author = {Vig, Jesse},
  year = {2019},
  doi = {10.48550/arxiv.1906.05714},
  eprint = {1906.05714},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1906.05714}
}
```

- [← previous A Modular Architecture for Unsupervised Sarcasm Generation](https://md.research.elamur.ai/free/en/papers/mishra-2019-modular-architecture-unsupervised-sarcasm-generation/)
- [next → A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling](https://md.research.elamur.ai/free/en/papers/e-2019-novel-bi-directional-interrelated-model/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proc%2E+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Identifying and controlling important neurons in neural machine translation. In Proc. ICLR. [find](https://scholar.google.com/scholar?q=Anthony+Bau%2C+Yonatan+Belinkov%2C+Hassan+Sajjad%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+and+James+Glass%2E+2019%2E+Identifying+and+controlling+important+neurons+in+neural+machine+translation%2E+In+Proc%2E+ICLR%2E)
3. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. TACL. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+James+Glass%2E+2019%2E+Analysis+methods+in+neural+language+processing%3A+A+survey%2E+TACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv Computation and Language. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding%2E+ArXiv+Computation+and+Language%2E)
5. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. CoRR, abs/1902.10186. [find](https://scholar.google.com/scholar?q=Sarthak+Jain+and+Byron+C%2E+Wallace%2E+2019%2E+Attention+is+not+explanation%2E+CoRR%2C+abs%2F1902%2E10186%2E) [our card](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/)
6. Llion Jones. 2017. Tensor2tensor transformer visualization. [https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/visualization](https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/visualization). [↗](https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/visualization)
7. Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive visualization and manipulation of attention-based neural machine translation. In EMNLP: System Demonstrations. [find](https://scholar.google.com/scholar?q=Jaesong+Lee%2C+Joong%2DHwi+Shin%2C+and+Jun%2DSeok+Kim%2E+2017%2E+Interactive+visualization+and+manipulation+of+attention%2Dbased+neural+machine+translation%2E+In+EMNLP%3A+System+Demonstrations%2E)
8. Shusen Liu, Tao Li, Zhimin Li, Vivek Srikumar, Valerio Pascucci, and Peer-Timo Bremer. 2018. Visual interrogation of attention-based models for natural language inference and machine comprehension. In EMNLP: System Demonstrations. [find](https://scholar.google.com/scholar?q=Shusen+Liu%2C+Tao+Li%2C+Zhimin+Li%2C+Vivek+Srikumar%2C+Valerio+Pascucci%2C+and+Peer%2DTimo+Bremer%2E+2018%2E+Visual+interrogation+of+attention%2Dbased+models+for+natural+language+inference+and+machine+comprehension%2E+In)
9. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2018. Gender bias in neural natural language processing. CoRR, abs/1807.11714. [find](https://scholar.google.com/scholar?q=Kaiji+Lu%2C+Piotr+Mardziel%2C+Fangjing+Wu%2C+Preetam+Amancharla%2C+and+Anupam+Datta%2E+2018%2E+Gender+bias+in+neural+natural+language+processing%2E+CoRR%2C+abs%2F1807%2E11714%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2018-gender-bias-neural-natural-language/)
10. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2E)
11. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In EMNLP Workshop: BlackboxNLP. [find](https://scholar.google.com/scholar?q=Alessandro+Raganato+and+Jo%CC%88rg+Tiedemann%2E+2018%2E+An+analysis+of+encoder+representations+in+transformer%2Dbased+machine+translation%2E+In+EMNLP+Workshop%3A+BlackboxNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/raganato-2018-analysis-encoder-representations-transformer-based/)
12. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In Proc. ICLR. [find](https://scholar.google.com/scholar?q=Tim+Rockta%CC%88schel%2C+Edward+Grefenstette%2C+Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+and+Phil+Blunsom%2E+2016%2E+Reasoning+about+entailment+with+neural+attention%2E+In+Proc%2E+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/rocktaschel-2015-reasoning-about-entailment-neural-attention/)
13. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proc. EMNLP. [find](https://scholar.google.com/scholar?q=Alexander+M%2E+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+Proc%2E+EMNLP%2E)
14. H. Strobelt, S. Gehrmann, M. Behrisch, A. Perer, H. Pfister, and A. M. Rush. 2018. Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models. ArXiv e-prints. [find](https://scholar.google.com/scholar?q=H%2E+Strobelt%2C+S%2E+Gehrmann%2C+M%2E+Behrisch%2C+A%2E+Perer%2C+H%2E+Pfister%2C+and+A%2E+M%2E+Rush%2E+2018%2E+Seq2Seq%2DVis%3A+A+Visual+Debugging+Tool+for+Sequence%2Dto%2DSequence+Models%2E+ArXiv+e%2Dprints%2E)
15. Edward Tufte. 1990. Envisioning Information. Graphics Press, Cheshire, CT, USA. [find](https://scholar.google.com/scholar?q=Edward+Tufte%2E+1990%2E+Envisioning+Information%2E+Graphics+Press%2C+Cheshire%2C+CT%2C+USA%2E)
16. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Samy+Bengio%2C+Eugene+Brevdo%2C+Francois+Chollet%2C+Aidan+N%2E+Gomez%2C+Stephan+Gouws%2C+Llion+Jones%2C+%C5%81ukasz+Kaiser%2C+Nal+Kalchbrenner%2C+Niki+Parmar%2C+Ryan+Sepassi%2C+Noam+Shazeer%2C+and+Jakob+Uszkoreit%2E) [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2018-tensor2tensor-neural-machine-translation/)
17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+arXiv+preprint+arXiv%3A1706%2E03762%2E)
18. Jesse Vig. 2019. BertViz: A tool for visualizing multi-head self-attention in the BERT model. In ICLR Workshop: Debugging Machine Learning Models. [find](https://scholar.google.com/scholar?q=Jesse+Vig%2E+2019%2E+BertViz%3A+A+tool+for+visualizing+multi%2Dhead+self%2Dattention+in+the+BERT+model%2E+In+ICLR+Workshop%3A+Debugging+Machine+Learning+Models%2E)
19. Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In ACL Workshop: BlackboxNLP. [find](https://scholar.google.com/scholar?q=Jesse+Vig+and+Yonatan+Belinkov%2E+2019%2E+Analyzing+the+structure+of+attention+in+a+transformer+language+model%2E+In+ACL+Workshop%3A+BlackboxNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/)
20. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418. [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+David+Talbot%2C+Fedor+Moiseev%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2019%2E+Analyzing+multi%2Dhead+self%2Dattention%3A+Specialized+heads+do+the+heavy+lifting%2C+the+rest+can+be+pruned%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/)
21. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+2018%2E+Gender+bias+in+coreference+resolution%3A+Evaluation+and+debiasing+methods%2E+In+NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/)