---
title: "Paying More Attention to Attention: Improving the Performance of"
description: "Knowledge transfer between CNNs usually means matching a teacher's outputs (knowledge… 26 references, cited by 104 papers here."
authors: ["Sergey Zagoruyko", "Nikos Komodakis"]
year: 2016
venue: "arXiv:1612.03928"
source: "https://arxiv.org/abs/1612.03928"
doi: "10.48550/arxiv.1612.03928"
arxiv: "1612.03928"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zagoruyko-2016-paying-more-attention-attention-improving/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zagoruyko-2016-paying-more-attention-attention-improving/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

# Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

**Authors:** [Sergey Zagoruyko](https://md.research.elamur.ai/free/en/authors/sergey-zagoruyko/) and Nikos Komodakis · **Year:** 2016 · **Venue:** arXiv:1612.03928 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1612.03928) · [PDF](https://arxiv.org/pdf/1612.03928) · [DOI 10.48550/arxiv.1612.03928](https://doi.org/10.48550/arxiv.1612.03928) · [arXiv 1612.03928](https://arxiv.org/abs/1612.03928) · [OpenAlex](https://openalex.org/W2561238782)

**Citations:** 1,438 citations (OpenAlex, 2026-10-14) · [104](https://md.research.elamur.ai/free/en/papers/zagoruyko-2016-paying-more-attention-attention-improving/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Knowledge transfer between CNNs usually means matching a teacher's outputs (knowledge distillation) or its intermediate features (FitNets). Zagoruyko and Komodakis instead transfer "attention": spatial maps showing which image regions a network relies on when deciding. A shallower or thinner student is trained with an extra loss that pulls its attention maps, taken at several depths, toward those of a stronger pretrained teacher. This adds no new weights. The authors study two kinds of maps. Activation-based maps aggregate absolute activation values across channels. Gradient-based maps come from gradients with respect to the input. Their visual analysis shows that more accurate networks have sharper, object-aligned attention. Mid-level maps highlight discriminative parts, while top layers cover whole objects. Experiments on CIFAR-10, an indoor Scenes dataset and ImageNet, with both residual and non-residual architectures, show consistent gains. Activation-based transfer outperforms matching full activations, and it can also be used alongside knowledge distillation. The method is a cheap, architecture-agnostic way to bring a strong model's behaviour into a smaller, faster one.

## Key points
- Activation-based attention maps use three channel-wise statistics of absolute activations: the plain sum, the sum of p-th powers, and the max of p-th powers (p > 1). Larger p concentrates weight on the most strongly activated regions.
- For visualization, the authors compared Network-In-Network (62% top-1), ResNet-34 (73%) and ResNet-101 (77.3%). Stronger models show attention peaks that weaker ones lack.
- On ImageNet, the attention transfer losses were added at epoch 60 of 100, with ResNet-34 as teacher. This cut ResNet-18's single-crop top-1/top-5 error from 30.4/10.8 to 29.3/10.0, against the teacher's 26.1/8.3.
- On CIFAR-10, transfer from WRN-16-2 to WRN-16-1 sped up convergence and improved final accuracy. It also outperformed full-activation transfer (F-ActT).
- Experiments ran in Torch, using fb.resnet.torch for ImageNet. Code and models are available at github.com/szagoruyko/attention-transfer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [École nationale des ponts et chaussées](https://md.research.elamur.ai/free/en/universities/ecole-nationale-des-ponts-et-chaussees/) — University, Champs-sur-Marne, France · 4 works · 3,074 citations
- [Laboratoire d'Informatique Gaspard-Monge](https://md.research.elamur.ai/free/en/labs/laboratoire-d-informatique-gaspard-monge/) — Research institute, Champs-sur-Marne, France · 3 works · 1,590 citations
- [Paris-Est Sup](https://md.research.elamur.ai/free/en/universities/paris-est-sup/) — University, Champs-sur-Marne, France · 3 works · 1,590 citations
- [Pôle de recherche et d'enseignement supérieur Université Paris-Est](https://md.research.elamur.ai/free/en/universities/pole-de-recherche-et-d-enseignement-superieur-universite-paris-est/) — University, Champs-sur-Marne, France · 3 works · 1,590 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Sergey Zagoruyko and Nikos Komodakis. 2016. Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer. arXiv:1612.03928.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zagoruyko2016paying,
  title = {{Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer}},
  author = {Sergey Zagoruyko and Nikos Komodakis},
  year = {2016},
  doi = {10.48550/arxiv.1612.03928},
  eprint = {1612.03928},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1612.03928}
}
```

- [← previous Improved Techniques for Training GANs](https://md.research.elamur.ai/free/en/papers/salimans-2016-improved-techniques-training-gans/)
- [next → Deep Reinforcement Learning for Dialogue Generation](https://md.research.elamur.ai/free/en/papers/li-2016-deep-reinforcement-learning-dialogue-generation/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). CoRR, abs/1409.0473, 2014. URL http://arxiv.org/abs/1409.0473. [↗](http://arxiv.org/abs/1409.0473)
2. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD, pp. 535–541, 2006. [find](https://scholar.google.com/scholar?q=Cristian+Bucila%2C+Rich+Caruana%2C+and+Alexandru+Niculescu%2DMizil%2E+Model+compression%2E+In+KDD%2C+pp%2E+535%E2%80%93541%2C+2006%2E)
3. ImageNet→CUB 28.5 27 (-1.5) 27 (-1.5) 26.5 ImageNet→Scenes 28.2 28.1 (-0.1) 27.1 (-1.1) 26 Taco S. Cohen and Max Welling. Group equivariant convolutional networks. abs/1602.07576, 2016. URL http://arxiv.org/abs/1602.07576. [↗](http://arxiv.org/abs/1602.07576)
4. Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. Learning where to attend with deep architectures for image tracking. Neural Computation, 2012. [find](https://scholar.google.com/scholar?q=Misha+Denil%2C+Loris+Bazzani%2C+Hugo+Larochelle%2C+and+Nando+de+Freitas%2E+Learning+where+to+attend+with+deep+architectures+for+image+tracking%2E+Neural+Computation%2C+2012%2E)
5. H. Drucker and Y LeCun. Improving generalization performance using double backpropagation. IEEE Transaction on Neural Networks, 3(6):991–997, 1992. [find](https://scholar.google.com/scholar?q=H%2E+Drucker+and+Y+LeCun%2E+Improving+generalization+performance+using+double+backpropagation%2E+IEEE+Transaction+on+Neural+Networks%2C+3%286%29%3A991%E2%80%93997%2C+1992%2E)
6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). CoRR, abs/1512.03385, 2015.
7. Geoffrey E. Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural networks. 2015. [find](https://scholar.google.com/scholar?q=Geoffrey+E%2E+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+Distilling+the+knowledge+in+a+neural+networks%2E+2015%2E)
8. Hugo Larochelle and Geoffrey E. Hinton. Learning to combine foveal glimpses with a third-order boltzmann machine. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 1243–1251. Curran Associates, Inc., 2… [find](https://scholar.google.com/scholar?q=Hugo+Larochelle+and+Geoffrey+E%2E+Hinton%2E+Learning+to+combine+foveal+glimpses+with+a+third%2Dorder+boltzmann+machine%2E+In+J%2E+D%2E+Lafferty%2C+C%2E+K%2E+I%2E+Williams%2C+J%2E+Shawe%2DTaylor%2C+R%2E+S%2E+Zemel%2C+and+A%2E+Culotta+%28ed)
9. Jimmy Ba Lei and Rich Caruana. Do deep nets really need to be deep? In NIPS, 2014. [find](https://scholar.google.com/scholar?q=Jimmy+Ba+Lei+and+Rich+Caruana%2E+Do+deep+nets+really+need+to+be+deep%3F+In+NIPS%2C+2014%2E)
10. Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. CoRR, abs/1312.4400, 2013. — [arXiv:1312.4400](https://arxiv.org/abs/1312.4400) [find](https://scholar.google.com/scholar?q=Min+Lin%2C+Qiang+Chen%2C+and+Shuicheng+Yan%2E+Network+in+network%2E+CoRR%2C+abs%2F1312%2E4400%2C+2013%2E+%E2%80%94+%5BarXiv%3A1312%2E4400%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1312%2E4400%29)
11. Volodymyr Mnih, Nicolas Heess, Alex Graves, and koray kavukcuoglu. Recurrent models of visual attention. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2204–2212. Curran Associates, Inc., 2014. URL http:… [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Nicolas+Heess%2C+Alex+Graves%2C+and+koray+kavukcuoglu%2E+Recurrent+models+of+visual+attention%2E+In+Z%2E+Ghahramani%2C+M%2E+Welling%2C+C%2E+Cortes%2C+N%2E+D%2E+Lawrence%2C+and+K%2E+Q%2E+Weinberger+%28eds%2E%29%2C+Advances+)
12. M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Is object localization for free? weakly-supervised learning with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. [find](https://scholar.google.com/scholar?q=M%2E+Oquab%2C+L%2E+Bottou%2C+I%2E+Laptev%2C+and+J%2E+Sivic%2E+Is+object+localization+for+free%3F+weakly%2Dsupervised+learning+with+convolutional+neural+networks%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+a)
13. O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In British Machine Vision Conference, 2015. [find](https://scholar.google.com/scholar?q=O%2E+M%2E+Parkhi%2C+A%2E+Vedaldi%2C+and+A%2E+Zisserman%2E+Deep+face+recognition%2E+In+British+Machine+Vision+Conference%2C+2015%2E)
14. A. Quattoni and A. Torralba. Recognizing indoor scenes. In CVPR, 2009. [find](https://scholar.google.com/scholar?q=A%2E+Quattoni+and+A%2E+Torralba%2E+Recognizing+indoor+scenes%2E+In+CVPR%2C+2009%2E)
15. Ronald A. Rensink. The dynamic representation of scenes. In Visual Cognition, pp. 17–42, 2000. [find](https://scholar.google.com/scholar?q=Ronald+A%2E+Rensink%2E+The+dynamic+representation+of+scenes%2E+In+Visual+Cognition%2C+pp%2E+17%E2%80%9342%2C+2000%2E)
16. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. Technical Report Arxiv report 1412.6550, arXiv, 2014. [find](https://scholar.google.com/scholar?q=Adriana+Romero%2C+Nicolas+Ballas%2C+Samira+Ebrahimi+Kahou%2C+Antoine+Chassang%2C+Carlo+Gatta%2C+and+Yoshua+Bengio%2E+FitNets%3A+Hints+for+thin+deep+nets%2E+Technical+Report+Arxiv+report+1412%2E6550%2C+arXiv%2C+2014%2E)
17. Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. 2016. [find](https://scholar.google.com/scholar?q=Ramprasaath+R%2E+Selvaraju%2C+Abhishek+Das%2C+Ramakrishna+Vedantam%2C+Devi+Parikh%2C+and+Dhruv+Batra%2E+Grad%2Dcam%3A+Why+did+you+say+that%3F+visual+explanations+from+deep+networks+via+gradient%2Dbased+localization%2E+2016)
18. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. [Deep inside convolutional networks: Visualising image classification models and saliency maps](https://md.research.elamur.ai/free/en/papers/simonyan-2013-deep-inside-convolutional-networks-visualising/). In ICLR Workshop, 2014.
19. J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. In arXiv:1412.6806, also appeared at ICLR 2015 Workshop Track, 2015. URL http://arxiv.org/abs/1412.6806. — [arXiv:1412.6806](https://arxiv.org/abs/1412.6806) [↗](http://arxiv.org/abs/1412.6806)
20. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. [Highway networks](https://md.research.elamur.ai/free/en/papers/srivastava-2015-highway-networks/). CoRR, abs/1505.00387, 2015.
21. C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. [find](https://scholar.google.com/scholar?q=C%2E+Wah%2C+S%2E+Branson%2C+P%2E+Welinder%2C+P%2E+Perona%2C+and+S%2E+Belongie%2E+The+Caltech%2DUCSD+Birds%2D200%2D2011+Dataset%2E+Technical+Report+CNS%2DTR%2D2011%2D001%2C+California+Institute+of+Technology%2C+2011%2E)
22. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. [Show, attend and tell: Neural image caption generation with visual attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). CoRR, abs/1502.03044, 2015. URL http://arxiv.org/abs/1502.03044. [↗](http://arxiv.org/abs/1502.03044)
23. CoRR, Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. [Stacked attention networks for image question answering](https://md.research.elamur.ai/free/en/papers/yang-2015-stacked-attention-networks-image-question/). CoRR, abs/1511.02274, 2015. URL http://arxiv.org/abs/1511.02274. [↗](http://arxiv.org/abs/1511.02274)
24. Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016. [find](https://scholar.google.com/scholar?q=Sergey+Zagoruyko+and+Nikos+Komodakis%2E+Wide+residual+networks%2E+In+BMVC%2C+2016%2E)
25. Matthew Zeiler and Rob Fergus. [Visualizing and understanding convolutional networks](https://md.research.elamur.ai/free/en/papers/zeiler-2013-visualizing-understanding-convolutional-networks/). In ECCV, 2014.
26. Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Computer Vision and Pattern Recognition, 2016. [find](https://scholar.google.com/scholar?q=Bolei+Zhou%2C+Aditya+Khosla%2C+Agata+Lapedriza%2C+Aude+Oliva%2C+and+Antonio+Torralba%2E+Learning+deep+features+for+discriminative+localization%2E+In+Computer+Vision+and+Pattern+Recognition%2C+2016%2E)

*Updated 2 November 2026*