---
title: "Accelerating Sparse Deep Neural Networks — Mishra 2021"
description: "The NVIDIA authors propose a fine-grained structured pattern called 2:4, where every four… 91.9 F1. 65 references, cited by 86 papers here."
authors: ["Asit Mishra", "Jorge Albericio Latorre", "Jeff Pool", "Darko Stosic", "Dusan Stosic", "Ganesh Venkatesh", "Chong Yu", "Paulius Micikevicius"]
year: 2021
venue: "arXiv 2021 (NVIDIA)"
source: "https://arxiv.org/abs/2104.08378"
arxiv: "2104.08378"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mishra-2021-accelerating-sparse-deep-neural-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mishra-2021-accelerating-sparse-deep-neural-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Accelerating Sparse Deep Neural Networks

# Accelerating Sparse Deep Neural Networks

**Authors:** Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu and Paulius Micikevicius · **Year:** 2021 · **Venue:** arXiv 2021 (NVIDIA) · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2104.08378) · [PDF](https://arxiv.org/pdf/2104.08378) · [arXiv 2104.08378](https://arxiv.org/abs/2104.08378) · [OpenAlex](https://openalex.org/W3156528192)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [86](https://md.research.elamur.ai/free/en/papers/mishra-2021-accelerating-sparse-deep-neural-networks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Pruning can shrink neural networks, but it rarely delivers both full accuracy and real speed. Unstructured sparsity keeps accuracy yet maps poorly onto GPU matrix units. Coarse structured sparsity runs fast but loses accuracy once sparsity gets high. The NVIDIA authors propose a fine-grained structured pattern called 2:4, where every four consecutive values contain at least two zeros. That gives 50% sparsity. They pair it with a compact storage format and with Sparse Tensor Cores in the Ampere GPU architecture, which double matrix-multiply math throughput when one operand is a compressed 2:4 tensor.

To produce compatible models, they use a simple recipe:
- train a dense network,
- prune it once,
- repeat the original training run with the sparsity mask frozen and the same hyper-parameters.

Across vision, translation and language-modeling networks, the sparse models match their dense counterparts, including under INT8 quantization. The work matters because it offers a single sparsity scheme that is both accurate and genuinely accelerated in commodity hardware, without per-model tuning.

## Key points
- **Storage format.** A 2:4 sparse matrix of size R × C is stored as R × C/2 nonzero values plus a 2-bit index per value. The metadata overhead is 12.5% for 16-bit weights and 25% for 8-bit weights. Total storage drops by about 44% (FP16) and about 38% (INT8) compared with dense.
- **Hardware.** Sparse Tensor Cores give 2× math throughput for GEMM-like operations (convolutions, matrix multiplies) whose first argument is 2:4 sparse.
- **Translation (BLEU, WMT'14 En-De).** GNMT goes from 24.6 dense to 24.9 sparse FP16 and 24.9 sparse INT8. The FairSeq Transformer goes from 28.2 to 28.5 (FP16) and 28.3 (INT8).
- **Language modeling.** Transformer-XL on enwik8 stays at 1.06 BPC. BERT LARGE on SQuAD v1.1 keeps 91.9 F1 in FP16, and scores 90.8 sparse vs 90.9 dense in INT8. For pre-trained models, pre-training is rerun after pruning, then the model is fine-tuned. Attention products are left unpruned because they contain no weights.
- **Tooling and limits.** Masks come from the PyTorch ASP library, which offers greedy or exhaustive search to minimize lost weight magnitude. The workflow's main cost is that it doubles training time.

## Interesting ideas
- **Hardware-friendly mid-level sparsity.** Choosing a moderate, uniform sparsity level (50% in every small group) instead of extreme unstructured sparsity keeps accuracy recoverable and memory access predictable. This trade-off could guide compression design for any model meant for real deployment.
- **Retrain without tuning.** Pruning once and then replaying the original training recipe with a fixed mask avoids hyper-parameter searches. It is a robust baseline that can be reused across many architectures and tasks, including Ukrainian NLP models.
- **Fixed-size groups with tiny metadata.** Tiny fixed-size groups mean the position of every stored value can be computed directly, with no data-dependent indirection. This idea is useful for any compact tensor or embedding storage.
- **Prune, then redo pre-training.** For pre-trained language models, pruning and then repeating pre-training before task fine-tuning was what recovered dense accuracy. This is a practical recipe for compressing BERT-like models for downstream tasks.
- **Test sparsity together with INT8.** Checking that sparsity and INT8 quantization compose without extra loss is a good evaluation habit, because deployment usually combines several compression methods.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nvidia](https://md.research.elamur.ai/free/en/companies/nvidia/) — Company, Santa Clara, United States · 37 works · 12,656 citations
- [Nvidia (United Kingdom)](https://md.research.elamur.ai/free/en/companies/nvidia-united-kingdom/) — Company, Reading, United Kingdom · 27 works · 11,141 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 188 works · [⚙️ Electrical and Electronic Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/electrical-and-electronic-engineering/) · 33 works

## Cite

Cite as: Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu and Paulius Micikevicius. 2021. Accelerating Sparse Deep Neural Networks. arXiv 2021 (NVIDIA).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mishra2021accelerating,
  title = {{Accelerating Sparse Deep Neural Networks}},
  author = {Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu and Paulius Micikevicius},
  year = {2021},
  eprint = {2104.08378},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.08378}
}
```

- [← previous Understanding by Understanding Not: Modeling Negation in Language Models](https://md.research.elamur.ai/free/en/papers/hosseini-2021-understanding-understanding-not-modeling-negation/)
- [next → AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding](https://md.research.elamur.ai/free/en/papers/antoun-2021-araelectra-pre-training-text-discriminators/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

65 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. [Deep learning scaling is predictable, empirically](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/). CoRR, abs/1712.00409, 2017.
2. Joel Hestness, Newsha Ardalani, and Greg Diamos. Beyond human-level accuracy: Computational challenges in deep learning. CoRR, abs/1909.01736, 2019. — [arXiv:1909.01736](https://arxiv.org/abs/1909.01736) [find](https://scholar.google.com/scholar?q=Joel+Hestness%2C+Newsha+Ardalani%2C+and+Greg+Diamos%2E+Beyond+human%2Dlevel+accuracy%3A+Computational+challenges+in+deep+learning%2E+CoRR%2C+abs%2F1909%2E01736%2C+2019%2E+%E2%80%94+%5BarXiv%3A1909%2E01736%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1909%2E017)
3. Md. Mostofa Ali Patwary, Milind Chabbi, Heewoo Jun, Jiaji Huang, Gregory Frederick Diamos, and Kenneth Church. Language modeling at scale. CoRR, abs/1810.10045, 2018. [find](https://scholar.google.com/scholar?q=Md%2E+Mostofa+Ali+Patwary%2C+Milind+Chabbi%2C+Heewoo+Jun%2C+Jiaji+Huang%2C+Gregory+Frederick+Diamos%2C+and+Kenneth+Church%2E+Language+modeling+at+scale%2E+CoRR%2C+abs%2F1810%2E10045%2C+2018%2E)
4. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. [Mesh-tensorflow: Deep learning for supercomputers](https://md.research.elamur.ai/free/en/papers/shazeer-2018-mesh-tensorflow-deep-learning-supercomputers/). CoRR, abs/1811.02084, 2018.
5. Nikko Ström. Sparse connection and pruning in large dynamic artificial neural networks. In EUROSPEECH, 1997. [find](https://scholar.google.com/scholar?q=Nikko+Stro%CC%88m%2E+Sparse+connection+and+pruning+in+large+dynamic+artificial+neural+networks%2E+In+EUROSPEECH%2C+1997%2E)
6. Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Shijian Tang, Erich Elsen, Bryan Catanzaro, John Tran, and William J. Dally. DSD: regularizing deep neural networks with dense-sparse-dense training flow. In ICLR, 2017. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Jeff+Pool%2C+Sharan+Narang%2C+Huizi+Mao%2C+Shijian+Tang%2C+Erich+Elsen%2C+Bryan+Catanzaro%2C+John+Tran%2C+and+William+J%2E+Dally%2E+DSD%3A+regularizing+deep+neural+networks+with+dense%2Dsparse%2Ddense+training+flow)
7. Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, pages 1135–1143, 2015. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Jeff+Pool%2C+John+Tran%2C+and+William+J+Dally%2E+Learning+both+weights+and+connections+for+efficient+neural+networks%2E+In+Proceedings+of+the+28th+International+Conference+on+Neural+Information+Proc)
8. Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William J Dally. Exploring the granularity of sparsity in convolutional neural networks. In CVPR Workshops, pages 13–20, 2017. [find](https://scholar.google.com/scholar?q=Huizi+Mao%2C+Song+Han%2C+Jeff+Pool%2C+Wenshuo+Li%2C+Xingyu+Liu%2C+Yu+Wang%2C+and+William+J+Dally%2E+Exploring+the+granularity+of+sparsity+in+convolutional+neural+networks%2E+In+CVPR+Workshops%2C+pages+13%E2%80%9320%2C+2017%2E)
9. Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In ICLR Workshops, 2018. [find](https://scholar.google.com/scholar?q=Michael+Zhu+and+Suyog+Gupta%2E+To+prune%2C+or+not+to+prune%3A+Exploring+the+efficacy+of+pruning+for+model+compression%2E+In+ICLR+Workshops%2C+2018%2E)
10. Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. CoRR, abs/1707.06168, 2017. — [arXiv:1707.06168](https://arxiv.org/abs/1707.06168) [find](https://scholar.google.com/scholar?q=Yihui+He%2C+Xiangyu+Zhang%2C+and+Jian+Sun%2E+Channel+pruning+for+accelerating+very+deep+neural+networks%2E+CoRR%2C+abs%2F1707%2E06168%2C+2017%2E+%E2%80%94+%5BarXiv%3A1707%2E06168%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1707%2E06168%29)
11. Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient transfer learning. CoRR, abs/1611.06440, 2016. — [arXiv:1611.06440](https://arxiv.org/abs/1611.06440) [find](https://scholar.google.com/scholar?q=Pavlo+Molchanov%2C+Stephen+Tyree%2C+Tero+Karras%2C+Timo+Aila%2C+and+Jan+Kautz%2E+Pruning+convolutional+neural+networks+for+resource+efficient+transfer+learning%2E+CoRR%2C+abs%2F1611%2E06440%2C+2016%2E+%E2%80%94+%5BarXiv%3A1611%2E06440%5D%28)
12. Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. CoRR, abs/1810.05270, 2018. [find](https://scholar.google.com/scholar?q=Zhuang+Liu%2C+Mingjie+Sun%2C+Tinghui+Zhou%2C+Gao+Huang%2C+and+Trevor+Darrell%2E+Rethinking+the+value+of+network+pruning%2E+CoRR%2C+abs%2F1810%2E05270%2C+2018%2E)
13. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural networks. CoRR, abs/1803.03635, 2018. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+The+lottery+ticket+hypothesis%3A+Training+pruned+neural+networks%2E+CoRR%2C+abs%2F1803%2E03635%2C+2018%2E)
14. [NVIDIA A100 tensor core GPU architecture](https://md.research.elamur.ai/free/en/papers/nvidia-2020-nvidia-a100-tensor-core-gpu/). [https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-ampere-architecture-whitepaper](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-ampere-architecture-whitepaper). pdf, 2020. [↗](https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/nvidia-ampere-architecture-whitepaper)
15. Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Pavlo+Molchanov%2C+Arun+Mallya%2C+Stephen+Tyree%2C+Iuri+Frosio%2C+and+Jan+Kautz%2E+Importance+estimation+for+neural+network+pruning%2E+In+CVPR%2C+2019%2E)
16. Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: single-shot network pruning based on connection sensitivity. In ICLR, 2019. [find](https://scholar.google.com/scholar?q=Namhoon+Lee%2C+Thalaiyasingam+Ajanthan%2C+and+Philip+Torr%2E+SNIP%3A+single%2Dshot+network+pruning+based+on+connection+sensitivity%2E+In+ICLR%2C+2019%2E)
17. Zhuliang Yao, Shijie Cao, Wencong Xiao, Chen Zhang, and Lanshun Nie. Balanced sparsity for efficient DNN inference on GPU. CoRR, abs/1811.00206, 2018. [find](https://scholar.google.com/scholar?q=Zhuliang+Yao%2C+Shijie+Cao%2C+Wencong+Xiao%2C+Chen+Zhang%2C+and+Lanshun+Nie%2E+Balanced+sparsity+for+efficient+DNN+inference+on+GPU%2E+CoRR%2C+abs%2F1811%2E00206%2C+2018%2E)
18. Huan Wang, Qiming Zhang, Yuehai Wang, and Haoji Hu. Structured pruning for efficient convnets via incremental regularization. CoRR, abs/1811.08390, 2018. — [arXiv:1811.08390](https://arxiv.org/abs/1811.08390) [find](https://scholar.google.com/scholar?q=Huan+Wang%2C+Qiming+Zhang%2C+Yuehai+Wang%2C+and+Haoji+Hu%2E+Structured+pruning+for+efficient+convnets+via+incremental+regularization%2E+CoRR%2C+abs%2F1811%2E08390%2C+2018%2E+%E2%80%94+%5BarXiv%3A1811%2E08390%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F181)
19. Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. CoRR, abs/1512.08571, 2015. — [arXiv:1512.08571](https://arxiv.org/abs/1512.08571) [find](https://scholar.google.com/scholar?q=Sajid+Anwar%2C+Kyuyeon+Hwang%2C+and+Wonyong+Sung%2E+Structured+pruning+of+deep+convolutional+neural+networks%2E+CoRR%2C+abs%2F1512%2E08571%2C+2015%2E+%E2%80%94+%5BarXiv%3A1512%2E08571%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1512%2E08571%29)
20. Mingbao Lin, Rongrong Ji, Shaojie Li, Qixiang Ye, Yonghong Tian, Jianzhuang Liu, and Qi Tian. Filter sketch for network pruning. CoRR, abs/2001.08514, 2020. — [arXiv:2001.08514](https://arxiv.org/abs/2001.08514) [find](https://scholar.google.com/scholar?q=Mingbao+Lin%2C+Rongrong+Ji%2C+Shaojie+Li%2C+Qixiang+Ye%2C+Yonghong+Tian%2C+Jianzhuang+Liu%2C+and+Qi+Tian%2E+Filter+sketch+for+network+pruning%2E+CoRR%2C+abs%2F2001%2E08514%2C+2020%2E+%E2%80%94+%5BarXiv%3A2001%2E08514%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
21. Noah Gamboa, Kais Kudrolli, Anand Dhoot, and Ardavan Pedram. Campfire: Compressible, regularization-free, structured sparse training for hardware accelerators. CoRR, abs/2001.03253, 2020. — [arXiv:2001.03253](https://arxiv.org/abs/2001.03253) [find](https://scholar.google.com/scholar?q=Noah+Gamboa%2C+Kais+Kudrolli%2C+Anand+Dhoot%2C+and+Ardavan+Pedram%2E+Campfire%3A+Compressible%2C+regularization%2Dfree%2C+structured+sparse+training+for+hardware+accelerators%2E+CoRR%2C+abs%2F2001%2E03253%2C+2020%2E+%E2%80%94+%5BarXiv%3A200)
22. Maohua Zhu, Tao Zhang, Zhenyu Gu, and Yuan Xie. Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus. In MICRO, 2019. [find](https://scholar.google.com/scholar?q=Maohua+Zhu%2C+Tao+Zhang%2C+Zhenyu+Gu%2C+and+Yuan+Xie%2E+Sparse+tensor+core%3A+Algorithm+and+hardware+co%2Ddesign+for+vector%2Dwise+sparse+neural+networks+on+modern+gpus%2E+In+MICRO%2C+2019%2E)
23. Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. [Learning structured sparsity in deep neural networks](https://md.research.elamur.ai/free/en/papers/wen-2016-learning-structured-sparsity-deep-neural/). CoRR, abs/1608.03665, 2016.
24. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. The lottery ticket hypothesis at scale. CoRR, abs/1903.01611, 2019. — [arXiv:1903.01611](https://arxiv.org/abs/1903.01611) [find](https://scholar.google.com/scholar?q=Jonathan+Frankle%2C+Gintare+Karolina+Dziugaite%2C+Daniel+M%2E+Roy%2C+and+Michael+Carbin%2E+The+lottery+ticket+hypothesis+at+scale%2E+CoRR%2C+abs%2F1903%2E01611%2C+2019%2E+%E2%80%94+%5BarXiv%3A1903%2E01611%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1903%2E016)
25. Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing fine-tuning and rewinding in neural network pruning. In ICLR, 2020. [find](https://scholar.google.com/scholar?q=Alex+Renda%2C+Jonathan+Frankle%2C+and+Michael+Carbin%2E+Comparing+fine%2Dtuning+and+rewinding+in+neural+network+pruning%2E+In+ICLR%2C+2020%2E)
26. Sharan Narang, Gregory F. Diamos, Shubho Sengupta, and Erich Elsen. Exploring sparsity in recurrent neural networks. CoRR, abs/1704.05119, 2017. [find](https://scholar.google.com/scholar?q=Sharan+Narang%2C+Gregory+F%2E+Diamos%2C+Shubho+Sengupta%2C+and+Erich+Elsen%2E+Exploring+sparsity+in+recurrent+neural+networks%2E+CoRR%2C+abs%2F1704%2E05119%2C+2017%2E)
27. Sharan Narang, Eric Undersander, and Gregory F. Diamos. Block-sparse recurrent neural networks. CoRR, abs/1711.02782, 2017. — [arXiv:1711.02782](https://arxiv.org/abs/1711.02782) [find](https://scholar.google.com/scholar?q=Sharan+Narang%2C+Eric+Undersander%2C+and+Gregory+F%2E+Diamos%2E+Block%2Dsparse+recurrent+neural+networks%2E+CoRR%2C+abs%2F1711%2E02782%2C+2017%2E+%E2%80%94+%5BarXiv%3A1711%2E02782%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1711%2E02782%29)
28. Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. CoRR, abs/1911.09723, 2019. — [arXiv:1911.09723](https://arxiv.org/abs/1911.09723) [find](https://scholar.google.com/scholar?q=Erich+Elsen%2C+Marat+Dukhan%2C+Trevor+Gale%2C+and+Karen+Simonyan%2E+Fast+sparse+convnets%2E+CoRR%2C+abs%2F1911%2E09723%2C+2019%2E+%E2%80%94+%5BarXiv%3A1911%2E09723%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1911%2E09723%29)
29. Mi Sun Park, Xiaofan Xu, and Cormac Brick. Squantizer: Simultaneous learning for both sparse and low-precision neural networks. CoRR, abs/1812.08301, 2018. — [arXiv:1812.08301](https://arxiv.org/abs/1812.08301) [find](https://scholar.google.com/scholar?q=Mi+Sun+Park%2C+Xiaofan+Xu%2C+and+Cormac+Brick%2E+Squantizer%3A+Simultaneous+learning+for+both+sparse+and+low%2Dprecision+neural+networks%2E+CoRR%2C+abs%2F1812%2E08301%2C+2018%2E+%E2%80%94+%5BarXiv%3A1812%2E08301%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1)
30. Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. CoRR, abs/1802.08435, 2018. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner%2C+Erich+Elsen%2C+Karen+Simonyan%2C+Seb+Noury%2C+Norman+Casagrande%2C+Edward+Lockhart%2C+Florian+Stimberg%2C+Aa%CC%88ron+van+den+Oord%2C+Sander+Dieleman%2C+and+Koray+Kavukcuoglu%2E+Efficient+neural+audio+synt)
31. Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. CoRR, abs/1911.11134, 2019. — [arXiv:1911.11134](https://arxiv.org/abs/1911.11134) [find](https://scholar.google.com/scholar?q=Utku+Evci%2C+Trevor+Gale%2C+Jacob+Menick%2C+Pablo+Samuel+Castro%2C+and+Erich+Elsen%2E+Rigging+the+lottery%3A+Making+all+tickets+winners%2E+CoRR%2C+abs%2F1911%2E11134%2C+2019%2E+%E2%80%94+%5BarXiv%3A1911%2E11134%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1911)
32. Scott Gray, Alec Radford, and Diederik P. Kingma. GPU Kernels for Block-Sparse Weights, Accessed April 13, 2021. [https://openai.com/blog/block-sparse-gpu-kernels/](https://openai.com/blog/block-sparse-gpu-kernels/). [↗](https://openai.com/blog/block-sparse-gpu-kernels/)
33. H. Kang. Accelerator-aware pruning for convolutional neural networks. TCSVT, 30(7):2093–2103, 2020. [find](https://scholar.google.com/scholar?q=H%2E+Kang%2E+Accelerator%2Daware+pruning+for+convolutional+neural+networks%2E+TCSVT%2C+30%287%29%3A2093%E2%80%932103%2C+2020%2E)
34. Sparse matrix storage formats. [https://www.gnu.org/software/gsl/doc/html/spmatrix.html](https://www.gnu.org/software/gsl/doc/html/spmatrix.html). [↗](https://www.gnu.org/software/gsl/doc/html/spmatrix.html)
35. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
36. NVIDIA’s Automatic SParsity (ASP) library. [https://github.com/NVIDIA/apex/tree/master/apex/contrib/sparsity](https://github.com/NVIDIA/apex/tree/master/apex/contrib/sparsity), 2020. [↗](https://github.com/NVIDIA/apex/tree/master/apex/contrib/sparsity)
37. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805, 2018.
38. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR, abs/1409.4842, 2014. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wei+Liu%2C+Yangqing+Jia%2C+Pierre+Sermanet%2C+Scott+E%2E+Reed%2C+Dragomir+Anguelov%2C+Dumitru+Erhan%2C+Vincent+Vanhoucke%2C+and+Andrew+Rabinovich%2E+Going+deeper+with+convolutions%2E+CoRR%2C+abs%2F1409%2E484)
39. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. [ImageNet Large Scale Visual Recognition Challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/). IJCV, 115(3):211–252, 2015.
40. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. [Microsoft COCO: common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). CoRR, abs/1405.0312, 2014.
41. Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. CoRR, abs/2004.09602, 2020. [find](https://scholar.google.com/scholar?q=Hao+Wu%2C+Patrick+Judd%2C+Xiaojie+Zhang%2C+Mikhail+Isaev%2C+and+Paulius+Micikevicius%2E+Integer+quantization+for+deep+learning+inference%3A+Principles+and+empirical+evaluation%2E+CoRR%2C+abs%2F2004%2E09602%2C+2020%2E)
42. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). CoRR, abs/1512.03385, 2015.
43. Karen Simonyan and Andrew Zisserman. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). In ICLR, 2015.
44. Sohil Shah, Pallabi Ghosh, Larry S. Davis, and Tom Goldstein. Stacked u-nets: A no-frills approach to natural image segmentation. CoRR, abs/1804.10343, 2018. — [arXiv:1804.10343](https://arxiv.org/abs/1804.10343) [find](https://scholar.google.com/scholar?q=Sohil+Shah%2C+Pallabi+Ghosh%2C+Larry+S%2E+Davis%2C+and+Tom+Goldstein%2E+Stacked+u%2Dnets%3A+A+no%2Dfrills+approach+to+natural+image+segmentation%2E+CoRR%2C+abs%2F1804%2E10343%2C+2018%2E+%E2%80%94+%5BarXiv%3A1804%2E10343%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
45. Fisher Yu, Vladlen Koltun, and Thomas A. Funkhouser. Dilated residual networks. CoRR, abs/1705.09914, 2017. [find](https://scholar.google.com/scholar?q=Fisher+Yu%2C+Vladlen+Koltun%2C+and+Thomas+A%2E+Funkhouser%2E+Dilated+residual+networks%2E+CoRR%2C+abs%2F1705%2E09914%2C+2017%2E)
46. Dhruv Kumar Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018. [find](https://scholar.google.com/scholar?q=Dhruv+Kumar+Mahajan%2C+Ross+B%2E+Girshick%2C+Vignesh+Ramanathan%2C+Kaiming+He%2C+Manohar+Paluri%2C+Yixuan+Li%2C+Ashwin+Bharambe%2C+and+Laurens+van+der+Maaten%2E+Exploring+the+limits+of+weakly+supervised+pretraining%2E+In)
47. I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. CoRR, abs/1905.00546, 2019. — [arXiv:1905.00546](https://arxiv.org/abs/1905.00546) [find](https://scholar.google.com/scholar?q=I%2E+Zeki+Yalniz%2C+Herve%CC%81+Je%CC%81gou%2C+Kan+Chen%2C+Manohar+Paluri%2C+and+Dhruv+Mahajan%2E+Billion%2Dscale+semi%2Dsupervised+learning+for+image+classification%2E+CoRR%2C+abs%2F1905%2E00546%2C+2019%2E+%E2%80%94+%5BarXiv%3A1905%2E00546%5D%28https%3A%2F%2Far)
48. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Mark+Sandler%2C+Andrew+Howard%2C+Menglong+Zhu%2C+Andrey+Zhmoginov%2C+and+Liang%2DChieh+Chen%2E+Mobilenetv2%3A+Inverted+residuals+and+linear+bottlenecks%2E+In+CVPR%2C+2018%2E)
49. Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \\<1mb model size. CoRR, abs/1602.07360, 2016. [find](https://scholar.google.com/scholar?q=Forrest+N%2E+Iandola%2C+Matthew+W%2E+Moskewicz%2C+Khalid+Ashraf%2C+Song+Han%2C+William+J%2E+Dally%2C+and+Kurt+Keutzer%2E+Squeezenet%3A+Alexnet%2Dlevel+accuracy+with+50x+fewer+parameters+and+%5C%3C1mb+model+size%2E+CoRR%2C+abs%2F1602)
50. Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. MnasNet: Platform-aware neural architecture search for mobile. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Mingxing+Tan%2C+Bo+Chen%2C+Ruoming+Pang%2C+Vijay+Vasudevan%2C+Mark+Sandler%2C+Andrew+Howard%2C+and+Quoc+V%2E+Le%2E+MnasNet%3A+Platform%2Daware+neural+architecture+search+for+mobile%2E+In+CVPR%2C+2019%2E)
51. Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018. [find](https://scholar.google.com/scholar?q=Ningning+Ma%2C+Xiangyu+Zhang%2C+Hai%2DTao+Zheng%2C+and+Jian+Sun%2E+Shufflenet+v2%3A+Practical+guidelines+for+efficient+cnn+architecture+design%2E+In+ECCV%2C+2018%2E)
52. Mingxing Tan and Quoc V. Le. [Efficientnet: Rethinking model scaling for convolutional neural networks](https://md.research.elamur.ai/free/en/papers/tan-2019-efficientnet-rethinking-model-scaling-convolutional/). In ICML, 2019.
53. Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. [https://github.com/facebookresearch/detectron2](https://github.com/facebookresearch/detectron2), 2019. [↗](https://github.com/facebookresearch/detectron2)
54. Nvidia deep learning examples. [https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Detection/SSD](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Detection/SSD). [↗](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Detection/SSD)
55. Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=Yi+Zhu%2C+Karan+Sapra%2C+Fitsum+A%2E+Reda%2C+Kevin+J%2E+Shih%2C+Shawn+Newsam%2C+Andrew+Tao%2C+and+Bryan+Catanzaro%2E+Improving+semantic+segmentation+via+video+propagation+and+label+relaxation%2E+In+CVPR%2C+2019%2E)
56. Chong Yu and Jeff Pool. Self-supervised generative adversarial compression. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Chong+Yu+and+Jeff+Pool%2E+Self%2Dsupervised+generative+adversarial+compression%2E+In+NeurIPS%2C+2020%2E)
57. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017. [find](https://scholar.google.com/scholar?q=Martin+Heusel%2C+Hubert+Ramsauer%2C+Thomas+Unterthiner%2C+Bernhard+Nessler%2C+and+Sepp+Hochreiter%2E+GANs+trained+by+a+two+time%2Dscale+update+rule+converge+to+a+local+nash+equilibrium%2E+In+NeurIPS%2C+2017%2E)
58. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+Luk)
59. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In NAACL-HLT: Demonstrations, 2019.
60. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. CoRR, abs/1901.02860, 2019. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+G%2E+Carbonell%2C+Quoc+V%2E+Le%2C+and+Ruslan+Salakhutdinov%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+CoRR%2C+abs%2F1901%2E02860%2C+2019%2E)
61. The Hutter Prize Wikipedia dataset, Accessed April 13, 2020. [http://prize.hutter1.net/](http://prize.hutter1.net/). [↗](http://prize.hutter1.net/)
62. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.[05250, 2016](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/).
63. Megatron-LM repo, Accessed April 13, 2020. [https://github.com/NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM). [↗](https://github.com/NVIDIA/Megatron-LM)
64. Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Seffi Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n:m transposable masks. CoRR, abs/2102.08124, 2021. — [arXiv:2102.08124](https://arxiv.org/abs/2102.08124) [find](https://scholar.google.com/scholar?q=Itay+Hubara%2C+Brian+Chmiel%2C+Moshe+Island%2C+Ron+Banner%2C+Seffi+Naor%2C+and+Daniel+Soudry%2E+Accelerated+sparse+neural+training%3A+A+provable+and+efficient+method+to+find+n%3Am+transposable+masks%2E+CoRR%2C+abs%2F2102%2E0)
65. Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning N:M fine-grained structured sparse neural networks from scratch. In ICLR, 2021. [find](https://scholar.google.com/scholar?q=Aojun+Zhou%2C+Yukun+Ma%2C+Junnan+Zhu%2C+Jianbo+Liu%2C+Zhijie+Zhang%2C+Kun+Yuan%2C+Wenxiu+Sun%2C+and+Hongsheng+Li%2E+Learning+N%3AM+fine%2Dgrained+structured+sparse+neural+networks+from+scratch%2E+In+ICLR%2C+2021%2E)

*Updated 2 November 2026*