---
title: "vDNN — paper by Rhu et al. (2016) · MICRO 2016"
description: "They build vDNN, a runtime memory manager that treats GPU and CPU memory as one virtual pool. 53 references, cited by 80 papers here."
authors: ["Minsoo Rhu", "Natalia Gimelshein", "Jason Clemons", "Arslan Zulfiqar", "Stephen W. Keckler"]
year: 2016
venue: "MICRO 2016"
source: "https://arxiv.org/abs/1602.08124"
arxiv: "1602.08124"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rhu-2016-vdnn-virtualized-deep-neural-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rhu-2016-vdnn-virtualized-deep-neural-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design

# vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design

**Authors:** Minsoo Rhu, [Natalia Gimelshein](https://md.research.elamur.ai/free/en/authors/natalia-gimelshein/), Jason Clemons, Arslan Zulfiqar and Stephen W. Keckler · **Year:** 2016 · **Venue:** MICRO 2016 · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1602.08124) · [PDF](https://arxiv.org/pdf/1602.08124) · [arXiv 1602.08124](https://arxiv.org/abs/1602.08124) · [OpenAlex](https://openalex.org/W4302392994)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [80](https://md.research.elamur.ai/free/en/papers/rhu-2016-vdnn-virtualized-deep-neural-networks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training a deep network on a GPU normally requires the whole network's memory footprint to fit in GPU DRAM. This forces researchers to shrink models, cut batch sizes, pick slower but leaner convolution algorithms, or split work across several GPUs. Rhu et al. (NVIDIA) argue that this limit is largely artificial. Frameworks reserve memory for every layer at once, but SGD training runs one layer at a time in a fixed order that repeats every iteration.

They build vDNN, a runtime memory manager that treats GPU and CPU memory as one virtual pool. Intermediate feature maps take most of the memory. vDNN frees them when no later reuse exists. Otherwise it moves them to host memory and fetches them back before backward propagation needs them, overlapping these copies with computation.

On a 12 GB Titan X, vDNN can train networks needing up to 67 GB. This includes VGG-16 at batch 256, which needs 28 GB. The slowdown is 1–18% compared with an ideal GPU that has unlimited memory. The approach is transparent to the programmer.

## Key points
- Under network-wide allocation, 53–79% of allocated GPU memory sits unused at any moment, and the waste grows with network depth.
- Baseline frameworks could not train 6 of the 10 studied DNNs, which required 14–67 GB against the Titan X's 12 GB. vDNN cut the average GPU memory use of these six by 73–98%.
- Average GPU memory reductions were 89% for AlexNet, 91% for OverFeat and 95% for GoogLeNet.
- VGG-16 with batch 256 (28 GB; a single GPU otherwise fits only batch 64) trains on one Titan X with an 18% slowdown relative to the oracular GPU.
- For non-linear graphs such as GoogLeNet, where outputs fork and join, a per-layer reference count of consumer layers decides when a feature map can be released.
- Memory is accounted for by type: weights, feature maps, gradient buffers (dX/dY, reused across layers) and the cuDNN workspace.

## Interesting ideas
- **Use the static execution order as a schedule.** Training repeats the same layer sequence millions of times, so data movement can be planned ahead rather than handled by reactive paging. Any workload with a fixed, repeating pipeline could prefetch or evict data this way.
- **Sort data by reuse distance.** Data with no future use is freed immediately. Data needed much later is moved to slower, larger memory and brought back just in time. This simple three-way policy (keep, offload, release) carries over to other memory hierarchies, such as GPU→CPU or RAM→disk.
- **Reference counting on the layer graph.** Counting how many consumers still need a tensor gives a cheap, safe release rule for DAG-shaped models, not only linear chains.
- **Hide transfers behind computation.** Running copies concurrently with layer computation is what keeps the slowdown small. This is worth copying whenever data must move between memory tiers.
- **Compare peak per-layer usage with total allocation.** Reporting what fraction of reserved memory is actually live is a quick diagnostic for over-provisioning. Benchmarking against an oracle with unlimited memory gives an honest upper bound on overhead.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nvidia](https://md.research.elamur.ai/free/en/companies/nvidia/) — Company, Santa Clara, United States · 37 works · 12,656 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar and Stephen W. Keckler. 2016. vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design. MICRO 2016.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rhu2016vdnn,
  title = {{vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design}},
  author = {Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar and Stephen W. Keckler},
  booktitle = {MICRO 2016},
  year = {2016},
  eprint = {1602.08124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1602.08124}
}
```

- [← previous ConvKN at SemEval-2016 Task 3: Answer and Question Selection for Question Answering on Arabic and English Fora](https://md.research.elamur.ai/free/en/papers/cedeno-2016-convkn-semeval-2016-task-3/)
- [next → CAMR at SemEval-2016 Task 8: An Extended Transition-based AMR Parser](https://md.research.elamur.ai/free/en/papers/wang-2016-camr-semeval-2016-task-8/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

53 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Krizhevsky, I. Sutskever, and G. Hinton, “[ImageNet Classification with Deep Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/),” in Proceedings of the Advances in Neural Information Processing Systems, 2012.
2. A. Graves and J. Schmidhuber, “Framewise Phoneme Classification With Bidirectional LSTM and Other Neural Network Architectures,” in Neural Networks, 2005. [find](https://scholar.google.com/scholar?q=A%2E+Graves+and+J%2E+Schmidhuber%2C+%E2%80%9CFramewise+Phoneme+Classification+With+Bidirectional+LSTM+and+Other+Neural+Network+Architectures%2C%E2%80%9D+in+Neural+Networks%2C+2005%2E)
3. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, “[Natural Language Processing (Almost) From Scratch](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/),” in arxiv.org, 2011.
4. Tensorflow, “https://www.tensorflow.org,” 2016. [find](https://scholar.google.com/scholar?q=Tensorflow%2C+%E2%80%9Chttps%3A%2F%2Fwww%2Etensorflow%2Eorg%2C%E2%80%9D+2016%2E)
5. Torch, “http://torch.ch,” 2016. [find](https://scholar.google.com/scholar?q=Torch%2C+%E2%80%9Chttp%3A%2F%2Ftorch%2Ech%2C%E2%80%9D+2016%2E)
6. Theano, “http://deeplearning.net/tutorial,” 2016. [find](https://scholar.google.com/scholar?q=Theano%2C+%E2%80%9Chttp%3A%2F%2Fdeeplearning%2Enet%2Ftutorial%2C%E2%80%9D+2016%2E)
7. Caffe, “http://caffe.berkeleyvision.org,” 2016. [find](https://scholar.google.com/scholar?q=Caffe%2C+%E2%80%9Chttp%3A%2F%2Fcaffe%2Eberkeleyvision%2Eorg%2C%E2%80%9D+2016%2E)
8. NVIDIA, “cuDNN: GPU Accelerated Deep Learning,” 2016. [find](https://scholar.google.com/scholar?q=NVIDIA%2C+%E2%80%9CcuDNN%3A+GPU+Accelerated+Deep+Learning%2C%E2%80%9D+2016%2E)
9. S. Bahrampour, N. Ramakrishnan, L. Schott, and M. Shah, “Comparative Study of Caffe, Neon, Theano, and Torch for Deep Learning,” in arxiv.org, 2016. [find](https://scholar.google.com/scholar?q=S%2E+Bahrampour%2C+N%2E+Ramakrishnan%2C+L%2E+Schott%2C+and+M%2E+Shah%2C+%E2%80%9CComparative+Study+of+Caffe%2C+Neon%2C+Theano%2C+and+Torch+for+Deep+Learning%2C%E2%80%9D+in+arxiv%2Eorg%2C+2016%2E)
10. The Next Platform (www.nextplatform.com), “Baidu Eyes Deep Learning Strategy in Wake of New GPU Options,” 2016. [find](https://scholar.google.com/scholar?q=The+Next+Platform+%28www%2Enextplatform%2Ecom%29%2C+%E2%80%9CBaidu+Eyes+Deep+Learning+Strategy+in+Wake+of+New+GPU+Options%2C%E2%80%9D+2016%2E)
11. G. Diamos, S. Sengupta, B. Catanzaro, M. Chrzanowski, A. Coates, E. Elsen, J. Engel, A. Hannun, and S. Satheesh, “Persistent RNNs: Stashing Recurrent Weights On-Chip,” in Proceedings of the International Conference on Machine Learning, 2016. [find](https://scholar.google.com/scholar?q=G%2E+Diamos%2C+S%2E+Sengupta%2C+B%2E+Catanzaro%2C+M%2E+Chrzanowski%2C+A%2E+Coates%2C+E%2E+Elsen%2C+J%2E+Engel%2C+A%2E+Hannun%2C+and+S%2E+Satheesh%2C+%E2%80%9CPersistent+RNNs%3A+Stashing+Recurrent+Weights+On%2DChip%2C%E2%80%9D+in+Proceedings+of+the+Internatio)
12. A. Krizhevsky, “[One Weird Trick For Parallelizing Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2014-one-weird-trick-parallelizing-convolutional/),” in arxiv.org, 2014.
13. ImageNet, “http://image-net.org,” 2016. [find](https://scholar.google.com/scholar?q=ImageNet%2C+%E2%80%9Chttp%3A%2F%2Fimage%2Dnet%2Eorg%2C%E2%80%9D+2016%2E)
14. K. Simonyan and A. Zisserman, “[Very Deep Convolutional Networks for Large-Scale Image Recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/),” in Proceedings of the International Conference on Learning Representations, 2015.
15. K. He, X. Zhang, S. Ren, and J. Sun, “[Deep Residual Learning for Image Recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/),” in arxiv.org, 2015.
16. Wired (www.wired.com), “Microsoft Neural Net Shows Deep Learning Can Get Way Deeper,” 2016. [find](https://scholar.google.com/scholar?q=Wired+%28www%2Ewired%2Ecom%29%2C+%E2%80%9CMicrosoft+Neural+Net+Shows+Deep+Learning+Can+Get+Way+Deeper%2C%E2%80%9D+2016%2E)
17. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going Deeper with Convolutions,” in arxiv.org, 2014. [find](https://scholar.google.com/scholar?q=C%2E+Szegedy%2C+W%2E+Liu%2C+Y%2E+Jia%2C+P%2E+Sermanet%2C+S%2E+Reed%2C+D%2E+Anguelov%2C+D%2E+Erhan%2C+V%2E+Vanhoucke%2C+and+A%2E+Rabinovich%2C+%E2%80%9CGoing+Deeper+with+Convolutions%2C%E2%80%9D+in+arxiv%2Eorg%2C+2014%2E)
18. G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger, “[Deep Networks with Stochastic Depth](https://md.research.elamur.ai/free/en/papers/huang-2016-deep-networks-stochastic-depth/),” in arxiv.org, 2016.
19. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” in Proceedings of the IEEE, 1998. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+L%2E+Bottou%2C+Y%2E+Bengio%2C+and+P%2E+Haffner%2C+%E2%80%9CGradient%2DBased+Learning+Applied+to+Document+Recognition%2C%E2%80%9D+in+Proceedings+of+the+IEEE%2C+1998%2E)
20. T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, and O. Temam, “DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning,” in Proceedings of International Conference on Architectural Support for Programming Languages and Operating Systems, 2014. [find](https://scholar.google.com/scholar?q=T%2E+Chen%2C+Z%2E+Du%2C+N%2E+Sun%2C+J%2E+Wang%2C+C%2E+Wu%2C+Y%2E+Chen%2C+and+O%2E+Temam%2C+%E2%80%9CDianNao%3A+a+small%2Dfootprint+high%2Dthroughput+accelerator+for+ubiquitous+machine%2Dlearning%2C%E2%80%9D+in+Proceedings+of+International+Conference+on+A)
21. Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang, L. Li, T. Chen, Z. Xu, N. Sun, and O. Temam, “DaDianNao: A Machine-Learning Supercomputer,” in Proceedings of ACM/IEEE International Symposium on Microarchitecture, 2014. [find](https://scholar.google.com/scholar?q=Y%2E+Chen%2C+T%2E+Luo%2C+S%2E+Liu%2C+S%2E+Zhang%2C+L%2E+He%2C+J%2E+Wang%2C+L%2E+Li%2C+T%2E+Chen%2C+Z%2E+Xu%2C+N%2E+Sun%2C+and+O%2E+Temam%2C+%E2%80%9CDaDianNao%3A+A+Machine%2DLearning+Supercomputer%2C%E2%80%9D+in+Proceedings+of+ACM%2FIEEE+International+Symposium+on+Mic)
22. Y. Chen, T. Krishna, J. Emer, and V. Sze, “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks,” in IEEE International Conference on Solid-State Circuits, 2016. [find](https://scholar.google.com/scholar?q=Y%2E+Chen%2C+T%2E+Krishna%2C+J%2E+Emer%2C+and+V%2E+Sze%2C+%E2%80%9CEyeriss%3A+An+Energy%2DEfficient+Reconfigurable+Accelerator+for+Deep+Convolutional+Neural+Networks%2C%E2%80%9D+in+IEEE+International+Conference+on+Solid%2DState+Circuits%2C+20)
23. S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. Horowitz, and W. Dally, “[EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://md.research.elamur.ai/free/en/papers/han-2016-eie-efficient-inference-engine-compressed/),” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016.
24. Y. Chen, J. Emer, and V. Sze, “Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. [find](https://scholar.google.com/scholar?q=Y%2E+Chen%2C+J%2E+Emer%2C+and+V%2E+Sze%2C+%E2%80%9CEyeriss%3A+A+Spatial+Architecture+for+Energy%2DEfficient+Dataflow+for+Convolutional+Neural+Networks%2C%E2%80%9D+in+Proceedings+of+ACM%2FIEEE+International+Symposium+on+Computer+Architec)
25. R. LiKamWa, Y. Hou, M. Polansky, Y. Gao, and L. Zhong, “RedEye: Analog ConvNet Image Sensor Architecture for Continuous Mobile Vision,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. [find](https://scholar.google.com/scholar?q=R%2E+LiKamWa%2C+Y%2E+Hou%2C+M%2E+Polansky%2C+Y%2E+Gao%2C+and+L%2E+Zhong%2C+%E2%80%9CRedEye%3A+Analog+ConvNet+Image+Sensor+Architecture+for+Continuous+Mobile+Vision%2C%E2%80%9D+in+Proceedings+of+ACM%2FIEEE+International+Symposium+on+Computer+A)
26. B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee, S. Lee, J. Miguel, H. Lobato, G. Wei, and D. Brooks, “Minerva: Enabling Low-Power, High-Accuracy Deep Neural Network Accelerators,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. [find](https://scholar.google.com/scholar?q=B%2E+Reagen%2C+P%2E+Whatmough%2C+R%2E+Adolf%2C+S%2E+Rama%2C+H%2E+Lee%2C+S%2E+Lee%2C+J%2E+Miguel%2C+H%2E+Lobato%2C+G%2E+Wei%2C+and+D%2E+Brooks%2C+%E2%80%9CMinerva%3A+Enabling+Low%2DPower%2C+High%2DAccuracy+Deep+Neural+Network+Accelerators%2C%E2%80%9D+in+Proceedings+o)
27. P. Chi, S. Li, C. Xu, T. Zhang, J. Zhao, Y. Liu, Y. Wang, and Y. Xie, “A Novel Processing-in-memory Architecture for Neural Network Computation in ReRAM-based Main Memory,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. [find](https://scholar.google.com/scholar?q=P%2E+Chi%2C+S%2E+Li%2C+C%2E+Xu%2C+T%2E+Zhang%2C+J%2E+Zhao%2C+Y%2E+Liu%2C+Y%2E+Wang%2C+and+Y%2E+Xie%2C+%E2%80%9CA+Novel+Processing%2Din%2Dmemory+Architecture+for+Neural+Network+Computation+in+ReRAM%2Dbased+Main+Memory%2C%E2%80%9D+in+Proceedings+of+ACM%2FIEEE+)
28. A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubra-monian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, “ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. [find](https://scholar.google.com/scholar?q=A%2E+Shafiee%2C+A%2E+Nag%2C+N%2E+Muralimanohar%2C+R%2E+Balasubra%2Dmonian%2C+J%2E+P%2E+Strachan%2C+M%2E+Hu%2C+R%2E+S%2E+Williams%2C+and+V%2E+Srikumar%2C+%E2%80%9CISAAC%3A+A+Convolutional+Neural+Network+Accelerator+with+In%2DSitu+Analog+Arithmetic+in+)
29. J. Albericio, P. Judd, T. Hetherington, T. Aamodt, N. E. Jerger, and A. Moshovos, “Cnvlutin: Ineffectual-Neuron-Free Deep Convolutional Neural Network Computing,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. [find](https://scholar.google.com/scholar?q=J%2E+Albericio%2C+P%2E+Judd%2C+T%2E+Hetherington%2C+T%2E+Aamodt%2C+N%2E+E%2E+Jerger%2C+and+A%2E+Moshovos%2C+%E2%80%9CCnvlutin%3A+Ineffectual%2DNeuron%2DFree+Deep+Convolutional+Neural+Network+Computing%2C%E2%80%9D+in+Proceedings+of+ACM%2FIEEE+Internatio)
30. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “[OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks](https://md.research.elamur.ai/free/en/papers/sermanet-2013-overfeat-integrated-recognition-localization-detection/),” in arxiv.org, 2013.
31. S. Chintala, “https://github.com/torch/nn/pull/235,” 2015. [find](https://scholar.google.com/scholar?q=S%2E+Chintala%2C+%E2%80%9Chttps%3A%2F%2Fgithub%2Ecom%2Ftorch%2Fnn%2Fpull%2F235%2C%E2%80%9D+2015%2E)
32. S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “[cuDNN: Efficient Primitives for Deep Learning](https://md.research.elamur.ai/free/en/papers/chetlur-2014-cudnn-efficient-primitives-deep-learning/),” in Proceedings of the Advances in Neural Information Processing Systems, 2014.
33. OpenMP Architecture Review Board, “OpenMP Application Program Interface (version 4.0),” 2013. [find](https://scholar.google.com/scholar?q=OpenMP+Architecture+Review+Board%2C+%E2%80%9COpenMP+Application+Program+Interface+%28version+4%2E0%29%2C%E2%80%9D+2013%2E)
34. T. Zheng, D. Nellans, A. Zulfiqar, M. Stephenson, and S. W. Keckler, “Toward High-Performance Paged-Memory for GPUs,” in Proceedings of IEEE International Symposium on High-Performance Computer Architecture, 2016. [find](https://scholar.google.com/scholar?q=T%2E+Zheng%2C+D%2E+Nellans%2C+A%2E+Zulfiqar%2C+M%2E+Stephenson%2C+and+S%2E+W%2E+Keckler%2C+%E2%80%9CToward+High%2DPerformance+Paged%2DMemory+for+GPUs%2C%E2%80%9D+in+Proceedings+of+IEEE+International+Symposium+on+High%2DPerformance+Computer+Archit)
35. NVIDIA, “NVIDIA NVLINK High-Speed Interconnect,” 2016. [find](https://scholar.google.com/scholar?q=NVIDIA%2C+%E2%80%9CNVIDIA+NVLINK+High%2DSpeed+Interconnect%2C%E2%80%9D+2016%2E)
36. NVIDIA, “NVIDIA CUDA Programming Guide,” 2016. [find](https://scholar.google.com/scholar?q=NVIDIA%2C+%E2%80%9CNVIDIA+CUDA+Programming+Guide%2C%E2%80%9D+2016%2E)
37. NVIDIA, “https://github.com/NVIDIA/cnmem,” 2016. [find](https://scholar.google.com/scholar?q=NVIDIA%2C+%E2%80%9Chttps%3A%2F%2Fgithub%2Ecom%2FNVIDIA%2Fcnmem%2C%E2%80%9D+2016%2E)
38. S. Gross and M. Wilber, “Training and Investigating Residual Nets,” 2016. [find](https://scholar.google.com/scholar?q=S%2E+Gross+and+M%2E+Wilber%2C+%E2%80%9CTraining+and+Investigating+Residual+Nets%2C%E2%80%9D+2016%2E)
39. T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang, “[MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://md.research.elamur.ai/free/en/papers/chen-2015-mxnet-flexible-efficient-machine-learning/),” in Proceedings of the 2015 Workshop on Machine Learning Systems, 2015.
40. NVIDIA, “GeForce GTX Titan X (Maxwell),” 2015. [find](https://scholar.google.com/scholar?q=NVIDIA%2C+%E2%80%9CGeForce+GTX+Titan+X+%28Maxwell%29%2C%E2%80%9D+2015%2E)
41. S. Chintala, “https://github.com/soumith/convnet-benchmarks,” 2016. [find](https://scholar.google.com/scholar?q=S%2E+Chintala%2C+%E2%80%9Chttps%3A%2F%2Fgithub%2Ecom%2Fsoumith%2Fconvnet%2Dbenchmarks%2C%E2%80%9D+2016%2E)
42. NVIDIA, “CUDA Toolkit 7.5 Documentation: Profiler,” 2016. [find](https://scholar.google.com/scholar?q=NVIDIA%2C+%E2%80%9CCUDA+Toolkit+7%2E5+Documentation%3A+Profiler%2C%E2%80%9D+2016%2E)
43. S. Hanson and L. Pratt, “Comparing Biases for Minimal Network Construction with Back-propagation,” in Proceedings of the Advances in Neural Information Processing Systems, 1989. [find](https://scholar.google.com/scholar?q=S%2E+Hanson+and+L%2E+Pratt%2C+%E2%80%9CComparing+Biases+for+Minimal+Network+Construction+with+Back%2Dpropagation%2C%E2%80%9D+in+Proceedings+of+the+Advances+in+Neural+Information+Processing+Systems%2C+1989%2E)
44. Y. LeCun, S. Denker, and S. Solla, “Optimal Brain Damage,” in Proceedings of the Advances in Neural Information Processing Systems, 1990. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+S%2E+Denker%2C+and+S%2E+Solla%2C+%E2%80%9COptimal+Brain+Damage%2C%E2%80%9D+in+Proceedings+of+the+Advances+in+Neural+Information+Processing+Systems%2C+1990%2E)
45. B. Hassibi and D. Stork, “Second Order Derivatives for Network Pruning: Optimal Brain Surgeon,” in Proceedings of the Advances in Neural Information Processing Systems, 1993. [find](https://scholar.google.com/scholar?q=B%2E+Hassibi+and+D%2E+Stork%2C+%E2%80%9CSecond+Order+Derivatives+for+Network+Pruning%3A+Optimal+Brain+Surgeon%2C%E2%80%9D+in+Proceedings+of+the+Advances+in+Neural+Information+Processing+Systems%2C+1993%2E)
46. S. Han, J. Pool, J. Tran, and W. Dally, “Learning Both Weights and Connections for Efficient Neural Networks,” in Proceedings of the Advances in Neural Information Processing Systems, 2015. [find](https://scholar.google.com/scholar?q=S%2E+Han%2C+J%2E+Pool%2C+J%2E+Tran%2C+and+W%2E+Dally%2C+%E2%80%9CLearning+Both+Weights+and+Connections+for+Efficient+Neural+Networks%2C%E2%80%9D+in+Proceedings+of+the+Advances+in+Neural+Information+Processing+Systems%2C+2015%2E)
47. S. Han, H. Mao, and W. Dally, “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding,” in Proceedings of the International Conference on Learning Representations, 2016. [find](https://scholar.google.com/scholar?q=S%2E+Han%2C+H%2E+Mao%2C+and+W%2E+Dally%2C+%E2%80%9CDeep+Compression%3A+Compressing+Deep+Neural+Networks+with+Pruning%2C+Trained+Quantization+and+Huffman+Coding%2C%E2%80%9D+in+Proceedings+of+the+International+Conference+on+Learning+Rep)
48. V. Vanhoucke, A. Senior, and M. Mao, “Improving the Speed of Neural Networks on CPUs,” in Proceedings of Deep Learning and Unsupervised Feature Learning, 2011. [find](https://scholar.google.com/scholar?q=V%2E+Vanhoucke%2C+A%2E+Senior%2C+and+M%2E+Mao%2C+%E2%80%9CImproving+the+Speed+of+Neural+Networks+on+CPUs%2C%E2%80%9D+in+Proceedings+of+Deep+Learning+and+Unsupervised+Feature+Learning%2C+2011%2E)
49. P. Judd, J. Albericio, T. Hetherington, T. Aamodt, N. E. Jerger, R. Urtasun, and A. Moshovos, “Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets,” in arxiv.org, 2016. [find](https://scholar.google.com/scholar?q=P%2E+Judd%2C+J%2E+Albericio%2C+T%2E+Hetherington%2C+T%2E+Aamodt%2C+N%2E+E%2E+Jerger%2C+R%2E+Urtasun%2C+and+A%2E+Moshovos%2C+%E2%80%9CReduced%2DPrecision+Strategies+for+Bounded+Memory+in+Deep+Neural+Nets%2C%E2%80%9D+in+arxiv%2Eorg%2C+2016%2E)
50. Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing Deep Convolutional Networks Using Vector Quantization,” in arxiv.org, 2014. [find](https://scholar.google.com/scholar?q=Y%2E+Gong%2C+L%2E+Liu%2C+M%2E+Yang%2C+and+L%2E+Bourdev%2C+%E2%80%9CCompressing+Deep+Convolutional+Networks+Using+Vector+Quantization%2C%E2%80%9D+in+arxiv%2Eorg%2C+2014%2E)
51. B. Pichai, L. Hsu, and A. Bhattacharjee, “Architectural Support for Address Translation on GPUs: Designing Memory Management Units for CPU/GPUs with Unified Address Spaces,” in Proceedings of ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2014. [find](https://scholar.google.com/scholar?q=B%2E+Pichai%2C+L%2E+Hsu%2C+and+A%2E+Bhattacharjee%2C+%E2%80%9CArchitectural+Support+for+Address+Translation+on+GPUs%3A+Designing+Memory+Management+Units+for+CPU%2FGPUs+with+Unified+Address+Spaces%2C%E2%80%9D+in+Proceedings+of+ACM+Inte)
52. J. Power, M. Hill, and D. Wood, “Supporting x86-64 Address Translation for 100s of GPU Lanes,” in Proceedings of IEEE International Symposium on High-Performance Computer Architecture, 2014. [find](https://scholar.google.com/scholar?q=J%2E+Power%2C+M%2E+Hill%2C+and+D%2E+Wood%2C+%E2%80%9CSupporting+x86%2D64+Address+Translation+for+100s+of+GPU+Lanes%2C%E2%80%9D+in+Proceedings+of+IEEE+International+Symposium+on+High%2DPerformance+Computer+Architecture%2C+2014%2E)
53. Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo, X. Feng, Y. Chen, and O. Temam, “ShiDianNao: Shifting Vision Processing Closer to the Sensor,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2015. [find](https://scholar.google.com/scholar?q=Z%2E+Du%2C+R%2E+Fasthuber%2C+T%2E+Chen%2C+P%2E+Ienne%2C+L%2E+Li%2C+T%2E+Luo%2C+X%2E+Feng%2C+Y%2E+Chen%2C+and+O%2E+Temam%2C+%E2%80%9CShiDianNao%3A+Shifting+Vision+Processing+Closer+to+the+Sensor%2C%E2%80%9D+in+Proceedings+of+ACM%2FIEEE+International+Symposium)

*Updated 2 November 2026*