---
title: "cuDNN: Efficient Primitives for Deep Learning (2014)"
description: "The authors propose cuDNN, an NVIDIA library that plays the role BLAS plays for linear algebra: a… 21 references, cited by 297 papers here."
authors: ["Sharan Chetlur", "Cliff Woolley", "Philippe Vandermersch", "Jonathan Cohen", "John Tran", "Bryan Catanzaro", "Evan Shelhamer"]
year: 2014
venue: "arXiv 2014 (NVIDIA)"
source: "https://arxiv.org/abs/1410.0759"
arxiv: "1410.0759"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chetlur-2014-cudnn-efficient-primitives-deep-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chetlur-2014-cudnn-efficient-primitives-deep-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/) / cuDNN: Efficient Primitives for Deep Learning

# cuDNN: Efficient Primitives for Deep Learning

**Authors:** Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, [Jonathan Cohen](https://md.research.elamur.ai/free/en/authors/jonathan-cohen/), John Tran, [Bryan Catanzaro](https://md.research.elamur.ai/free/en/authors/bryan-catanzaro/) and [Evan Shelhamer](https://md.research.elamur.ai/free/en/authors/evan-shelhamer/) · **Year:** 2014 · **Venue:** arXiv 2014 (NVIDIA) · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1410.0759) · [PDF](https://arxiv.org/pdf/1410.0759) · [arXiv 1410.0759](https://arxiv.org/abs/1410.0759) · [OpenAlex](https://openalex.org/W1667652561)

**Citations:** 943 citations (OpenAlex, 2026-10-14) · [297](https://md.research.elamur.ai/free/en/papers/chetlur-2014-cudnn-efficient-primitives-deep-learning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Deep learning frameworks such as Caffe, Theano and Torch7 each maintained hand-tuned GPU kernels for convolution, pooling and activations. That code is hard to write and must be re-tuned whenever hardware changes. The authors propose cuDNN, an NVIDIA library that plays the role BLAS plays for linear algebra: a shared set of optimized deep learning primitives behind a low-level C API.

cuDNN offers forward and backward routines for:
- convolution and cross-correlation,
- pooling,
- activation functions,
- softmax,
- tensor transformations.

All routines work in single and double precision on GPU-resident buffers that the caller owns. Its convolution builds on fast matrix multiplication but avoids the large auxiliary buffers that explicit lowering needs, which saves scarce GPU memory.

Plugging cuDNN into Caffe cut overall training time by 36% on a reference model while also lowering memory use. At Baidu, it sped up convolutional layers in PADDLE by about 30%. The broader point is that frameworks can leave hardware-specific tuning to the library and still get performance as GPUs evolve.

## Key points
- **API design:** cuDNN exposes a host-callable, thread-safe C API modeled on cuBLAS, with data kept on the GPU. Tensors and filters are described by opaque descriptors that accept arbitrary per-dimension strides, so frameworks keep their own memory layouts.
- **Supported operations:**
  - convolution in both true and cross-correlation forms, with stride and zero-padding parameters,
  - average and max pooling,
  - Sigmoid, ReLU and Tanh,
  - numerically stable softmax,
  - tensor addition with broadcasting.
- **Why lowering is costly:** explicit lowering reshapes filters into a K × CRS matrix and copies input data into a CRS × NPQ matrix. Each input element can then appear up to RS times, which is the memory overhead cuDNN avoids.
- **Caffe results:** with cuDNN R1 on a Tesla K40, training bvlc reference caffenet for 200 iterations gave these speedups:
  - forward pass: 1.45×
  - backward pass: 1.30×
  - testing: 1.49×
  - overall: 1.36×
- **Baidu results:** in PADDLE on a Tesla K10, convolution benchmarks ran about 30% faster than a matrix-multiply lowering approach. Baidu also uses cuDNN for speech and language, where convolving non-square inputs with asymmetric padding is useful.

## Interesting ideas
- **Primitives rather than layers:** the library provides low-level operations instead of whole layer abstractions. This lets very different frameworks adopt it without bending their own designs, which is a good pattern for any shared toolkit, for example tokenization or alignment routines.
- **Layout described by strides:** callers describe their existing tensors with strides instead of converting them into a required format. This removes copying and integration friction, and could apply to libraries for corpora or annotation arrays.
- **Memory as a first-class goal:** the design aims to fill GPU memory with data and parameters, not algorithm scratch space. Treating memory footprint as a design criterion matters wherever bigger models or batches are the real bottleneck.
- **Drop-in integration with fallback:** Caffe enables cuDNN layers with one compile flag and lets each layer choose its implementation via an ENGINE parameter. Unsupported cases automatically fall back to the original code. This is a low-risk way to introduce an accelerated backend into an existing pipeline.
- **Accumulate-or-overwrite gradient mode:** routines can either return raw gradients or add them into an existing buffer. This small option directly supports shared parameters and graph-shaped models, and is worth copying in any autodiff-style component.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 209 works · 244,782 citations
- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 84 works · 17,681 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

## Cite

Cite as: Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro and Evan Shelhamer. 2014. cuDNN: Efficient Primitives for Deep Learning. arXiv 2014 (NVIDIA).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chetlur2014cudnn,
  title = {{cuDNN: Efficient Primitives for Deep Learning}},
  author = {Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro and Evan Shelhamer},
  year = {2014},
  eprint = {1410.0759},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1410.0759}
}
```

- [← previous ImageNet Large Scale Visual Recognition Challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
- [next → Adaptive Recursive Neural Network for Target-dependent Twitter Sentiment Classification](https://md.research.elamur.ai/free/en/papers/dong-2014-adaptive-recursive-neural-network-target/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/)

## References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. convnet-benchmarks. [https://github.com/soumith/convnet-benchmarks](https://github.com/soumith/convnet-benchmarks), 2014. [↗](https://github.com/soumith/convnet-benchmarks)
2. Netlib BLAS. [http://www.netlib.org/blas/](http://www.netlib.org/blas/), 2014. [↗](http://www.netlib.org/blas/)
3. NVIDIA cuDNN - GPU accelerated deep learning. [https://developer.nvidia.com/cuDNN](https://developer.nvidia.com/cuDNN), 2014. [↗](https://developer.nvidia.com/cuDNN)
4. Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Unsupervised feature learning and deep learning: A review and new perspectives. CoRR, abs/1206.[5538, 2012](https://md.research.elamur.ai/free/en/papers/bengio-2012-representation-learning-review-new-perspectives/).
5. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a cpu and gpu math expression compiler. In SciPy, volume 4, page 3, 2010. [find](https://scholar.google.com/scholar?q=James+Bergstra%2C+Olivier+Breuleux%2C+Fre%CC%81de%CC%81ric+Bastien%2C+Pascal+Lamblin%2C+Razvan+Pascanu%2C+Guillaume+Desjardins%2C+Joseph+Turian%2C+David+Warde%2DFarley%2C+and+Yoshua+Bengio%2E+Theano%3A+a+cpu+and+gpu+math+expression+)
6. Kumar Chellapilla, Sidd Puri, Patrice Simard, et al. High performance convolutional neural networks for document processing. In Workshop on Frontiers in Handwriting Recognition, 2006. [find](https://scholar.google.com/scholar?q=Kumar+Chellapilla%2C+Sidd+Puri%2C+Patrice+Simard%2C+et+al%2E+High+performance+convolutional+neural+networks+for+document+processing%2E+In+Workshop+on+Frontiers+in+Handwriting+Recognition%2C+2006%2E)
7. Adam Coates, Brody Huval, Tao Wang, David Wu, Bryan Catanzaro, and Andrew Ng. Deep learning with COTS HPC systems. In ICML, pages 1337–1345, 2013. [find](https://scholar.google.com/scholar?q=Adam+Coates%2C+Brody+Huval%2C+Tao+Wang%2C+David+Wu%2C+Bryan+Catanzaro%2C+and+Andrew+Ng%2E+Deep+learning+with+COTS+HPC+systems%2E+In+ICML%2C+pages+1337%E2%80%931345%2C+2013%2E)
8. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, 2011. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Koray+Kavukcuoglu%2C+and+Cle%CC%81ment+Farabet%2E+Torch7%3A+A+matlab%2Dlike+environment+for+machine+learning%2E+In+BigLearn%2C+NIPS+Workshop%2C+2011%2E)
9. George E Dahl, Dong Yu, Li Deng, and Alex Acero. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. Audio, Speech, and Language Processing, IEEE Transactions on, 20(1):30–42, 2012. [find](https://scholar.google.com/scholar?q=George+E+Dahl%2C+Dong+Yu%2C+Li+Deng%2C+and+Alex+Acero%2E+Context%2Ddependent+pre%2Dtrained+deep+neural+networks+for+large%2Dvocabulary+speech+recognition%2E+Audio%2C+Speech%2C+and+Language+Processing%2C+IEEE+Transactions+o)
10. Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEE… [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Li+Deng%2C+Dong+Yu%2C+George+E+Dahl%2C+Abdel%2Drahman+Mohamed%2C+Navdeep+Jaitly%2C+Andrew+Senior%2C+Vincent+Vanhoucke%2C+Patrick+Nguyen%2C+Tara+N+Sainath%2C+et+al%2E+Deep+neural+networks+for+acoustic+model)
11. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. [Caffe: Convolutional architecture for fast feature embedding](https://md.research.elamur.ai/free/en/papers/jia-2014-caffe-convolutional-architecture-fast-feature/). arXiv preprint arXiv:1408.5093, 2014.
12. Alex Krizhevsky. cudaconvnet2. [https://code.google.com/p/cuda-convnet2/](https://code.google.com/p/cuda-convnet2/), 2014. [↗](https://code.google.com/p/cuda-convnet2/)
13. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In NIPS, pages 1097–1105, 2012.
14. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Le%CC%81on+Bottou%2C+Yoshua+Bengio%2C+and+Patrick+Haffner%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+1998%2E)
15. Andrew L Maas, Awni Y Hannun, Daniel Jurafsky, and Andrew Y Ng. First-pass large vocabulary continuous speech recognition using bi-directional recurrent dnns. arXiv preprint arXiv:1408.2873, 2014. [find](https://scholar.google.com/scholar?q=Andrew+L+Maas%2C+Awni+Y+Hannun%2C+Daniel+Jurafsky%2C+and+Andrew+Y+Ng%2E+First%2Dpass+large+vocabulary+continuous+speech+recognition+using+bi%2Ddirectional+recurrent+dnns%2E+arXiv+preprint+arXiv%3A1408%2E2873%2C+2014%2E)
16. Michael Mathieu, Mikael Henaff, and Yann LeCun. Fast training of convolutional networks through ffts. arXiv preprint arXiv:1312.5851, 2013. — [arXiv:1312.5851](https://arxiv.org/abs/1312.5851) [find](https://scholar.google.com/scholar?q=Michael+Mathieu%2C+Mikael+Henaff%2C+and+Yann+LeCun%2E+Fast+training+of+convolutional+networks+through+ffts%2E+arXiv+preprint+arXiv%3A1312%2E5851%2C+2013%2E+%E2%80%94+%5BarXiv%3A1312%2E5851%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1312%2E5851%29)
17. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. [Imagenet large scale visual recognition challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/), 2014.
18. Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann Le-Cun. [Overfeat: Integrated recognition, localization and detection using convolutional networks](https://md.research.elamur.ai/free/en/papers/sermanet-2013-overfeat-integrated-recognition-localization-detection/). arXiv preprint arXiv:1312.6229, 2013.
19. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wei+Liu%2C+Yangqing+Jia%2C+Pierre+Sermanet%2C+Scott+Reed%2C+Dragomir+Anguelov%2C+Dumitru+Erhan%2C+Vincent+Vanhoucke%2C+and+Andrew+Rabinovich%2E+Going+deeper+with+convolutions%2E+arXiv+preprint+arXiv%3A)
20. Guangming Tan, Linchuan Li, Sean Treichler, Everett Phillips, Yungang Bao, and Ninghui Sun. Fast implementation of DGEMM on Fermi GPU. In Supercomputing 2011, SC ’11, pages 35:1–35:11, New York, NY, USA, 2011. ACM. [find](https://scholar.google.com/scholar?q=Guangming+Tan%2C+Linchuan+Li%2C+Sean+Treichler%2C+Everett+Phillips%2C+Yungang+Bao%2C+and+Ninghui+Sun%2E+Fast+implementation+of+DGEMM+on+Fermi+GPU%2E+In+Supercomputing+2011%2C+SC+%E2%80%9911%2C+pages+35%3A1%E2%80%9335%3A11%2C+New+York%2C+NY%2C+U)
21. Henry S. Warren. Hacker’s Delight. Addison-Wesley Professional, 2002. [find](https://scholar.google.com/scholar?q=Henry+S%2E+Warren%2E+Hacker%E2%80%99s+Delight%2E+Addison%2DWesley+Professional%2C+2002%2E)

*Updated 2 November 2026*