---
title: "Binarized Neural Networks — paper by Courbariaux et al. (2016)"
description: "The authors propose Binarized Neural Networks (BNNs), in which weights and hidden activations… 55 references, cited by 128 papers here."
authors: ["Matthieu Courbariaux", "Itay Hubara", "Daniel Soudry", "Ran El-Yaniv", "Yoshua Bengio"]
year: 2016
venue: "arXiv:1602.02830"
source: "https://arxiv.org/abs/1602.02830"
doi: "10.48550/arxiv.1602.02830"
arxiv: "1602.02830"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/courbariaux-2016-binarized-neural-networks-training-deep/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/courbariaux-2016-binarized-neural-networks-training-deep/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

# Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

**Authors:** [Matthieu Courbariaux](https://md.research.elamur.ai/free/en/authors/matthieu-courbariaux/), [Itay Hubara](https://md.research.elamur.ai/free/en/authors/itay-hubara/), [Daniel Soudry](https://md.research.elamur.ai/free/en/authors/daniel-soudry/), [Ran El-Yaniv](https://md.research.elamur.ai/free/en/authors/ran-el-yaniv/) and [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/) · **Year:** 2016 · **Venue:** arXiv:1602.02830 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1602.02830) · [PDF](https://arxiv.org/pdf/1602.02830) · [DOI 10.48550/arxiv.1602.02830](https://doi.org/10.48550/arxiv.1602.02830) · [arXiv 1602.02830](https://arxiv.org/abs/1602.02830) · [OpenAlex](https://openalex.org/W2319920447)

**Citations:** 2,015 citations (OpenAlex, 2026-10-14) · [128](https://md.research.elamur.ai/free/en/papers/courbariaux-2016-binarized-neural-networks-training-deep/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This work tackles the cost of running deep networks on low-power hardware. Modern DNNs depend on energy-hungry GPUs and full-precision arithmetic. The authors propose Binarized Neural Networks (BNNs), in which weights and hidden activations take only the values +1 or −1. These binary values are used both at inference and during the forward computation of gradients in training. Real-valued copies of the weights are kept only to accumulate small SGD updates. Gradients pass through the non-differentiable sign function via a saturating straight-through estimator. To remove most multiplications from training, the authors also introduce shift-based variants of batch normalization and AdaMax. Experiments in both Torch7 and Theano on MNIST, CIFAR-10 and SVHN reach accuracy close to the state of the art of the time. A custom binary matrix-multiplication GPU kernel runs the MNIST network 7 times faster than an unoptimized kernel, with no accuracy loss. This matters because it shows that whole networks can be reduced to bit-wise (XNOR/popcount) operations, which opens a path to very efficient dedicated hardware.

## Key points
- Two binarization schemes are used. The deterministic one is the sign function. The stochastic one outputs +1 with probability equal to a "hard sigmoid" of the input. The deterministic scheme is used almost everywhere, because random bits are expensive in hardware.
- The gradient estimator through sign() is the incoming gradient multiplied by an indicator that |r| ≤ 1, which equals the derivative of hard tanh. The authors report that dropping this cancellation for large inputs clearly hurts performance.
- Real-valued weights are clipped to [−1, 1] after each update. Without clipping they would drift without affecting their binary versions.
- Shift-based BatchNorm and shift-based AdaMax replace multiplications with power-of-two approximations and bit shifts. The suggested AdaMax defaults are α = 2⁻¹⁰, 1−β₁ = 2⁻³ and 1−β₂ = 2⁻¹⁰. No accuracy loss was observed compared with vanilla BN.
- Code for both frameworks is released as BinaryNet on GitHub. The authors also argue that repeated binary convolution kernels could let dedicated hardware cut time complexity by about 60%.

## Interesting ideas
- **Two copies of each weight.** A low-precision copy is used for computation, while a high-precision "shadow" copy accumulates gradient updates. This is a general recipe for training any heavily quantized or discrete model, because tiny noisy SGD steps would otherwise be lost to rounding.
- **Saturating straight-through estimator.** The gradient passes through a hard threshold unchanged, but is zeroed where the input is already saturated. This cheap trick can be reused wherever a model contains discrete decisions, such as hashing layers or discrete codes.
- **Quantization as regularization.** The authors view binarization noise as a relative of Dropout and DropConnect. This suggests that compression and generalization need not be in tension, which is a useful framing when deciding how aggressively to compress a model.
- **Bit-plane handling of non-binary input.** The first layer receives 8-bit inputs, so the network splits them into 8 bit planes. It computes a binary dot product for each plane and sums the results weighted by powers of two. The approach keeps the whole pipeline binary-friendly, and similar decompositions could apply to other fixed-point inputs.
- **Power-of-two approximations.** Costly multiplications in normalization and optimizer updates are replaced with shifts based on the most significant bit. This is a hardware-friendly pattern applicable to edge deployment of many models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 188 works · [⚙️ Electrical and Electronic Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/electrical-and-electronic-engineering/) · 33 works

## Cite

Cite as: Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv and Yoshua Bengio. 2016. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. arXiv:1602.02830.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{courbariaux2016binarized,
  title = {{Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1}},
  author = {Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv and Yoshua Bengio},
  year = {2016},
  doi = {10.48550/arxiv.1602.02830},
  eprint = {1602.02830},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1602.02830}
}
```

- [← previous Deep Networks with Stochastic Depth](https://md.research.elamur.ai/free/en/papers/huang-2016-deep-networks-stochastic-depth/)
- [next → Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

55 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In ICLR’2015, arXiv:1409.0473, 2015.
2. Baldassi, Carlo, Ingrosso, Alessandro, Lucibello, Carlo, Saglietti, Luca, and Zecchina, Riccardo. Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses. Physical Review Letters, 115(12):1–5, 2015. [find](https://scholar.google.com/scholar?q=Baldassi%2C+Carlo%2C+Ingrosso%2C+Alessandro%2C+Lucibello%2C+Carlo%2C+Saglietti%2C+Luca%2C+and+Zecchina%2C+Riccardo%2E+Subdominant+Dense+Clusters+Allow+for+Simple+Learning+and+High+Computational+Performance+in+Neural+Netw)
3. Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian J., Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. [Theano: new features and speed improvements](https://md.research.elamur.ai/free/en/papers/bastien-2012-theano-new-features-speed-improvements/). Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
4. Beauchamp, Michael J, Hauck, Scott, Underwood, Keith D, and Hemmert, K Scott. Embedded floating-point units in FPGAs. In Proceedings of the 2006 ACM/SIGDA 14th international symposium on Field programmable gate arrays, pp. 12–20. [find](https://scholar.google.com/scholar?q=Beauchamp%2C+Michael+J%2C+Hauck%2C+Scott%2C+Underwood%2C+Keith+D%2C+and+Hemmert%2C+K+Scott%2E+Embedded+floating%2Dpoint+units+in+FPGAs%2E+In+Proceedings+of+the+2006+ACM%2FSIGDA+14th+international+symposium+on+Field+program)
5. Bengio, Yoshua. Estimating or propagating gradients through stochastic neurons. Technical Report arXiv:1305.2982, Universite de Montreal, 2013. — [arXiv:1305.2982](https://arxiv.org/abs/1305.2982) [find](https://scholar.google.com/scholar?q=Bengio%2C+Yoshua%2E+Estimating+or+propagating+gradients+through+stochastic+neurons%2E+Technical+Report+arXiv%3A1305%2E2982%2C+Universite+de+Montreal%2C+2013%2E+%E2%80%94+%5BarXiv%3A1305%2E2982%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1305%2E2982%29)
6. Bergstra, James, Breuleux, Olivier, Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 20… [find](https://scholar.google.com/scholar?q=Bergstra%2C+James%2C+Breuleux%2C+Olivier%2C+Bastien%2C+Fre%CC%81de%CC%81ric%2C+Lamblin%2C+Pascal%2C+Pascanu%2C+Razvan%2C+Desjardins%2C+Guillaume%2C+Turian%2C+Joseph%2C+Warde%2DFarley%2C+David%2C+and+Bengio%2C+Yoshua%2E+Theano%3A+a+CPU+and+GPU+math+ex)
7. Chen, Tianshi, Du, Zidong, Sun, Ninghui, Wang, Jia, Wu, Chengyong, Chen, Yunji, and Temam, Olivier. Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning. In Proceedings of the 19th international conference on Architectural support for programming languages and opera… [find](https://scholar.google.com/scholar?q=Chen%2C+Tianshi%2C+Du%2C+Zidong%2C+Sun%2C+Ninghui%2C+Wang%2C+Jia%2C+Wu%2C+Chengyong%2C+Chen%2C+Yunji%2C+and+Temam%2C+Olivier%2E+Diannao%3A+A+small%2Dfootprint+high%2Dthroughput+accelerator+for+ubiquitous+machine%2Dlearning%2E+In+Proceedin)
8. Chen, Yunji, Luo, Tao, Liu, Shaoli, Zhang, Shijin, He, Liqiang, Wang, Jia, Li, Ling, Chen, Tianshi, Xu, Zhiwei, Sun, Ninghui, et al. Dadiannao: A machine-learning supercomputer. In Microarchitecture (MICRO), 2014 47th Annual IEEE/ACM International Symposium on, pp. 609–622. [find](https://scholar.google.com/scholar?q=Chen%2C+Yunji%2C+Luo%2C+Tao%2C+Liu%2C+Shaoli%2C+Zhang%2C+Shijin%2C+He%2C+Liqiang%2C+Wang%2C+Jia%2C+Li%2C+Ling%2C+Chen%2C+Tianshi%2C+Xu%2C+Zhiwei%2C+Sun%2C+Ninghui%2C+et+al%2E+Dadiannao%3A+A+machine%2Dlearning+supercomputer%2E+In+Microarchitecture+%28)
9. Cheng, Zhiyong, Soudry, Daniel, Mao, Zexi, and Lan, Zhenzhong. Training binary multilayer neural networks for image classification using expectation backpropgation. arXiv preprint arXiv:1503.03562, 2015. — [arXiv:1503.03562](https://arxiv.org/abs/1503.03562) [find](https://scholar.google.com/scholar?q=Cheng%2C+Zhiyong%2C+Soudry%2C+Daniel%2C+Mao%2C+Zexi%2C+and+Lan%2C+Zhenzhong%2E+Training+binary+multilayer+neural+networks+for+image+classification+using+expectation+backpropgation%2E+arXiv+preprint+arXiv%3A1503%2E03562%2C+20)
10. Coates, Adam, Huval, Brody, Wang, Tao, Wu, David, Catanzaro, Bryan, and Andrew, Ng. Deep learning with COTS HPC systems. In Proceedings of the 30th international conference on machine learning, pp. 1337–1345, 2013. [find](https://scholar.google.com/scholar?q=Coates%2C+Adam%2C+Huval%2C+Brody%2C+Wang%2C+Tao%2C+Wu%2C+David%2C+Catanzaro%2C+Bryan%2C+and+Andrew%2C+Ng%2E+Deep+learning+with+COTS+HPC+systems%2E+In+Proceedings+of+the+30th+international+conference+on+machine+learning%2C+pp%2E+13)
11. Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clément. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, 2011. [find](https://scholar.google.com/scholar?q=Collobert%2C+Ronan%2C+Kavukcuoglu%2C+Koray%2C+and+Farabet%2C+Cle%CC%81ment%2E+Torch7%3A+A+matlab%2Dlike+environment+for+machine+learning%2E+In+BigLearn%2C+NIPS+Workshop%2C+2011%2E)
12. Courbariaux, Matthieu, Bengio, Yoshua, and David, Jean-Pierre. Training deep neural networks with low precision multiplications. ArXiv e-prints, abs/1412.7024, December 2014. — [arXiv:1412.7024](https://arxiv.org/abs/1412.7024) [find](https://scholar.google.com/scholar?q=Courbariaux%2C+Matthieu%2C+Bengio%2C+Yoshua%2C+and+David%2C+Jean%2DPierre%2E+Training+deep+neural+networks+with+low+precision+multiplications%2E+ArXiv+e%2Dprints%2C+abs%2F1412%2E7024%2C+December+2014%2E+%E2%80%94+%5BarXiv%3A1412%2E7024%5D%28https)
13. Courbariaux, Matthieu, Bengio, Yoshua, and David, Jean-Pierre. [Binaryconnect: Training deep neural networks with binary weights during propagations](https://md.research.elamur.ai/free/en/papers/courbariaux-2015-binaryconnect-training-deep-neural-networks/). ArXiv e-prints, abs/1511.00363, November 2015.
14. Devlin, Jacob, Zbib, Rabih, Huang, Zhongqiang, Lamar, Thomas, Schwartz, Richard, and Makhoul, John. [Fast and robust neural network joint models for statistical machine translation](https://md.research.elamur.ai/free/en/papers/devlin-2014-fast-robust-neural-network-joint/). In Proc. ACL’2014, 2014.
15. Dieleman, Sander, Schlter, Jan, Raffel, Colin, Olson, Eben, Snderby, Sren Kaae, Nouri, Daniel, Maturana, Daniel, Thoma, Martin, Battenberg, Eric, Kelly, Jack, Fauw, Jeffrey De, Heilman, Michael, diogo149, McFee, Brian, Weideman, Hendrik, takacsg84, peterderivaz, Jon, instagibbs, Rasul, Dr. Kashif, C… [find](https://scholar.google.com/scholar?q=Dieleman%2C+Sander%2C+Schlter%2C+Jan%2C+Raffel%2C+Colin%2C+Olson%2C+Eben%2C+Snderby%2C+Sren+Kaae%2C+Nouri%2C+Daniel%2C+Maturana%2C+Daniel%2C+Thoma%2C+Martin%2C+Battenberg%2C+Eric%2C+Kelly%2C+Jack%2C+Fauw%2C+Jeffrey+De%2C+Heilman%2C+Michael%2C+diogo)
16. Esser, Steve K, Appuswamy, Rathinakumar, Merolla, Paul, Arthur, John V, and Modha, Dharmendra S. Backpropagation for energy-efficient neuromorphic computing. In Advances in Neural Information Processing Systems, pp. 1117–1125, 2015. [find](https://scholar.google.com/scholar?q=Esser%2C+Steve+K%2C+Appuswamy%2C+Rathinakumar%2C+Merolla%2C+Paul%2C+Arthur%2C+John+V%2C+and+Modha%2C+Dharmendra+S%2E+Backpropagation+for+energy%2Defficient+neuromorphic+computing%2E+In+Advances+in+Neural+Information+Processi)
17. Farabet, Clément, LeCun, Yann, Kavukcuoglu, Koray, Culurciello, Eugenio, Martini, Berin, Akselrod, Polina, and Talay, Selcuk. Large-scale FPGA-based convolutional networks. Machine Learning on Very Large Data Sets, 1, 2011a. [find](https://scholar.google.com/scholar?q=Farabet%2C+Cle%CC%81ment%2C+LeCun%2C+Yann%2C+Kavukcuoglu%2C+Koray%2C+Culurciello%2C+Eugenio%2C+Martini%2C+Berin%2C+Akselrod%2C+Polina%2C+and+Talay%2C+Selcuk%2E+Large%2Dscale+FPGA%2Dbased+convolutional+networks%2E+Machine+Learning+on+Very+L)
18. Farabet, Clément, Martini, Berin, Corda, Benoit, Akselrod, Polina, Culurciello, Eugenio, and LeCun, Yann. Neuflow: A runtime reconfigurable dataflow processor for vision. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on, pp. 109–116. [find](https://scholar.google.com/scholar?q=Farabet%2C+Cle%CC%81ment%2C+Martini%2C+Berin%2C+Corda%2C+Benoit%2C+Akselrod%2C+Polina%2C+Culurciello%2C+Eugenio%2C+and+LeCun%2C+Yann%2E+Neuflow%3A+A+runtime+reconfigurable+dataflow+processor+for+vision%2E+In+Computer+Vision+and+Patte)
19. Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In AIS-TATS’2010, 2010. [find](https://scholar.google.com/scholar?q=Glorot%2C+Xavier+and+Bengio%2C+Yoshua%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+AIS%2DTATS%E2%80%992010%2C+2010%2E)
20. Gong, Yunchao, Liu, Liu, Yang, Ming, and Bourdev, Lubomir. Compressing Deep Convolutional Networks using Vector Quantization. pp. 1–10. Gong, Yunchao, Liu, Liu, Yang, Ming, and Bourdev, Lubomir. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) [find](https://scholar.google.com/scholar?q=Gong%2C+Yunchao%2C+Liu%2C+Liu%2C+Yang%2C+Ming%2C+and+Bourdev%2C+Lubomir%2E+Compressing+Deep+Convolutional+Networks+using+Vector+Quantization%2E+pp%2E+1%E2%80%9310%2E+Gong%2C+Yunchao%2C+Liu%2C+Liu%2C+Yang%2C+Ming%2C+and+Bourdev%2C+Lubomir%2E+Compr)
21. Goodfellow, Ian J., Warde-Farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout Networks. arXiv preprint, pp. 1319–1327. Goodfellow, Ian J., Warde-Farley, David, Lamblin, Pascal, Dumoulin, Vincent, Mirza, Mehdi, Pascanu, Razvan, Bergstra, James, Bastien, Frédéric, and Bengio, Y… — [arXiv:1308.4214](https://arxiv.org/abs/1308.4214) [find](https://scholar.google.com/scholar?q=Goodfellow%2C+Ian+J%2E%2C+Warde%2DFarley%2C+David%2C+Mirza%2C+Mehdi%2C+Courville%2C+Aaron%2C+and+Bengio%2C+Yoshua%2E+Maxout+Networks%2E+arXiv+preprint%2C+pp%2E+1319%E2%80%931327%2E+Goodfellow%2C+Ian+J%2E%2C+Warde%2DFarley%2C+David%2C+Lamblin%2C+Pascal%2C+D)
22. Govindu, Gokul, Zhuo, Ling, Choi, Seonil, and Prasanna, Viktor. Analysis of high-performance floating-point arithmetic on FPGAs. In Parallel and Distributed Processing Symposium, 2004. Proceedings. 18th International, pp. 149. [find](https://scholar.google.com/scholar?q=Govindu%2C+Gokul%2C+Zhuo%2C+Ling%2C+Choi%2C+Seonil%2C+and+Prasanna%2C+Viktor%2E+Analysis+of+high%2Dperformance+floating%2Dpoint+arithmetic+on+FPGAs%2E+In+Parallel+and+Distributed+Processing+Symposium%2C+2004%2E+Proceedings%2E+18)
23. Graham, Benjamin. Spatially-sparse convolutional neural networks. arXiv preprint arXiv:1409.6070, 2014. — [arXiv:1409.6070](https://arxiv.org/abs/1409.6070) [find](https://scholar.google.com/scholar?q=Graham%2C+Benjamin%2E+Spatially%2Dsparse+convolutional+neural+networks%2E+arXiv+preprint+arXiv%3A1409%2E6070%2C+2014%2E+%E2%80%94+%5BarXiv%3A1409%2E6070%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1409%2E6070%29)
24. Graves, Alex. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pp. 2348–2356, 2011. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2E+Practical+variational+inference+for+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+2348%E2%80%932356%2C+2011%2E)
25. Han, Song, Pool, Jeff, Tran, John, and Dally, William. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pp. 1135–1143, 2015. [find](https://scholar.google.com/scholar?q=Han%2C+Song%2C+Pool%2C+Jeff%2C+Tran%2C+John%2C+and+Dally%2C+William%2E+Learning+both+weights+and+connections+for+efficient+neural+network%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+1135%E2%80%931143%2C+2015%2E)
26. Hinton, Geoffrey. Neural networks for machine learning. Coursera, video lectures, 2012. [find](https://scholar.google.com/scholar?q=Hinton%2C+Geoffrey%2E+Neural+networks+for+machine+learning%2E+Coursera%2C+video+lectures%2C+2012%2E)
27. Hinton, Geoffrey, Deng, Li, Dahl, George E., Mohamed, Abdelrahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara, and Kingsbury, Brian. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, Nov. 2012. [find](https://scholar.google.com/scholar?q=Hinton%2C+Geoffrey%2C+Deng%2C+Li%2C+Dahl%2C+George+E%2E%2C+Mohamed%2C+Abdelrahman%2C+Jaitly%2C+Navdeep%2C+Senior%2C+Andrew%2C+Vanhoucke%2C+Vincent%2C+Nguyen%2C+Patrick%2C+Sainath%2C+Tara%2C+and+Kingsbury%2C+Brian%2E+Deep+neural+networks+for+a)
28. Horowitz, Mark. Computing’s Energy Problem (and what we can do about it). IEEE Interational Solid State Circuits Conference, pp. 10–14, 2014. [find](https://scholar.google.com/scholar?q=Horowitz%2C+Mark%2E+Computing%E2%80%99s+Energy+Problem+%28and+what+we+can+do+about+it%29%2E+IEEE+Interational+Solid+State+Circuits+Conference%2C+pp%2E+10%E2%80%9314%2C+2014%2E)
29. Hwang, Kyuyeon and Sung, Wonyong. Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In Signal Processing Systems (SiPS), 2014 IEEE Workshop on, pp. 1–6. [find](https://scholar.google.com/scholar?q=Hwang%2C+Kyuyeon+and+Sung%2C+Wonyong%2E+Fixed%2Dpoint+feedforward+deep+neural+network+design+using+weights%2B+1%2C+0%2C+and%2D+1%2E+In+Signal+Processing+Systems+%28SiPS%29%2C+2014+IEEE+Workshop+on%2C+pp%2E+1%E2%80%936%2E)
30. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015. [find](https://scholar.google.com/scholar?q=Ioffe%2C+Sergey+and+Szegedy%2C+Christian%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+2015%2E)
31. Judd, Patrick, Albericio, Jorge, Hetherington, Tayler, Aamodt, Tor, Jerger, Natalie Enright, Urtasun, Raquel, and Moshovos, Andreas. Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets. pp. 12. Kim, Jonghong, Hwang, Kyuyeon, and Sung, Wonyong. X1000 real-time phoneme recognition vlsi… [find](https://scholar.google.com/scholar?q=Judd%2C+Patrick%2C+Albericio%2C+Jorge%2C+Hetherington%2C+Tayler%2C+Aamodt%2C+Tor%2C+Jerger%2C+Natalie+Enright%2C+Urtasun%2C+Raquel%2C+and+Moshovos%2C+Andreas%2E+Reduced%2DPrecision+Strategies+for+Bounded+Memory+in+Deep+Neural+Nets)
32. Kim, M. and Smaragdis, P. Bitwise Neural Networks. ArXiv eprints, January 2016. [find](https://scholar.google.com/scholar?q=Kim%2C+M%2E+and+Smaragdis%2C+P%2E+Bitwise+Neural+Networks%2E+ArXiv+eprints%2C+January+2016%2E)
33. Kim, Minje and Paris, Smaragdis. Bitwise Neural Networks. ICML Workshop on Resource-Efficient Machine Learning, 37, 2015. [find](https://scholar.google.com/scholar?q=Kim%2C+Minje+and+Paris%2C+Smaragdis%2E+Bitwise+Neural+Networks%2E+ICML+Workshop+on+Resource%2DEfficient+Machine+Learning%2C+37%2C+2015%2E)
34. Kingma, Diederik and Ba, Jimmy. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014.
35. Krizhevsky, A., Sutskever, I., and Hinton, G. [ImageNet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In NIPS’2012. 2012.
36. LeCun, Yann, Bottou, Leon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998. [find](https://scholar.google.com/scholar?q=LeCun%2C+Yann%2C+Bottou%2C+Leon%2C+Bengio%2C+Yoshua%2C+and+Haffner%2C+Patrick%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+November+1998%2E)
37. Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick, Zhang, Zhengyou, and Tu, Zhuowen. Deeply-supervised nets. arXiv preprint arXiv:1409.5185, 2014. [find](https://scholar.google.com/scholar?q=Lee%2C+Chen%2DYu%2C+Xie%2C+Saining%2C+Gallagher%2C+Patrick%2C+Zhang%2C+Zhengyou%2C+and+Tu%2C+Zhuowen%2E+Deeply%2Dsupervised+nets%2E+arXiv+preprint+arXiv%3A1409%2E5185%2C+2014%2E)
38. Lee, Chen-Yu, Gallagher, Patrick W, and Tu, Zhuowen. Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. arXiv preprint arXiv:1509.08985, 2015. — [arXiv:1509.08985](https://arxiv.org/abs/1509.08985) [find](https://scholar.google.com/scholar?q=Lee%2C+Chen%2DYu%2C+Gallagher%2C+Patrick+W%2C+and+Tu%2C+Zhuowen%2E+Generalizing+pooling+functions+in+convolutional+neural+networks%3A+Mixed%2C+gated%2C+and+tree%2E+arXiv+preprint+arXiv%3A1509%2E08985%2C+2015%2E+%E2%80%94+%5BarXiv%3A1509%2E08985)
39. Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network In Network. arXiv preprint, pp. 10. Lin, Zhouhan, Courbariaux, Matthieu, Memisevic, Roland, and Bengio, Yoshua. Neural networks with few multiplications. ArXiv e-prints, abs/1510.03009, October 2015. — [arXiv:1510.03009](https://arxiv.org/abs/1510.03009) [find](https://scholar.google.com/scholar?q=Lin%2C+Min%2C+Chen%2C+Qiang%2C+and+Yan%2C+Shuicheng%2E+Network+In+Network%2E+arXiv+preprint%2C+pp%2E+10%2E+Lin%2C+Zhouhan%2C+Courbariaux%2C+Matthieu%2C+Memisevic%2C+Roland%2C+and+Bengio%2C+Yoshua%2E+Neural+networks+with+few+multiplicati)
40. Mnih, Volodymyr, Kavukcuoglo, Koray, Silver, David, Rusu, Andrei A., Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidgeland, Andreas K., Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharsan, Wierstra, Daan, Legg, S… [find](https://scholar.google.com/scholar?q=Mnih%2C+Volodymyr%2C+Kavukcuoglo%2C+Koray%2C+Silver%2C+David%2C+Rusu%2C+Andrei+A%2E%2C+Veness%2C+Joel%2C+Bellemare%2C+Marc+G%2E%2C+Graves%2C+Alex%2C+Riedmiller%2C+Martin%2C+Fidgeland%2C+Andreas+K%2E%2C+Ostrovski%2C+Georg%2C+Petersen%2C+Stig%2C+Beatti)
41. Mordvintsev, Alexander, Olah, Christopher, and Tyka, Mike. In-ceptionism: Going deeper into neural networks, 2015. Accessed: 2015-06-30. [find](https://scholar.google.com/scholar?q=Mordvintsev%2C+Alexander%2C+Olah%2C+Christopher%2C+and+Tyka%2C+Mike%2E+In%2Dceptionism%3A+Going+deeper+into+neural+networks%2C+2015%2E+Accessed%3A+2015%2D06%2D30%2E)
42. Pham, Phi-Hung, Jelaca, Darko, Farabet, Clement, Martini, Berin, LeCun, Yann, and Culurciello, Eugenio. Neuflow: dataflow vision processing system-on-a-chip. In Circuits and Systems (MWSCAS), 2012 IEEE 55th International Midwest Symposium on, pp. 1044–1047. [find](https://scholar.google.com/scholar?q=Pham%2C+Phi%2DHung%2C+Jelaca%2C+Darko%2C+Farabet%2C+Clement%2C+Martini%2C+Berin%2C+LeCun%2C+Yann%2C+and+Culurciello%2C+Eugenio%2E+Neuflow%3A+dataflow+vision+processing+system%2Don%2Da%2Dchip%2E+In+Circuits+and+Systems+%28MWSCAS%29%2C+2012+IEE)
43. Romero, Adriana, Ballas, Nicolas, Kahou, Samira Ebrahimi, Chassang, Antoine, Gatta, Carlo, and Bengio, Yoshua. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. [find](https://scholar.google.com/scholar?q=Romero%2C+Adriana%2C+Ballas%2C+Nicolas%2C+Kahou%2C+Samira+Ebrahimi%2C+Chassang%2C+Antoine%2C+Gatta%2C+Carlo%2C+and+Bengio%2C+Yoshua%2E+Fitnets%3A+Hints+for+thin+deep+nets%2E+arXiv+preprint+arXiv%3A1412%2E6550%2C+2014%2E)
44. Sainath, Tara, rahman Mohamed, Abdel, Kingsbury, Brian, and Ramabhadran, Bhuvana. Deep convolutional neural networks for LVCSR. In ICASSP 2013, 2013. [find](https://scholar.google.com/scholar?q=Sainath%2C+Tara%2C+rahman+Mohamed%2C+Abdel%2C+Kingsbury%2C+Brian%2C+and+Ramabhadran%2C+Bhuvana%2E+Deep+convolutional+neural+networks+for+LVCSR%2E+In+ICASSP+2013%2C+2013%2E)
45. Silver, David, Huang, Aja, Maddison, Chris J., Guez, Arthur, Sifre, Laurent, van den Driessche, George, Schrittwieser, Julian, Antonoglou, Ioannis, Panneershelvam, Veda, Lanctot, Marc, Dieleman, Sander, Grewe, Dominik, Nham, John, Kalchbrenner, Nal, Sutskever, Ilya, Lillicrap, Timothy, Leach, Madele… [find](https://scholar.google.com/scholar?q=Silver%2C+David%2C+Huang%2C+Aja%2C+Maddison%2C+Chris+J%2E%2C+Guez%2C+Arthur%2C+Sifre%2C+Laurent%2C+van+den+Driessche%2C+George%2C+Schrittwieser%2C+Julian%2C+Antonoglou%2C+Ioannis%2C+Panneershelvam%2C+Veda%2C+Lanctot%2C+Marc%2C+Dieleman%2C+Sande)
46. Simonyan, Karen and Zisserman, Andrew. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). In ICLR, 2015.
47. Soudry, Daniel, Di Castro, Dotan, Gal, Asaf, Kolodny, Avinoam, and Kvatinsky, Shahar. Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training. IEEE Transactions on Neural Networks and Learning Systems, (10): 2408–2421. Soudry, Daniel, Hubara, Itay, and Meir, Ron. Expectation… [find](https://scholar.google.com/scholar?q=Soudry%2C+Daniel%2C+Di+Castro%2C+Dotan%2C+Gal%2C+Asaf%2C+Kolodny%2C+Avinoam%2C+and+Kvatinsky%2C+Shahar%2E+Memristor%2DBased+Multilayer+Neural+Networks+With+Online+Gradient+Descent+Training%2E+IEEE+Transactions+on+Neural+Netw)
48. Srivastava, Nitish. Improving neural networks with dropout. Master’s thesis, U. Toronto, 2013. [find](https://scholar.google.com/scholar?q=Srivastava%2C+Nitish%2E+Improving+neural+networks+with+dropout%2E+Master%E2%80%99s+thesis%2C+U%2E+Toronto%2C+2013%2E)
49. Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. [Dropout: A simple way to prevent neural networks from overfitting](https://md.research.elamur.ai/free/en/papers/srivastava-2014-dropout-simple-way-prevent/). Journal of Machine Learning Research, 15:1929–1958, 2014.
50. Sung, Wonyong, Shin, Sungho, and Hwang, Kyuyeon. Resiliency of Deep Neural Networks under Quantization. (2014):1–9. [find](https://scholar.google.com/scholar?q=Sung%2C+Wonyong%2C+Shin%2C+Sungho%2C+and+Hwang%2C+Kyuyeon%2E+Resiliency+of+Deep+Neural+Networks+under+Quantization%2E+%282014%29%3A1%E2%80%939%2E)
51. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In NIPS’2014, 2014.
52. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. Technical report, arXiv:1409.4842, 2014. [find](https://scholar.google.com/scholar?q=Szegedy%2C+Christian%2C+Liu%2C+Wei%2C+Jia%2C+Yangqing%2C+Sermanet%2C+Pierre%2C+Reed%2C+Scott%2C+Anguelov%2C+Dragomir%2C+Erhan%2C+Dumitru%2C+Vanhoucke%2C+Vincent%2C+and+Rabinovich%2C+Andrew%2E+Going+deeper+with+convolutions%2E+Technical+re)
53. Tang, Yichuan. Deep learning using linear support vector machines. Workshop on Challenges in Representation Learning, ICML, 2013. [find](https://scholar.google.com/scholar?q=Tang%2C+Yichuan%2E+Deep+learning+using+linear+support+vector+machines%2E+Workshop+on+Challenges+in+Representation+Learning%2C+ICML%2C+2013%2E)
54. Vanhoucke, Vincent, Senior, Andrew, and Mao, Mark Z. Improving the speed of neural networks on CPUs. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, 2011. [find](https://scholar.google.com/scholar?q=Vanhoucke%2C+Vincent%2C+Senior%2C+Andrew%2C+and+Mao%2C+Mark+Z%2E+Improving+the+speed+of+neural+networks+on+CPUs%2E+In+Proc%2E+Deep+Learning+and+Unsupervised+Feature+Learning+NIPS+Workshop%2C+2011%2E)
55. Wan, Li, Zeiler, Matthew, Zhang, Sixin, LeCun, Yann, and Fergus, Rob. Regularization of neural networks using dropconnect. In ICML’2013, 2013. [find](https://scholar.google.com/scholar?q=Wan%2C+Li%2C+Zeiler%2C+Matthew%2C+Zhang%2C+Sixin%2C+LeCun%2C+Yann%2C+and+Fergus%2C+Rob%2E+Regularization+of+neural+networks+using+dropconnect%2E+In+ICML%E2%80%992013%2C+2013%2E)

*Updated 2 November 2026*