---
title: "Quantized Neural Networks — paper by Hubara et al. (2016)"
description: "The authors train Quantized Neural Networks (QNNs), where weights and activations use very few… 72 references, cited by 123 papers here."
authors: ["Itay Hubara", "Matthieu Courbariaux", "Daniel Soudry", "Ran El-Yaniv", "Yoshua Bengio"]
year: 2016
venue: "arXiv:1609.07061"
source: "https://arxiv.org/abs/1609.07061"
doi: "10.48550/arxiv.1609.07061"
arxiv: "1609.07061"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hubara-2016-quantized-neural-networks-training-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hubara-2016-quantized-neural-networks-training-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

# Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

**Authors:** [Itay Hubara](https://md.research.elamur.ai/free/en/authors/itay-hubara/), [Matthieu Courbariaux](https://md.research.elamur.ai/free/en/authors/matthieu-courbariaux/), [Daniel Soudry](https://md.research.elamur.ai/free/en/authors/daniel-soudry/), [Ran El-Yaniv](https://md.research.elamur.ai/free/en/authors/ran-el-yaniv/) and [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/) · **Year:** 2016 · **Venue:** arXiv:1609.07061 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1609.07061) · [PDF](https://arxiv.org/pdf/1609.07061) · [DOI 10.48550/arxiv.1609.07061](https://doi.org/10.48550/arxiv.1609.07061) · [arXiv 1609.07061](https://arxiv.org/abs/1609.07061) · [OpenAlex](https://openalex.org/W2524428287)

**Citations:** 1,487 citations (OpenAlex, 2026-10-14) · [123](https://md.research.elamur.ai/free/en/papers/hubara-2016-quantized-neural-networks-training-neural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles the high cost of running deep networks, whose many multiply-accumulate operations make them slow and power-hungry on ordinary hardware and low-power devices. The authors train Quantized Neural Networks (QNNs), where weights and activations use very few bits. In the extreme case of 1 bit each, the result is a Binarized Neural Network (BNN). The quantized values are used both at inference and when computing gradients during training. Full-precision copies of the weights are kept only to accumulate updates. Gradients pass through the non-differentiable sign function via a straight-through estimator. The method is evaluated on MNIST, CIFAR-10, SVHN and ImageNet (AlexNet, GoogleNet), and on Penn Treebank language modeling with vanilla RNNs and LSTMs. Accuracy stays close to 32-bit baselines. The authors also write a binary matrix multiplication GPU kernel that runs an MNIST network substantially faster. This matters because most arithmetic becomes cheap bit-wise logic, which cuts memory traffic and should sharply reduce energy use.

## Key points
- A quantized AlexNet with 1-bit weights and 2-bit activations reaches about 51% top-1 accuracy on ImageNet.
- Parameter gradients can be quantized to 6 bits with only minor accuracy loss, in preliminary experiments.
- On Penn Treebank, recurrent QNNs (RNN and LSTM) with 4-bit weights and activations perform about as well as float32 models.
- A custom binary GPU kernel runs the MNIST BNN 7× faster than an unoptimized kernel, with no accuracy drop. The authors also argue that dedicated hardware exploiting repeated binary convolution kernels could cut time complexity by about 60%.
- Experiments were run in both Theano and Torch7, and the code is released as BinaryNet on GitHub (MatthieuCourbariaux and itayhubara repositories).

## Interesting ideas
- **Two copies of the weights.** Binary weights are used in the forward and backward passes, while gradient updates accumulate in hidden real-valued weights. This separation could help train any model whose deployed parameters must be discrete, such as compact models for edge devices.
- **Saturating straight-through estimator.** The gradient passes through the sign function unchanged, but only when the input lies in [-1, 1]; outside that range it is set to zero. Real-valued weights are also clipped to the same range so they cannot drift without bound. This simple rule is reusable for other hard thresholds or discrete choices inside networks.
- **Quantization as noise regularization.** The authors present binarization as a Dropout-like noise source during training. Lowering precision may therefore help generalization rather than only costing accuracy.
- **Replacing multiplications with shifts.** Batch normalization, AdaMax and learning-rate decay are approximated with bit shifts, for example halving the learning rate via a 1-bit right shift every N epochs. This trick fits integer-only or very low-power training pipelines.
- **Keep the output layer full-precision.** The final classifier layer is not binarized. Leaving boundary layers in higher precision is a cheap design choice that protects accuracy in compressed models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 254 works · 56,747 citations
- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 87 works · 91,597 citations
- [Technion – Israel Institute of Technology](https://md.research.elamur.ai/free/en/universities/technion-israel-institute-of-technology/) — University, Haifa, Israel · 54 works · 7,237 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 188 works · [⚙️ Electrical and Electronic Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/electrical-and-electronic-engineering/) · 33 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv and Yoshua Bengio. 2016. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. arXiv:1609.07061.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hubara2016quantized,
  title = {{Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations}},
  author = {Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv and Yoshua Bengio},
  year = {2016},
  doi = {10.48550/arxiv.1609.07061},
  eprint = {1609.07061},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1609.07061}
}
```

- [← previous A Decomposable Attention Model for Natural Language Inference](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/)
- [next → End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://md.research.elamur.ai/free/en/papers/miwa-2016-end-end-relation-extraction-lstms/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

72 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Renzo Andri, Lukas Cavigelli, Davide Rossi, and Luca Benini. Yodann: An ultra-low power convolutional neural network accelerator based on binary weights. arXiv preprint arXiv:1606.05487, 2016. [find](https://scholar.google.com/scholar?q=Renzo+Andri%2C+Lukas+Cavigelli%2C+Davide+Rossi%2C+and+Luca+Benini%2E+Yodann%3A+An+ultra%2Dlow+power+convolutional+neural+network+accelerator+based+on+binary+weights%2E+arXiv+preprint+arXiv%3A1606%2E05487%2C+2016%2E)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In ICLR’2015, arXiv:1409.0473, 2015.
3. Carlo Baldassi, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina. Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses. Physical Review Letters, 115(12): 1–5, 2015. ISSN 10797114. doi: 10.1103/Phy… — [doi:10.1103/physrevlett.115.128101](https://doi.org/10.1103/physrevlett.115.128101) [find](https://scholar.google.com/scholar?q=Carlo+Baldassi%2C+Alessandro+Ingrosso%2C+Carlo+Lucibello%2C+Luca+Saglietti%2C+and+Riccardo+Zecchina%2E+Subdominant+Dense+Clusters+Allow+for+Simple+Learning+and+High+Computational+Performance+in+Neural+Networks+)
4. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. [Theano: new features and speed improvements](https://md.research.elamur.ai/free/en/papers/bastien-2012-theano-new-features-speed-improvements/). Deep Learning and Unsupervised Feature Learning NIPS 2012
5. Michael J Beauchamp, Scott Hauck, Keith D Underwood, and K Scott Hemmert. Embedded floating-point units in FPGAs. In Proceedings of the 2006 ACM/SIGDA 14th international symposium on Field programmable gate arrays, pages 12–20. [find](https://scholar.google.com/scholar?q=Michael+J+Beauchamp%2C+Scott+Hauck%2C+Keith+D+Underwood%2C+and+K+Scott+Hemmert%2E+Embedded+floating%2Dpoint+units+in+FPGAs%2E+In+Proceedings+of+the+2006+ACM%2FSIGDA+14th+international+symposium+on+Field+programmabl)
6. Yoshua Bengio. Estimating or propagating gradients through stochastic neurons. Technical Report arXiv:1305.2982, Universite de Montreal, 2013. — [arXiv:1305.2982](https://arxiv.org/abs/1305.2982) [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2E+Estimating+or+propagating+gradients+through+stochastic+neurons%2E+Technical+Report+arXiv%3A1305%2E2982%2C+Universite+de+Montreal%2C+2013%2E+%E2%80%94+%5BarXiv%3A1305%2E2982%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1305%2E2982%29)
7. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral… [find](https://scholar.google.com/scholar?q=James+Bergstra%2C+Olivier+Breuleux%2C+Fre%CC%81de%CC%81ric+Bastien%2C+Pascal+Lamblin%2C+Razvan+Pascanu%2C+Guillaume+Desjardins%2C+Joseph+Turian%2C+David+Warde%2DFarley%2C+and+Yoshua+Bengio%2E+Theano%3A+a+CPU+and+GPU+math+expression+)
8. Tianshi Chen, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji Chen, and Olivier Temam. Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning. In Proceedings of the 19th international conference on Architectural support for programming languages and operating sy… [find](https://scholar.google.com/scholar?q=Tianshi+Chen%2C+Zidong+Du%2C+Ninghui+Sun%2C+Jia+Wang%2C+Chengyong+Wu%2C+Yunji+Chen%2C+and+Olivier+Temam%2E+Diannao%3A+A+small%2Dfootprint+high%2Dthroughput+accelerator+for+ubiquitous+machine%2Dlearning%2E+In+Proceedings+of+t)
9. Wenlin Chen, James T Wilson, Stephen Tyree, Kilian Q Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. arXiv preprint arXiv:1504.04788, 2015. — [arXiv:1504.04788](https://arxiv.org/abs/1504.04788) [find](https://scholar.google.com/scholar?q=Wenlin+Chen%2C+James+T+Wilson%2C+Stephen+Tyree%2C+Kilian+Q+Weinberger%2C+and+Yixin+Chen%2E+Compressing+neural+networks+with+the+hashing+trick%2E+arXiv+preprint+arXiv%3A1504%2E04788%2C+2015%2E+%E2%80%94+%5BarXiv%3A1504%2E04788%5D%28https%3A%2F)
10. Yunji Chen, Tao Luo, Shaoli Liu, Shijin Zhang, Liqiang He, Jia Wang, Ling Li, Tianshi Chen, Zhiwei Xu, Ninghui Sun, et al. Dadiannao: A machine-learning supercomputer. In Microarchitecture (MICRO), 2014 47th Annual IEEE/ACM International Symposium on, pages 609–622. [find](https://scholar.google.com/scholar?q=Yunji+Chen%2C+Tao+Luo%2C+Shaoli+Liu%2C+Shijin+Zhang%2C+Liqiang+He%2C+Jia+Wang%2C+Ling+Li%2C+Tianshi+Chen%2C+Zhiwei+Xu%2C+Ninghui+Sun%2C+et+al%2E+Dadiannao%3A+A+machine%2Dlearning+supercomputer%2E+In+Microarchitecture+%28MICRO%29%2C+20)
11. Zhiyong Cheng, Daniel Soudry, Zexi Mao, and Zhenzhong Lan. Training binary multilayer neural networks for image classification using expectation backpropgation. arXiv preprint arXiv:1503.03562, 2015. — [arXiv:1503.03562](https://arxiv.org/abs/1503.03562) [find](https://scholar.google.com/scholar?q=Zhiyong+Cheng%2C+Daniel+Soudry%2C+Zexi+Mao%2C+and+Zhenzhong+Lan%2E+Training+binary+multilayer+neural+networks+for+image+classification+using+expectation+backpropgation%2E+arXiv+preprint+arXiv%3A1503%2E03562%2C+2015%2E+)
12. Adam Coates, Brody Huval, Tao Wang, David Wu, Bryan Catanzaro, and Ng Andrew. Deep learning with COTS HPC systems. In Proceedings of the 30th international conference on machine learning, pages 1337–1345, 2013. [find](https://scholar.google.com/scholar?q=Adam+Coates%2C+Brody+Huval%2C+Tao+Wang%2C+David+Wu%2C+Bryan+Catanzaro%2C+and+Ng+Andrew%2E+Deep+learning+with+COTS+HPC+systems%2E+In+Proceedings+of+the+30th+international+conference+on+machine+learning%2C+pages+1337%E2%80%931)
13. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, 2011. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Koray+Kavukcuoglu%2C+and+Cle%CC%81ment+Farabet%2E+Torch7%3A+A+matlab%2Dlike+environment+for+machine+learning%2E+In+BigLearn%2C+NIPS+Workshop%2C+2011%2E)
14. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. ArXiv e-prints, abs/1412.7024, December 2014. — [arXiv:1412.7024](https://arxiv.org/abs/1412.7024) [find](https://scholar.google.com/scholar?q=Matthieu+Courbariaux%2C+Yoshua+Bengio%2C+and+Jean%2DPierre+David%2E+Training+deep+neural+networks+with+low+precision+multiplications%2E+ArXiv+e%2Dprints%2C+abs%2F1412%2E7024%2C+December+2014%2E+%E2%80%94+%5BarXiv%3A1412%2E7024%5D%28https%3A%2F%2F)
15. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. [Binaryconnect: Training deep neural networks with binary weights during propagations](https://md.research.elamur.ai/free/en/papers/courbariaux-2015-binaryconnect-training-deep-neural-networks/). ArXiv e-prints, abs/1511.00363, November 2015a.
16. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://md.research.elamur.ai/free/en/papers/courbariaux-2015-binaryconnect-training-deep-neural-networks/). Nips, pages 1–9, 2015b. URL http://arxiv.org/abs/1511.00363. [↗](http://arxiv.org/abs/1511.00363)
17. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. [Fast and robust neural network joint models for statistical machine translation](https://md.research.elamur.ai/free/en/papers/devlin-2014-fast-robust-neural-network-joint/). In Proc. ACL’2014, 2014.
18. Sander Dieleman, Jan Schlter, Colin Raffel, Eben Olson, Sren Kaae Snderby, Daniel Nouri, Daniel Maturana, Martin Thoma, Eric Battenberg, Jack Kelly, Jeffrey De Fauw, Michael Heilman, diogo149, Brian McFee, Hendrik Weideman, takacsg84, peterderivaz, Jon, in-stagibbs, Dr. Kashif Rasul, CongLiu, Britef… — [doi:10.5281/zenodo.27878](https://doi.org/10.5281/zenodo.27878) [find](https://scholar.google.com/scholar?q=Sander+Dieleman%2C+Jan+Schlter%2C+Colin+Raffel%2C+Eben+Olson%2C+Sren+Kaae+Snderby%2C+Daniel+Nouri%2C+Daniel+Maturana%2C+Martin+Thoma%2C+Eric+Battenberg%2C+Jack+Kelly%2C+Jeffrey+De+Fauw%2C+Michael+Heilman%2C+diogo149%2C+Brian+M)
19. Steve K Esser, Rathinakumar Appuswamy, Paul Merolla, John V Arthur, and Dharmendra S Modha. Backpropagation for energy-efficient neuromorphic computing. In Advances in Neural Information Processing Systems, pages 1117–1125, 2015. [find](https://scholar.google.com/scholar?q=Steve+K+Esser%2C+Rathinakumar+Appuswamy%2C+Paul+Merolla%2C+John+V+Arthur%2C+and+Dharmendra+S+Modha%2E+Backpropagation+for+energy%2Defficient+neuromorphic+computing%2E+In+Advances+in+Neural+Information+Processing+Sy)
20. Clément Farabet, Yann LeCun, Koray Kavukcuoglu, Eugenio Culurciello, Berin Martini, Polina Akselrod, and Selcuk Talay. Large-scale FPGA-based convolutional networks. Machine Learning on Very Large Data Sets, 1, 2011a. [find](https://scholar.google.com/scholar?q=Cle%CC%81ment+Farabet%2C+Yann+LeCun%2C+Koray+Kavukcuoglu%2C+Eugenio+Culurciello%2C+Berin+Martini%2C+Polina+Akselrod%2C+and+Selcuk+Talay%2E+Large%2Dscale+FPGA%2Dbased+convolutional+networks%2E+Machine+Learning+on+Very+Large+Da)
21. Clément Farabet, Berin Martini, Benoit Corda, Polina Akselrod, Eugenio Culurciello, and Yann LeCun. Neuflow: A runtime reconfigurable dataflow processor for vision. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on, pages 109–116. [find](https://scholar.google.com/scholar?q=Cle%CC%81ment+Farabet%2C+Berin+Martini%2C+Benoit+Corda%2C+Polina+Akselrod%2C+Eugenio+Culurciello%2C+and+Yann+LeCun%2E+Neuflow%3A+A+runtime+reconfigurable+dataflow+processor+for+vision%2E+In+Computer+Vision+and+Pattern+Rec)
22. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS’2010, 2010. [find](https://scholar.google.com/scholar?q=Xavier+Glorot+and+Yoshua+Bengio%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+AISTATS%E2%80%992010%2C+2010%2E)
23. Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) [find](https://scholar.google.com/scholar?q=Yunchao+Gong%2C+Liu+Liu%2C+Ming+Yang%2C+and+Lubomir+Bourdev%2E+Compressing+deep+convolutional+networks+using+vector+quantization%2E+arXiv+preprint+arXiv%3A1412%2E6115%2C+2014%2E+%E2%80%94+%5BarXiv%3A1412%2E6115%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
24. Ian J. Goodfellow, David Warde-Farley, Pascal Lamblin, Vincent Dumoulin, Mehdi Mirza, Razvan Pascanu, James Bergstra, Frédéric Bastien, and Yoshua Bengio. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214, 2013a. — [arXiv:1308.4214](https://arxiv.org/abs/1308.4214) [find](https://scholar.google.com/scholar?q=Ian+J%2E+Goodfellow%2C+David+Warde%2DFarley%2C+Pascal+Lamblin%2C+Vincent+Dumoulin%2C+Mehdi+Mirza%2C+Razvan+Pascanu%2C+James+Bergstra%2C+Fre%CC%81de%CC%81ric+Bastien%2C+and+Yoshua+Bengio%2E+Pylearn2%3A+a+machine+learning+research+libra)
25. Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout Networks. arXiv preprint, pages 1319–1327, 2013b. URL http://arxiv.org/abs/1302.4389. — [arXiv:1302.4389](https://arxiv.org/abs/1302.4389) [↗](http://arxiv.org/abs/1302.4389)
26. Gokul Govindu, Ling Zhuo, Seonil Choi, and Viktor Prasanna. Analysis of high-performance floating-point arithmetic on FPGAs. In Parallel and Distributed Processing Symposium, 2004. Proceedings. 18th International, page 149. [find](https://scholar.google.com/scholar?q=Gokul+Govindu%2C+Ling+Zhuo%2C+Seonil+Choi%2C+and+Viktor+Prasanna%2E+Analysis+of+high%2Dperformance+floating%2Dpoint+arithmetic+on+FPGAs%2E+In+Parallel+and+Distributed+Processing+Symposium%2C+2004%2E+Proceedings%2E+18th+I)
27. Benjamin Graham. Spatially-sparse convolutional neural networks. arXiv:1409.6070, 2014. — [arXiv:1409.6070](https://arxiv.org/abs/1409.6070) [find](https://scholar.google.com/scholar?q=Benjamin+Graham%2E+Spatially%2Dsparse+convolutional+neural+networks%2E+arXiv%3A1409%2E6070%2C+2014%2E+%E2%80%94+%5BarXiv%3A1409%2E6070%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1409%2E6070%29)
28. Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pages 2348–2356, 2011. [find](https://scholar.google.com/scholar?q=Alex+Graves%2E+Practical+variational+inference+for+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+2348%E2%80%932356%2C+2011%2E)
29. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. CoRR, abs/1502.02551, 392, 2015. arXiv preprint Philipp Gysel, Mohammad Motamedi, and Soheil Ghiasi. Hardware-oriented approximation of convolutional neural networks. arXiv prep… [find](https://scholar.google.com/scholar?q=Suyog+Gupta%2C+Ankur+Agrawal%2C+Kailash+Gopalakrishnan%2C+and+Pritish+Narayanan%2E+Deep+learning+with+limited+numerical+precision%2E+CoRR%2C+abs%2F1502%2E02551%2C+392%2C+2015%2E+arXiv+preprint+Philipp+Gysel%2C+Mohammad+Motam)
30. Huizi Mao Han, Song and William J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv preprint, pages 1–11, 2015. URL http://arxiv.org/abs/1510.00149. [↗](http://arxiv.org/abs/1510.00149)
31. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015a. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J+Dally%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+huffman+coding%2E+arXiv+preprint+arXiv%3A1510%2E00149%2C+2015a%2E)
32. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pages 1135–1143, 2015b. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Jeff+Pool%2C+John+Tran%2C+and+William+Dally%2E+Learning+both+weights+and+connections+for+efficient+neural+network%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+1135%E2%80%931143%2C+2015b%2E)
33. Geoffrey Hinton. Neural networks for machine learning. Coursera, video lectures, 2012. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2E+Neural+networks+for+machine+learning%2E+Coursera%2C+video+lectures%2C+2012%2E)
34. Geoffrey Hinton, Li Deng, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29(6):82–97, Nov. 2012. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Li+Deng%2C+George+E%2E+Dahl%2C+Abdel%2Drahman+Mohamed%2C+Navdeep+Jaitly%2C+Andrew+Senior%2C+Vincent+Vanhoucke%2C+Patrick+Nguyen%2C+Tara+Sainath%2C+and+Brian+Kingsbury%2E+Deep+neural+networks+for+acoustic+m)
35. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
36. Mark Horowitz. Computing’s Energy Problem (and what we can do about it). IEEE Interational Solid State Circuits Conference, pages 10–14, 2014. ISSN 0018-9200. doi: 10.1109/JSSC.2014.2361354. — [doi:10.1109/jssc.2014.2361354](https://doi.org/10.1109/jssc.2014.2361354) [find](https://scholar.google.com/scholar?q=Mark+Horowitz%2E+Computing%E2%80%99s+Energy+Problem+%28and+what+we+can+do+about+it%29%2E+IEEE+Interational+Solid+State+Circuits+Conference%2C+pages+10%E2%80%9314%2C+2014%2E+ISSN+0018%2D9200%2E+doi%3A+10%2E1109%2FJSSC%2E2014%2E2361354%2E+%E2%80%94+%5Bdoi%3A10)
37. Kyuyeon Hwang and Wonyong Sung. Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In Signal Processing Systems (SiPS), 2014 IEEE Workshop on, pages 1–6. [find](https://scholar.google.com/scholar?q=Kyuyeon+Hwang+and+Wonyong+Sung%2E+Fixed%2Dpoint+feedforward+deep+neural+network+design+using+weights%2B+1%2C+0%2C+and%2D+1%2E+In+Signal+Processing+Systems+%28SiPS%29%2C+2014+IEEE+Workshop+on%2C+pages+1%E2%80%936%2E)
38. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015. [find](https://scholar.google.com/scholar?q=Sergey+Ioffe+and+Christian+Szegedy%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+2015%2E)
39. M. Kim and P. Smaragdis. Bitwise Neural Networks. ArXiv e-prints, January 2016. [find](https://scholar.google.com/scholar?q=M%2E+Kim+and+P%2E+Smaragdis%2E+Bitwise+Neural+Networks%2E+ArXiv+e%2Dprints%2C+January+2016%2E)
40. Diederik Kingma and Jimmy Ba. [Adam: A Method for Stochastic Optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv:1412.6980 [cs], pages 1–13, 2014a. URL http://arxiv.org/abs/1412.6980$\\ delimiter"026E30F$nhttp://www.arxiv.org/pdf/1412.6980.pdf. [↗](http://arxiv.org/abs/1412.6980$\\)
41. Diederik Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014b.
42. A. Krizhevsky, I. Sutskever, and G. Hinton. [ImageNet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In NIPS’2012. 2012.
43. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Leon+Bottou%2C+Yoshua+Bengio%2C+and+Patrick+Haffner%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proceedings+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+November+1998%2E)
44. Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. arXiv preprint arXiv:1409.5185, 2014. [find](https://scholar.google.com/scholar?q=Chen%2DYu+Lee%2C+Saining+Xie%2C+Patrick+Gallagher%2C+Zhengyou+Zhang%2C+and+Zhuowen+Tu%2E+Deeply%2Dsupervised+nets%2E+arXiv+preprint+arXiv%3A1409%2E5185%2C+2014%2E)
45. Chen-Yu Lee, Patrick W Gallagher, and Zhuowen Tu. Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. arXiv preprint arXiv:1509.08985, 2015. — [arXiv:1509.08985](https://arxiv.org/abs/1509.08985) [find](https://scholar.google.com/scholar?q=Chen%2DYu+Lee%2C+Patrick+W+Gallagher%2C+and+Zhuowen+Tu%2E+Generalizing+pooling+functions+in+convolutional+neural+networks%3A+Mixed%2C+gated%2C+and+tree%2E+arXiv+preprint+arXiv%3A1509%2E08985%2C+2015%2E+%E2%80%94+%5BarXiv%3A1509%2E08985%5D%28h)
46. Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. Neural networks with few multiplications. ArXiv e-prints, abs/1510.03009, October 2015a. — [arXiv:1510.03009](https://arxiv.org/abs/1510.03009) [find](https://scholar.google.com/scholar?q=Zhouhan+Lin%2C+Matthieu+Courbariaux%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+Neural+networks+with+few+multiplications%2E+ArXiv+e%2Dprints%2C+abs%2F1510%2E03009%2C+October+2015a%2E+%E2%80%94+%5BarXiv%3A1510%2E03009%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
47. Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. Neural Networks with Few Multiplications. Iclr, pages 1–8, 2015b. URL http://arxiv.org/abs/1510.03009. — [arXiv:1510.03009](https://arxiv.org/abs/1510.03009) [↗](http://arxiv.org/abs/1510.03009)
48. Chris Lomont. Fast inverse square root. Tech-315 nical Report, page 32, 2003. [find](https://scholar.google.com/scholar?q=Chris+Lomont%2E+Fast+inverse+square+root%2E+Tech%2D315+nical+Report%2C+page+32%2C+2003%2E)
49. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. [Building a large annotated corpus of english: The penn treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational linguistics, 19(2):313– 330, 1993.
50. Paul Merolla, Rathinakumar Appuswamy, John Arthur, Steve K Esser, and Dharmendra Modha. Deep neural networks are robust to weight binarization and other non-linear distortions. arXiv preprint arXiv:1606.01981, 2016. — [arXiv:1606.01981](https://arxiv.org/abs/1606.01981) [find](https://scholar.google.com/scholar?q=Paul+Merolla%2C+Rathinakumar+Appuswamy%2C+John+Arthur%2C+Steve+K+Esser%2C+and+Dharmendra+Modha%2E+Deep+neural+networks+are+robust+to+weight+binarization+and+other+non%2Dlinear+distortions%2E+arXiv+preprint+arXiv%3A16)
51. Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In SLT, pages 234–239, 2012. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov+and+Geoffrey+Zweig%2E+Context+dependent+recurrent+neural+network+language+model%2E+In+SLT%2C+pages+234%E2%80%93239%2C+2012%2E)
52. Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv:1603.01025, 2016. — [arXiv:1603.01025](https://arxiv.org/abs/1603.01025) [find](https://scholar.google.com/scholar?q=Daisuke+Miyashita%2C+Edward+H+Lee%2C+and+Boris+Murmann%2E+Convolutional+neural+networks+using+logarithmic+data+representation%2E+arXiv+preprint+arXiv%3A1603%2E01025%2C+2016%2E+%E2%80%94+%5BarXiv%3A1603%2E01025%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
53. Volodymyr Mnih, Koray Kavukcuoglo, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidgeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharsan Kumaran, Daan Wierstra, Shane Legg, and Demis Ha… [find](https://scholar.google.com/scholar?q=Volodymyr+Mnih%2C+Koray+Kavukcuoglo%2C+David+Silver%2C+Andrei+A%2E+Rusu%2C+Joel+Veness%2C+Marc+G%2E+Bellemare%2C+Alex+Graves%2C+Martin+Riedmiller%2C+Andreas+K%2E+Fidgeland%2C+Georg+Ostrovski%2C+Stig+Petersen%2C+Charles+Beattie%2C+)
54. Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks, 2015. URL [http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html](http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html). Accessed: 2015-06-30. [↗](http://googleresearch.blogspot.co.uk/2015/06/inceptionism-going-deeper-into-neural.html)
55. Joachim Ott, Zhouhan Lin, Ying Zhang, Shih-Chii Liu, and Yoshua Bengio. Recurrent neural networks with limited numerical precision. arXiv preprint arXiv:1608.06902, 2016. — [arXiv:1608.06902](https://arxiv.org/abs/1608.06902) [find](https://scholar.google.com/scholar?q=Joachim+Ott%2C+Zhouhan+Lin%2C+Ying+Zhang%2C+Shih%2DChii+Liu%2C+and+Yoshua+Bengio%2E+Recurrent+neural+networks+with+limited+numerical+precision%2E+arXiv+preprint+arXiv%3A1608%2E06902%2C+2016%2E+%E2%80%94+%5BarXiv%3A1608%2E06902%5D%28https%3A%2F%2F)
56. Phi-Hung Pham, Darko Jelaca, Clement Farabet, Berin Martini, Yann LeCun, and Eugenio Culurciello. Neuflow: Dataflow vision processing system-on-a-chip. In Circuits and Systems (MWSCAS), 2012 IEEE 55th International Midwest Symposium on, pages 1044–1047. [find](https://scholar.google.com/scholar?q=Phi%2DHung+Pham%2C+Darko+Jelaca%2C+Clement+Farabet%2C+Berin+Martini%2C+Yann+LeCun%2C+and+Eugenio+Culurciello%2E+Neuflow%3A+Dataflow+vision+processing+system%2Don%2Da%2Dchip%2E+In+Circuits+and+Systems+%28MWSCAS%29%2C+2012+IEEE+55th)
57. Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. [Xnor-net: Imagenet classification using binary convolutional neural networks](https://md.research.elamur.ai/free/en/papers/rastegari-2016-xnor-net-imagenet-classification-using/). arXiv preprint arXiv:1603.05279, 2016.
58. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. [find](https://scholar.google.com/scholar?q=Adriana+Romero%2C+Nicolas+Ballas%2C+Samira+Ebrahimi+Kahou%2C+Antoine+Chassang%2C+Carlo+Gatta%2C+and+Yoshua+Bengio%2E+Fitnets%3A+Hints+for+thin+deep+nets%2E+arXiv+preprint+arXiv%3A1412%2E6550%2C+2014%2E)
59. Tara Sainath, Abdel rahman Mohamed, Brian Kingsbury, and Bhuvana Ramabhadran. Deep convolutional neural networks for LVCSR. In ICASSP 2013, 2013. [find](https://scholar.google.com/scholar?q=Tara+Sainath%2C+Abdel+rahman+Mohamed%2C+Brian+Kingsbury%2C+and+Bhuvana+Ramabhadran%2E+Deep+convolutional+neural+networks+for+LVCSR%2E+In+ICASSP+2013%2C+2013%2E)
60. David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukc… [find](https://scholar.google.com/scholar?q=David+Silver%2C+Aja+Huang%2C+Chris+J%2E+Maddison%2C+Arthur+Guez%2C+Laurent+Sifre%2C+George+van+den+Driessche%2C+Julian+Schrittwieser%2C+Ioannis+Antonoglou%2C+Veda+Panneershelvam%2C+Marc+Lanctot%2C+Sander+Dieleman%2C+Dominik+)
61. Karen Simonyan and Andrew Zisserman. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). In ICLR, 2015.
62. Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In NIPS’2014, 2014. [find](https://scholar.google.com/scholar?q=Daniel+Soudry%2C+Itay+Hubara%2C+and+Ron+Meir%2E+Expectation+backpropagation%3A+Parameter%2Dfree+training+of+multilayer+neural+networks+with+continuous+or+discrete+weights%2E+In+NIPS%E2%80%992014%2C+2014%2E)
63. H Spang and P Schultheiss. Reduction of quantizing noise by use of feedback. IRE Transactions on Communications Systems, 10(4):373–380, 1962. [find](https://scholar.google.com/scholar?q=H+Spang+and+P+Schultheiss%2E+Reduction+of+quantizing+noise+by+use+of+feedback%2E+IRE+Transactions+on+Communications+Systems%2C+10%284%29%3A373%E2%80%93380%2C+1962%2E)
64. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. [Dropout: A simple way to prevent neural networks from overfitting](https://md.research.elamur.ai/free/en/papers/srivastava-2014-dropout-simple-way-prevent/). Journal of Machine Learning Research, 15:1929–1958, 2014. URL [http://jmlr.org/papers/v15/srivastava14a.html](http://jmlr.org/papers/v15/srivastava14a.html). [↗](http://jmlr.org/papers/v15/srivastava14a.html)
65. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In NIPS’2014, 2014.
66. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. Technical report, arXiv:1409.4842, 2014. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wei+Liu%2C+Yangqing+Jia%2C+Pierre+Sermanet%2C+Scott+Reed%2C+Dragomir+Anguelov%2C+Dumitru+Erhan%2C+Vincent+Vanhoucke%2C+and+Andrew+Rabinovich%2E+Going+deeper+with+convolutions%2E+Technical+report%2C+arX)
67. Yichuan Tang. Deep learning using linear support vector machines. Workshop on Challenges in Representation Learning, ICML, 2013. [find](https://scholar.google.com/scholar?q=Yichuan+Tang%2E+Deep+learning+using+linear+support+vector+machines%2E+Workshop+on+Challenges+in+Representation+Learning%2C+ICML%2C+2013%2E)
68. Naoya Torii, Hirotaka Kokubo, Dai Yamamoto, Kouichi Itoh, Masahiko Takenaka, and Tsutomu Matsumoto. Asic implementation of random number generators using sr latches and its evaluation. EURASIP Journal on Information Security, 2016(1):1–12, 2016. [find](https://scholar.google.com/scholar?q=Naoya+Torii%2C+Hirotaka+Kokubo%2C+Dai+Yamamoto%2C+Kouichi+Itoh%2C+Masahiko+Takenaka%2C+and+Tsutomu+Matsumoto%2E+Asic+implementation+of+random+number+generators+using+sr+latches+and+its+evaluation%2E+EURASIP+Journal)
69. Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks on CPUs. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, 2011. [find](https://scholar.google.com/scholar?q=Vincent+Vanhoucke%2C+Andrew+Senior%2C+and+Mark+Z+Mao%2E+Improving+the+speed+of+neural+networks+on+CPUs%2E+In+Proc%2E+Deep+Learning+and+Unsupervised+Feature+Learning+NIPS+Workshop%2C+2011%2E)
70. Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICML’2013, 2013. [find](https://scholar.google.com/scholar?q=Li+Wan%2C+Matthew+Zeiler%2C+Sixin+Zhang%2C+Yann+LeCun%2C+and+Rob+Fergus%2E+Regularization+of+neural+networks+using+dropconnect%2E+In+ICML%E2%80%992013%2C+2013%2E)
71. Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, and Jian Sun. Efficient and accurate approximations of nonlinear convolutional networks. pages 1984–1992, 2015. [find](https://scholar.google.com/scholar?q=Xiangyu+Zhang%2C+Jianhua+Zou%2C+Xiang+Ming%2C+Kaiming+He%2C+and+Jian+Sun%2E+Efficient+and+accurate+approximations+of+nonlinear+convolutional+networks%2E+pages+1984%E2%80%931992%2C+2015%2E)
72. Weiyi Zheng and Yina Tang. Binarized neural networks for language modeling. Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. [Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients](https://md.research.elamur.ai/free/en/papers/zhou-2016-dorefa-net-training-low-bitwidth/). arXiv preprint arXiv:1606.06160, 2016.

*Updated 2 November 2026*