---
title: "Demystifying Parallel and Distributed Deep Learning: An In-Depth"
description: "Training modern deep neural networks has become so computationally heavy that it effectively… 269 references, cited by 72 papers here."
authors: ["Tal Ben-Nun", "Torsten Hoefler"]
year: 2018
venue: "ACM Computing Surveys 2019"
source: "https://arxiv.org/abs/1802.09941"
arxiv: "1802.09941"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nun-2018-demystifying-parallel-distributed-deep-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nun-2018-demystifying-parallel-distributed-deep-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

# Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

**Authors:** Tal Ben-Nun and [Torsten Hoefler](https://md.research.elamur.ai/free/en/authors/torsten-hoefler/) · **Year:** 2018 · **Venue:** ACM Computing Surveys 2019 · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1802.09941) · [PDF](https://arxiv.org/pdf/1802.09941) · [arXiv 1802.09941](https://arxiv.org/abs/1802.09941) · [OpenAlex](https://openalex.org/W2788193959)

**Citations:** 203 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/nun-2018-demystifying-parallel-distributed-deep-learning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training modern deep neural networks has become so computationally heavy that it effectively requires high-performance clusters. However, the many ways to parallelize this work are scattered across machine learning, systems and hardware research. Ben-Nun and Hoefler organize this landscape in a survey of 240 works, gathered by following citations backward from seminal papers (as far back as 1984) and adding results from keyword searches on Google Scholar and arXiv.

They first set up foundations: supervised learning, SGD, the Work-Depth model and MPI. They then discuss how concurrency trades off against generalization. From there they move up the stack:
- concurrency inside single operators;
- concurrency across whole networks during inference and training;
- distributed training, covering asynchronous optimization, parameter-server versus decentralized designs, communication reduction and parallel neural architecture search.

Where feasible, each approach is characterized by its work, depth and average parallelism, which gives a common yardstick for comparison. The central conclusion is that almost every level of deep learning exposes parallelism. Loosening consistency requirements can unlock even more of it while keeping accuracy acceptable. This makes the survey a practical map for choosing a scaling strategy.

## Key points
- The survey covers 240 papers. Coverage of arXiv may be incomplete because deep learning output there grows roughly quadratically (cs.CV went from 577 to 5,693 yearly papers in Table 1).
- Concurrency is analyzed with the Work-Depth (W-D) model across four levels: operators, networks, training, and distributed systems with agents and parameter servers.
- Genetic-algorithm architecture search has run at very large scale: 18,000 nodes of the Titan supercomputer for 24 hours.
- Regularized Evolution (AmoebaNets) reached 3.8% error on ImageNet and 2.13% on CIFAR-10. The best DenseNet instances reached 5.29% and 3.62%.
- Whole-graph compilers (TensorFlow XLA, Tensor Comprehensions, Latte, TVM) report about 4× speedup over hand-tuned individual operators. The authors expect future work to target inter-layer and whole-DNN optimization.

## Interesting ideas
- **A shared cost model.** Describing very different techniques with one abstract model (work, depth, average parallelism) makes them directly comparable. The same approach could rank data pipelines or NLP preprocessing steps by how well they parallelize.
- **Relaxed consistency.** Because nonlinear optimization tolerates noise, strict synchronization can often be loosened to gain concurrency at small accuracy cost. This is worth testing whenever a sequential bottleneck appears in iterative training.
- **Tournament selection.** Comparing population members in pairs replaces a global, synchronous "pick the fittest" step. The general trick is useful for any parallel search or hyperparameter sweep that would otherwise stall on a global reduction.
- **Age-based removal.** Regularized Evolution drops the oldest candidate each iteration instead of the worst. This cheap rule regularizes the search and could be applied to other evolutionary or population-based model selection.
- **Literature-collection method.** The survey combines recursive tracking of citations from seminal papers with keyword searches and publishes the categorized list online. This is a reusable, transparent recipe for building literature reviews in a shared research library.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) — University, Zurich, Switzerland · 84 works · 9,181 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

## Cite

Cite as: Tal Ben-Nun and Torsten Hoefler. 2018. Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis. ACM Computing Surveys 2019.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bennun2018demystifying,
  title = {{Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis}},
  author = {Tal Ben-Nun and Torsten Hoefler},
  booktitle = {ACM Computing Surveys 2019},
  year = {2018},
  eprint = {1802.09941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.09941}
}
```

- [← previous What do RNN Language Models Learn about Filler-Gap Dependencies?](https://md.research.elamur.ai/free/en/papers/wilcox-2018-rnn-language-models-learn-about/)
- [next → Zero-Shot Transfer Learning for Event Extraction](https://md.research.elamur.ai/free/en/papers/huang-2018-zero-shot-transfer-learning-event/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

269 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Abadi et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). [http://www.tensorflow.org](http://www.tensorflow.org) [↗](http://www.tensorflow.org)
2. A. Agarwal and J. C. Duchi. 2011. Distributed Delayed Stochastic Optimization. In Advances in Neural Information Processing Systems 24. 873–881. [find](https://scholar.google.com/scholar?q=A%2E+Agarwal+and+J%2E+C%2E+Duchi%2E+2011%2E+Distributed+Delayed+Stochastic+Optimization%2E+In+Advances+in+Neural+Information+Processing+Systems+24%2E+873%E2%80%93881%2E)
3. A. F. Aji and K. Heafield. 2017. Sparse Communication for Distributed Gradient Descent. (2017). arXiv:1704.05021 [find](https://scholar.google.com/scholar?q=A%2E+F%2E+Aji+and+K%2E+Heafield%2E+2017%2E+Sparse+Communication+for+Distributed+Gradient+Descent%2E+%282017%29%2E+arXiv%3A1704%2E05021)
4. F. Akopyan et al. 2015. TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 34, 10 (2015), 1537–1557. [find](https://scholar.google.com/scholar?q=F%2E+Akopyan+et+al%2E+2015%2E+TrueNorth%3A+Design+and+Tool+Flow+of+a+65+mW+1+Million+Neuron+Programmable+Neurosynaptic+Chip%2E+IEEE+Transactions+on+Computer%2DAided+Design+of+Integrated+Circuits+and+Systems+34%2C+1)
5. D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. 2017. [QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding](https://md.research.elamur.ai/free/en/papers/alistarh-2016-qsgd-communication-efficient-sgd-gradient/). In Advances in Neural Information Processing Systems 30. 1709–1720.
6. D. Amodei et al. 2016. [Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/). In Proc. 33rd International Conference on Machine Learning, Vol. 48. 173–182.
7. J. Appleyard, T. Kociský, and P. Blunsom. 2016. Optimizing Performance of Recurrent Neural Networks on GPUs. (2016). arXiv:1604.01946 [find](https://scholar.google.com/scholar?q=J%2E+Appleyard%2C+T%2E+Kocisk%C3%BD%2C+and+P%2E+Blunsom%2E+2016%2E+Optimizing+Performance+of+Recurrent+Neural+Networks+on+GPUs%2E+%282016%29%2E+arXiv%3A1604%2E01946)
8. N. S. Arora, R. D. Blumofe, and C. G. Plaxton. 1998. Thread Scheduling for Multiprogrammed Multiprocessors. In Proc. Tenth Annual ACM Symposium on Parallel Algorithms and Architectures (SPAA ’98). 119–129. [find](https://scholar.google.com/scholar?q=N%2E+S%2E+Arora%2C+R%2E+D%2E+Blumofe%2C+and+C%2E+G%2E+Plaxton%2E+1998%2E+Thread+Scheduling+for+Multiprogrammed+Multiprocessors%2E+In+Proc%2E+Tenth+Annual+ACM+Symposium+on+Parallel+Algorithms+and+Architectures+%28SPAA+%E2%80%9998%29%2E+119)
9. A. A. Awan, K. Hamidouche, J. M. Hashmi, and D. K. Panda. 2017. S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters. In Proc. 22nd ACM SIGPLAN Symp. on PPoPP. 193–205. [find](https://scholar.google.com/scholar?q=A%2E+A%2E+Awan%2C+K%2E+Hamidouche%2C+J%2E+M%2E+Hashmi%2C+and+D%2E+K%2E+Panda%2E+2017%2E+S%2DCaffe%3A+Co%2Ddesigning+MPI+Runtimes+and+Caffe+for+Scalable+Deep+Learning+on+Modern+GPU+Clusters%2E+In+Proc%2E+22nd+ACM+SIGPLAN+Symp%2E+on+PPoPP)
10. J. Ba and R. Caruana. 2014. Do Deep Nets Really Need to be Deep? In Advances in Neural Information Processing Systems 27. 2654–2662. [find](https://scholar.google.com/scholar?q=J%2E+Ba+and+R%2E+Caruana%2E+2014%2E+Do+Deep+Nets+Really+Need+to+be+Deep%3F+In+Advances+in+Neural+Information+Processing+Systems+27%2E+2654%E2%80%932662%2E)
11. J. Ba, R. Grosse, and J. Martens. 2017. Distributed Second-Order Optimization using Kronecker-Factored Approximations. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=J%2E+Ba%2C+R%2E+Grosse%2C+and+J%2E+Martens%2E+2017%2E+Distributed+Second%2DOrder+Optimization+using+Kronecker%2DFactored+Approximations%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
12. B. Baker, O. Gupta, N. Naik, and R. Raskar. 2017. Designing Neural Network Architectures using Reinforcement Learning. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=B%2E+Baker%2C+O%2E+Gupta%2C+N%2E+Naik%2C+and+R%2E+Raskar%2E+2017%2E+Designing+Neural+Network+Architectures+using+Reinforcement+Learning%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
13. B. Baker, O. Gupta, R. Raskar, and N. Naik. 2017. Practical Neural Network Performance Prediction for Early Stopping. (2017). arXiv:1705.10823 — [arXiv:1705.10823](https://arxiv.org/abs/1705.10823) [find](https://scholar.google.com/scholar?q=B%2E+Baker%2C+O%2E+Gupta%2C+R%2E+Raskar%2C+and+N%2E+Naik%2E+2017%2E+Practical+Neural+Network+Performance+Prediction+for+Early+Stopping%2E+%282017%29%2E+arXiv%3A1705%2E10823+%E2%80%94+%5BarXiv%3A1705%2E10823%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1705%2E10823%29)
14. J. Ballé, V. Laparra, and E. P. Simoncelli. 2017. End-to-end optimized image compression. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=J%2E+Ball%C3%A9%2C+V%2E+Laparra%2C+and+E%2E+P%2E+Simoncelli%2E+2017%2E+End%2Dto%2Dend+optimized+image+compression%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
15. R. Belli and T. Hoefler. 2015. Notified Access: Extending Remote Memory Access Programming Models for Producer-Consumer Synchronization. In Proc. 29th IEEE Int’l Parallel & Distributed Processing Symposium (IPDPS’15). [find](https://scholar.google.com/scholar?q=R%2E+Belli+and+T%2E+Hoefler%2E+2015%2E+Notified+Access%3A+Extending+Remote+Memory+Access+Programming+Models+for+Producer%2DConsumer+Synchronization%2E+In+Proc%2E+29th+IEEE+Int%E2%80%99l+Parallel+%26+Distributed+Processing+Symp)
16. T. Ben-Nun, E. Levy, A. Barak, and E. Rubin. 2015. Memory Access Patterns: The Missing Piece of the Multi-GPU Puzzle. In Proc. Int’l Conf. for High Performance Computing, Networking, Storage and Analysis (SC ’15). 19:1–19:12. [find](https://scholar.google.com/scholar?q=T%2E+Ben%2DNun%2C+E%2E+Levy%2C+A%2E+Barak%2C+and+E%2E+Rubin%2E+2015%2E+Memory+Access+Patterns%3A+The+Missing+Piece+of+the+Multi%2DGPU+Puzzle%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analy)
17. Y. Bengio. 2013. Deep Learning of Representations: Looking Forward. In Statistical Language and Speech Processing, SLSP. Proceedings. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2E+2013%2E+Deep+Learning+of+Representations%3A+Looking+Forward%2E+In+Statistical+Language+and+Speech+Processing%2C+SLSP%2E+Proceedings%2E)
18. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. 2007. Greedy Layer-Wise Training of Deep Networks. In Advances in Neural Information Processing Systems 19. 153–160. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2C+P%2E+Lamblin%2C+D%2E+Popovici%2C+and+H%2E+Larochelle%2E+2007%2E+Greedy+Layer%2DWise+Training+of+Deep+Networks%2E+In+Advances+in+Neural+Information+Processing+Systems+19%2E+153%E2%80%93160%2E)
19. Y. Bengio, P. Simard, and P. Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5, 2 (1994), 157–166. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2C+P%2E+Simard%2C+and+P%2E+Frasconi%2E+1994%2E+Learning+long%2Dterm+dependencies+with+gradient+descent+is+difficult%2E+IEEE+Transactions+on+Neural+Networks+5%2C+2+%281994%29%2C+157%E2%80%93166%2E)
20. P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer. 2017. Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent. In Advances in Neural Information Processing Systems 30. 119–129. [find](https://scholar.google.com/scholar?q=P%2E+Blanchard%2C+E%2E+M%2E+El+Mhamdi%2C+R%2E+Guerraoui%2C+and+J%2E+Stainer%2E+2017%2E+Machine+Learning+with+Adversaries%3A+Byzantine+Tolerant+Gradient+Descent%2E+In+Advances+in+Neural+Information+Processing+Systems+30%2E+119%E2%80%93)
21. R. D. Blumofe and C. E. Leiserson. 1999. Scheduling Multithreaded Computations by Work Stealing. J. ACM 46, 5 (1999), 720–748. [find](https://scholar.google.com/scholar?q=R%2E+D%2E+Blumofe+and+C%2E+E%2E+Leiserson%2E+1999%2E+Scheduling+Multithreaded+Computations+by+Work+Stealing%2E+J%2E+ACM+46%2C+5+%281999%29%2C+720%E2%80%93748%2E)
22. M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. 2016. End to End Learning for Self-Driving Cars. (2016). arXiv:1604.07316 — [arXiv:1604.07316](https://arxiv.org/abs/1604.07316) [find](https://scholar.google.com/scholar?q=M%2E+Bojarski%2C+D%2E+Del+Testa%2C+D%2E+Dworakowski%2C+B%2E+Firner%2C+B%2E+Flepp%2C+P%2E+Goyal%2C+L%2E+D%2E+Jackel%2C+M%2E+Monfort%2C+U%2E+Muller%2C+J%2E+Zhang%2C+X%2E+Zhang%2C+J%2E+Zhao%2C+and+K%2E+Zieba%2E+2016%2E+End+to+End+Learning+for+Self%2DDriving+Car)
23. L. Bottou, F. E. Curtis, and J. Nocedal. 2016. Optimization Methods for Large-Scale Machine Learning. (2016). arXiv:1606.04838 — [arXiv:1606.04838](https://arxiv.org/abs/1606.04838) [find](https://scholar.google.com/scholar?q=L%2E+Bottou%2C+F%2E+E%2E+Curtis%2C+and+J%2E+Nocedal%2E+2016%2E+Optimization+Methods+for+Large%2DScale+Machine+Learning%2E+%282016%29%2E+arXiv%3A1606%2E04838+%E2%80%94+%5BarXiv%3A1606%2E04838%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1606%2E04838%29)
24. S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah. 2005. Gossip algorithms: design, analysis and applications. In Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies., Vol. 3. 1653–1664. [find](https://scholar.google.com/scholar?q=S%2E+Boyd%2C+A%2E+Ghosh%2C+B%2E+Prabhakar%2C+and+D%2E+Shah%2E+2005%2E+Gossip+algorithms%3A+design%2C+analysis+and+applications%2E+In+Proceedings+IEEE+24th+Annual+Joint+Conference+of+the+IEEE+Computer+and+Communications+Socie)
25. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. 2011. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Found. Trends Mach. Learn. 3, 1 (2011), 1–122. [find](https://scholar.google.com/scholar?q=S%2E+Boyd%2C+N%2E+Parikh%2C+E%2E+Chu%2C+B%2E+Peleato%2C+and+J%2E+Eckstein%2E+2011%2E+Distributed+Optimization+and+Statistical+Learning+via+the+Alternating+Direction+Method+of+Multipliers%2E+Found%2E+Trends+Mach%2E+Learn%2E+3%2C+1+%282)
26. R. P. Brent. 1974. The Parallel Evaluation of General Arithmetic Expressions. J. ACM 21, 2 (1974), 201–206. [find](https://scholar.google.com/scholar?q=R%2E+P%2E+Brent%2E+1974%2E+The+Parallel+Evaluation+of+General+Arithmetic+Expressions%2E+J%2E+ACM+21%2C+2+%281974%29%2C+201%E2%80%93206%2E)
27. A. Brock, T. Lim, J. M. Ritchie, and N. Weston. 2017. SMASH: One-Shot Model Architecture Search through HyperNetworks. (2017). arXiv:1708.05344 — [arXiv:1708.05344](https://arxiv.org/abs/1708.05344) [find](https://scholar.google.com/scholar?q=A%2E+Brock%2C+T%2E+Lim%2C+J%2E+M%2E+Ritchie%2C+and+N%2E+Weston%2E+2017%2E+SMASH%3A+One%2DShot+Model+Architecture+Search+through+HyperNetworks%2E+%282017%29%2E+arXiv%3A1708%2E05344+%E2%80%94+%5BarXiv%3A1708%2E05344%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1708%2E05344%29)
28. R. H. Byrd, S. L. Hansen, J. Nocedal, and Y. Singer. 2016. A Stochastic Quasi-Newton Method for Large-Scale Optimization. SIAM Journal on Optimization 26, 2 (2016), 1008–1031. [find](https://scholar.google.com/scholar?q=R%2E+H%2E+Byrd%2C+S%2E+L%2E+Hansen%2C+J%2E+Nocedal%2C+and+Y%2E+Singer%2E+2016%2E+A+Stochastic+Quasi%2DNewton+Method+for+Large%2DScale+Optimization%2E+SIAM+Journal+on+Optimization+26%2C+2+%282016%29%2C+1008%E2%80%931031%2E)
29. E. Chan, M. Heimlich, A. Purkayastha, and R. van de Geijn. 2007. Collective Communication: Theory, Practice, and Experience: Research Articles. Concurr. Comput. : Pract. Exper. 19, 13 (2007), 1749–1783. [find](https://scholar.google.com/scholar?q=E%2E+Chan%2C+M%2E+Heimlich%2C+A%2E+Purkayastha%2C+and+R%2E+van+de+Geijn%2E+2007%2E+Collective+Communication%3A+Theory%2C+Practice%2C+and+Experience%3A+Research+Articles%2E+Concurr%2E+Comput%2E+%3A+Pract%2E+Exper%2E+19%2C+13+%282007%29%2C+1749%E2%80%93178)
30. W. Chan, N. Jaitly, Q. Le, and O. Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In IEEE Int’l Conf. on Acoustics, Speech and Signal Processing (ICASSP). 4960–4964. [find](https://scholar.google.com/scholar?q=W%2E+Chan%2C+N%2E+Jaitly%2C+Q%2E+Le%2C+and+O%2E+Vinyals%2E+2016%2E+Listen%2C+attend+and+spell%3A+A+neural+network+for+large+vocabulary+conversational+speech+recognition%2E+In+IEEE+Int%E2%80%99l+Conf%2E+on+Acoustics%2C+Speech+and+Signal+)
31. K. Chellapilla, S. Puri, and P. Simard. 2006. High Performance Convolutional Neural Networks for Document Processing. In Tenth International Workshop on Frontiers in Handwriting Recognition. [find](https://scholar.google.com/scholar?q=K%2E+Chellapilla%2C+S%2E+Puri%2C+and+P%2E+Simard%2E+2006%2E+High+Performance+Convolutional+Neural+Networks+for+Document+Processing%2E+In+Tenth+International+Workshop+on+Frontiers+in+Handwriting+Recognition%2E)
32. C.-Y. Chen, J. Choi, D. Brand, A. Agrawal, W. Zhang, and K. Gopalakrishnan. 2017. AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training. (2017). arXiv:1712.02679 — [arXiv:1712.02679](https://arxiv.org/abs/1712.02679) [find](https://scholar.google.com/scholar?q=C%2E%2DY%2E+Chen%2C+J%2E+Choi%2C+D%2E+Brand%2C+A%2E+Agrawal%2C+W%2E+Zhang%2C+and+K%2E+Gopalakrishnan%2E+2017%2E+AdaComp+%3A+Adaptive+Residual+Gradient+Compression+for+Data%2DParallel+Distributed+Training%2E+%282017%29%2E+arXiv%3A1712%2E02679+%E2%80%94+%5Ba)
33. K. Chen and Q. Huo. 2016. Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering. In IEEE ICASSP. 5880–5884. [find](https://scholar.google.com/scholar?q=K%2E+Chen+and+Q%2E+Huo%2E+2016%2E+Scalable+training+of+deep+learning+machines+by+incremental+block+training+with+intra%2Dblock+parallel+optimization+and+blockwise+model%2Dupdate+filtering%2E+In+IEEE+ICASSP%2E+5880%E2%80%9358)
34. T. Chen et al. 2018. [TVM: End-to-End Optimization Stack for Deep Learning](https://md.research.elamur.ai/free/en/papers/chen-2018-tvm-automated-end-end-optimizing/). (2018). arXiv:1802.04799
35. T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, and O. Temam. 2014. DianNao: A Small-footprint High-throughput Accelerator for Ubiquitous Machine-learning. In Proc. 19th Int’l Conf. on ASPLOS. 269–284. [find](https://scholar.google.com/scholar?q=T%2E+Chen%2C+Z%2E+Du%2C+N%2E+Sun%2C+J%2E+Wang%2C+C%2E+Wu%2C+Y%2E+Chen%2C+and+O%2E+Temam%2E+2014%2E+DianNao%3A+A+Small%2Dfootprint+High%2Dthroughput+Accelerator+for+Ubiquitous+Machine%2Dlearning%2E+In+Proc%2E+19th+Int%E2%80%99l+Conf%2E+on+ASPLOS%2E+269%E2%80%9328)
36. T. Chen, B. Xu, C. Zhang, and C. Guestrin. 2016. Training Deep Nets with Sublinear Memory Cost. (2016). arXiv:1604.06174 [find](https://scholar.google.com/scholar?q=T%2E+Chen%2C+B%2E+Xu%2C+C%2E+Zhang%2C+and+C%2E+Guestrin%2E+2016%2E+Training+Deep+Nets+with+Sublinear+Memory+Cost%2E+%282016%29%2E+arXiv%3A1604%2E06174)
37. Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, and J. Feng. 2017. Dual Path Networks. In Advances in Neural Information Processing Systems 30. 4470–4478. [find](https://scholar.google.com/scholar?q=Y%2E+Chen%2C+J%2E+Li%2C+H%2E+Xiao%2C+X%2E+Jin%2C+S%2E+Yan%2C+and+J%2E+Feng%2E+2017%2E+Dual+Path+Networks%2E+In+Advances+in+Neural+Information+Processing+Systems+30%2E+4470%E2%80%934478%2E)
38. S. Chetlur et al. 2014. [cuDNN: Efficient Primitives for Deep Learning](https://md.research.elamur.ai/free/en/papers/chetlur-2014-cudnn-efficient-primitives-deep-learning/). (2014). arXiv:1410.0759
39. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. 2014. Project Adam: Building an Efficient and Scalable Deep Learning Training System. In 11th USENIX Symposium on Operating Systems Design and Implementation. 571–582. [find](https://scholar.google.com/scholar?q=T%2E+Chilimbi%2C+Y%2E+Suzue%2C+J%2E+Apacible%2C+and+K%2E+Kalyanaraman%2E+2014%2E+Project+Adam%3A+Building+an+Efficient+and+Scalable+Deep+Learning+Training+System%2E+In+11th+USENIX+Symposium+on+Operating+Systems+Design+and+)
40. K. Cho et al. 2014. [Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/). In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP). 1724–1734.
41. F. Chollet. 2016. Xception: Deep Learning with Depthwise Separable Convolutions. (2016). arXiv:1610.02357 [find](https://scholar.google.com/scholar?q=F%2E+Chollet%2E+2016%2E+Xception%3A+Deep+Learning+with+Depthwise+Separable+Convolutions%2E+%282016%29%2E+arXiv%3A1610%2E02357)
42. C. Chu, S. K. Kim, Y. Lin, Y. Yu, G. Bradski, K. Olukotun, and A. Y. Ng. 2007. Map-Reduce for Machine Learning on Multicore. In Advances in Neural Information Processing Systems 19. 281–288. [find](https://scholar.google.com/scholar?q=C%2E+Chu%2C+S%2E+K%2E+Kim%2C+Y%2E+Lin%2C+Y%2E+Yu%2C+G%2E+Bradski%2C+K%2E+Olukotun%2C+and+A%2E+Y%2E+Ng%2E+2007%2E+Map%2DReduce+for+Machine+Learning+on+Multicore%2E+In+Advances+in+Neural+Information+Processing+Systems+19%2E+281%E2%80%93288%2E)
43. I. H. Chung et al. 2017. Parallel Deep Neural Network Training for Big Data on Blue Gene/Q. IEEE Transactions on Parallel and Distributed Systems 28, 6 (2017), 1703–1714. [find](https://scholar.google.com/scholar?q=I%2E+H%2E+Chung+et+al%2E+2017%2E+Parallel+Deep+Neural+Network+Training+for+Big+Data+on+Blue+Gene%2FQ%2E+IEEE+Transactions+on+Parallel+and+Distributed+Systems+28%2C+6+%282017%29%2C+1703%E2%80%931714%2E)
44. D. C. Cireşan, A. Giusti, L. M. Gambardella, and J. Schmidhuber. 2013. Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks. In Medical Image Computing and Computer-Assisted Intervention. 411–418. [find](https://scholar.google.com/scholar?q=D%2E+C%2E+Cire%C5%9Fan%2C+A%2E+Giusti%2C+L%2E+M%2E+Gambardella%2C+and+J%2E+Schmidhuber%2E+2013%2E+Mitosis+Detection+in+Breast+Cancer+Histology+Images+with+Deep+Neural+Networks%2E+In+Medical+Image+Computing+and+Computer%2DAssisted+I)
45. A. Coates, B. Huval, T. Wang, D. J. Wu, A. Y. Ng, and B. Catanzaro. 2013. Deep Learning with COTS HPC Systems. In Proc. 30th International Conference on Machine Learning - Volume 28 (ICML’13). III–1337–III–1345. [find](https://scholar.google.com/scholar?q=A%2E+Coates%2C+B%2E+Huval%2C+T%2E+Wang%2C+D%2E+J%2E+Wu%2C+A%2E+Y%2E+Ng%2C+and+B%2E+Catanzaro%2E+2013%2E+Deep+Learning+with+COTS+HPC+Systems%2E+In+Proc%2E+30th+International+Conference+on+Machine+Learning+%2D+Volume+28+%28ICML%E2%80%9913%29%2E+III%E2%80%93133)
46. N. Cohen, O. Sharir, and A. Shashua. 2016. Deep SimNets. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4782–4791. [find](https://scholar.google.com/scholar?q=N%2E+Cohen%2C+O%2E+Sharir%2C+and+A%2E+Shashua%2E+2016%2E+Deep+SimNets%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2E+4782%E2%80%934791%2E)
47. N. Cohen, O. Sharir, and A. Shashua. 2016. On the Expressive Power of Deep Learning: A Tensor Analysis. In 29th Annual Conference on Learning Theory, Vol. 49. 698–728. [find](https://scholar.google.com/scholar?q=N%2E+Cohen%2C+O%2E+Sharir%2C+and+A%2E+Shashua%2E+2016%2E+On+the+Expressive+Power+of+Deep+Learning%3A+A+Tensor+Analysis%2E+In+29th+Annual+Conference+on+Learning+Theory%2C+Vol%2E+49%2E+698%E2%80%93728%2E)
48. R. Collobert, K. Kavukcuoglu, and C. Farabet. 2011. Torch7: A Matlab-like Environment for Machine Learning. In BigLearn, NIPS Workshop. [find](https://scholar.google.com/scholar?q=R%2E+Collobert%2C+K%2E+Kavukcuoglu%2C+and+C%2E+Farabet%2E+2011%2E+Torch7%3A+A+Matlab%2Dlike+Environment+for+Machine+Learning%2E+In+BigLearn%2C+NIPS+Workshop%2E)
49. J. Cong and B. Xiao. 2014. Minimizing Computation in Convolutional Neural Networks. In Int’l Conf. on Artificial Neural Networks (ICANN). 281–290. [find](https://scholar.google.com/scholar?q=J%2E+Cong+and+B%2E+Xiao%2E+2014%2E+Minimizing+Computation+in+Convolutional+Neural+Networks%2E+In+Int%E2%80%99l+Conf%2E+on+Artificial+Neural+Networks+%28ICANN%29%2E+281%E2%80%93290%2E)
50. M. Courbariaux and Y. Bengio. 2016. [BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1](https://md.research.elamur.ai/free/en/papers/courbariaux-2016-binarized-neural-networks-training-deep/). (2016). arXiv:1602.02830
51. M. Courbariaux, Y. Bengio, and J.-P. David. 2015. [BinaryConnect: Training Deep Neural Networks with Binary Weights During Propagations](https://md.research.elamur.ai/free/en/papers/courbariaux-2015-binaryconnect-training-deep-neural-networks/). In Proc. 28th Int’l Conf. on NIPS - Volume 2. 3123–3131.
52. H. Cui, H. Zhang, G. R. Ganger, P. B. Gibbons, and E. P. Xing. 2016. GeePS: Scalable Deep Learning on Distributed GPUs with a GPU-specialized Parameter Server. In Proc. EuroSys. 4:1–4:16. [find](https://scholar.google.com/scholar?q=H%2E+Cui%2C+H%2E+Zhang%2C+G%2E+R%2E+Ganger%2C+P%2E+B%2E+Gibbons%2C+and+E%2E+P%2E+Xing%2E+2016%2E+GeePS%3A+Scalable+Deep+Learning+on+Distributed+GPUs+with+a+GPU%2Dspecialized+Parameter+Server%2E+In+Proc%2E+EuroSys%2E+4%3A1%E2%80%934%3A16%2E)
53. D. Culler, R. Karp, D. Patterson, A. Sahay, K. E. Schauser, E. Santos, R. Subramonian, and T. von Eicken. 1993. LogP: Towards a Realistic Model of Parallel Computation. In Proc. Fourth ACM SIGPLAN Symp. on PPoPP. 1–12. [find](https://scholar.google.com/scholar?q=D%2E+Culler%2C+R%2E+Karp%2C+D%2E+Patterson%2C+A%2E+Sahay%2C+K%2E+E%2E+Schauser%2C+E%2E+Santos%2C+R%2E+Subramonian%2C+and+T%2E+von+Eicken%2E+1993%2E+LogP%3A+Towards+a+Realistic+Model+of+Parallel+Computation%2E+In+Proc%2E+Fourth+ACM+SIGPLAN+Sym)
54. J. Daily et al. 2018. GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent. (2018). arXiv:1803.05880 [find](https://scholar.google.com/scholar?q=J%2E+Daily+et+al%2E+2018%2E+GossipGraD%3A+Scalable+Deep+Learning+using+Gossip+Communication+based+Asynchronous+Gradient+Descent%2E+%282018%29%2E+arXiv%3A1803%2E05880)
55. C. De Sa, C. Zhang, K. Olukotun, and C. Ré. 2015. Taming the Wild: A Unified Analysis of HOGWILD!-style Algorithms. In Proc. 28th Int’l Conf. on NIPS - Volume 2. 2674–2682. [find](https://scholar.google.com/scholar?q=C%2E+De+Sa%2C+C%2E+Zhang%2C+K%2E+Olukotun%2C+and+C%2E+R%C3%A9%2E+2015%2E+Taming+the+Wild%3A+A+Unified+Analysis+of+HOGWILD%21%2Dstyle+Algorithms%2E+In+Proc%2E+28th+Int%E2%80%99l+Conf%2E+on+NIPS+%2D+Volume+2%2E+2674%E2%80%932682%2E)
56. J. Dean et al. 2012. Large Scale Distributed Deep Networks. In Proc. 25th International Conference on Neural Information Processing Systems - Volume 1 (NIPS’12). 1223–1231. [find](https://scholar.google.com/scholar?q=J%2E+Dean+et+al%2E+2012%2E+Large+Scale+Distributed+Deep+Networks%2E+In+Proc%2E+25th+International+Conference+on+Neural+Information+Processing+Systems+%2D+Volume+1+%28NIPS%E2%80%9912%29%2E+1223%E2%80%931231%2E)
57. J. Dean and S. Ghemawat. 2008. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM 51, 1 (2008), 107–113. [find](https://scholar.google.com/scholar?q=J%2E+Dean+and+S%2E+Ghemawat%2E+2008%2E+MapReduce%3A+Simplified+Data+Processing+on+Large+Clusters%2E+Commun%2E+ACM+51%2C+1+%282008%29%2C+107%E2%80%93113%2E)
58. O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao. 2012. Optimal Distributed Online Prediction Using Mini-batches. JMLR 13, 1 (2012), 165–202. [find](https://scholar.google.com/scholar?q=O%2E+Dekel%2C+R%2E+Gilad%2DBachrach%2C+O%2E+Shamir%2C+and+L%2E+Xiao%2E+2012%2E+Optimal+Distributed+Online+Prediction+Using+Mini%2Dbatches%2E+JMLR+13%2C+1+%282012%29%2C+165%E2%80%93202%2E)
59. O. Delalleau and Y. Bengio. 2011. Shallow vs. Deep Sum-Product Networks. In Advances in Neural Information Processing Systems 24. 666–674. [find](https://scholar.google.com/scholar?q=O%2E+Delalleau+and+Y%2E+Bengio%2E+2011%2E+Shallow+vs%2E+Deep+Sum%2DProduct+Networks%2E+In+Advances+in+Neural+Information+Processing+Systems+24%2E+666%E2%80%93674%2E)
60. J. Demmel and G. Dinh. 2018. Communication-Optimal Convolutional Neural Nets. (2018). arXiv:1802.06905 — [arXiv:1802.06905](https://arxiv.org/abs/1802.06905) [find](https://scholar.google.com/scholar?q=J%2E+Demmel+and+G%2E+Dinh%2E+2018%2E+Communication%2DOptimal+Convolutional+Neural+Nets%2E+%282018%29%2E+arXiv%3A1802%2E06905+%E2%80%94+%5BarXiv%3A1802%2E06905%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1802%2E06905%29)
61. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=J%2E+Deng%2C+W%2E+Dong%2C+R%2E+Socher%2C+L%2E%2DJ%2E+Li%2C+K%2E+Li%2C+and+L%2E+Fei%2DFei%2E+2009%2E+ImageNet%3A+A+Large%2DScale+Hierarchical+Image+Database%2E+In+The+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2E)
62. L. Deng, D. Yu, and J. Platt. 2012. Scalable stacking and learning for building deep architectures. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2133–2136. [find](https://scholar.google.com/scholar?q=L%2E+Deng%2C+D%2E+Yu%2C+and+J%2E+Platt%2E+2012%2E+Scalable+stacking+and+learning+for+building+deep+architectures%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2E+2133%E2%80%932136%2E)
63. T. Dettmers. 2015. 8-Bit Approximations for Parallelism in Deep Learning. (2015). arXiv:1511.04561 [find](https://scholar.google.com/scholar?q=T%2E+Dettmers%2E+2015%2E+8%2DBit+Approximations+for+Parallelism+in+Deep+Learning%2E+%282015%29%2E+arXiv%3A1511%2E04561)
64. G. Diamos et al. 2016. Persistent RNNs: Stashing Recurrent Weights On-Chip. In Proc. 33rd International Conference on Machine Learning, Vol. 48. 2024–2033. [find](https://scholar.google.com/scholar?q=G%2E+Diamos+et+al%2E+2016%2E+Persistent+RNNs%3A+Stashing+Recurrent+Weights+On%2DChip%2E+In+Proc%2E+33rd+International+Conference+on+Machine+Learning%2C+Vol%2E+48%2E+2024%E2%80%932033%2E)
65. T. G. Dietterich. 2000. Ensemble Methods in Machine Learning. In Proc. First International Workshop on Multiple Classifier Systems (MCS ’00). 1–15. [find](https://scholar.google.com/scholar?q=T%2E+G%2E+Dietterich%2E+2000%2E+Ensemble+Methods+in+Machine+Learning%2E+In+Proc%2E+First+International+Workshop+on+Multiple+Classifier+Systems+%28MCS+%E2%80%9900%29%2E+1%E2%80%9315%2E)
66. Z. Drezner and A. Barak. 1986. An asynchronous algorithm for scattering information between the active nodes of a multicomputer system. J. Parallel and Distrib. Comput. 3, 3 (1986), 344–351. [find](https://scholar.google.com/scholar?q=Z%2E+Drezner+and+A%2E+Barak%2E+1986%2E+An+asynchronous+algorithm+for+scattering+information+between+the+active+nodes+of+a+multicomputer+system%2E+J%2E+Parallel+and+Distrib%2E+Comput%2E+3%2C+3+%281986%29%2C+344%E2%80%93351%2E)
67. N. Dryden, T. Moon, S. A. Jacobs, and B. V. Essen. 2016. Communication Quantization for Data-Parallel Training of Deep Neural Networks. In 2nd Workshop on Machine Learning in HPC Environments (MLHPC). 1–8. [find](https://scholar.google.com/scholar?q=N%2E+Dryden%2C+T%2E+Moon%2C+S%2E+A%2E+Jacobs%2C+and+B%2E+V%2E+Essen%2E+2016%2E+Communication+Quantization+for+Data%2DParallel+Training+of+Deep+Neural+Networks%2E+In+2nd+Workshop+on+Machine+Learning+in+HPC+Environments+%28MLHPC%29%2E)
68. J. Duchi, E. Hazan, and Y. Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. JMLR 12 (2011), 2121–2159. [find](https://scholar.google.com/scholar?q=J%2E+Duchi%2C+E%2E+Hazan%2C+and+Y%2E+Singer%2E+2011%2E+Adaptive+Subgradient+Methods+for+Online+Learning+and+Stochastic+Optimization%2E+JMLR+12+%282011%29%2C+2121%E2%80%932159%2E)
69. V. Dumoulin and F. Visin. 2016. A guide to convolution arithmetic for deep learning. (2016). arXiv:1603.07285 — [arXiv:1603.07285](https://arxiv.org/abs/1603.07285) [find](https://scholar.google.com/scholar?q=V%2E+Dumoulin+and+F%2E+Visin%2E+2016%2E+A+guide+to+convolution+arithmetic+for+deep+learning%2E+%282016%29%2E+arXiv%3A1603%2E07285+%E2%80%94+%5BarXiv%3A1603%2E07285%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1603%2E07285%29)
70. J. L. Elman. 1990. Finding Structure in Time. Cognitive Science 14, 2 (1990), 179–211. [find](https://scholar.google.com/scholar?q=J%2E+L%2E+Elman%2E+1990%2E+Finding+Structure+in+Time%2E+Cognitive+Science+14%2C+2+%281990%29%2C+179%E2%80%93211%2E)
71. T. Elsken, J.-H. Metzen, and F. Hutter. 2017. Simple And Efficient Architecture Search for Convolutional Neural Networks. (2017). arXiv:1711.04528 [find](https://scholar.google.com/scholar?q=T%2E+Elsken%2C+J%2E%2DH%2E+Metzen%2C+and+F%2E+Hutter%2E+2017%2E+Simple+And+Efficient+Architecture+Search+for+Convolutional+Neural+Networks%2E+%282017%29%2E+arXiv%3A1711%2E04528)
72. L. Ericson and R. Mbuvha. 2017. On the Performance of Network Parallel Training in Artificial Neural Networks. (2017). arXiv:1701.05130 — [arXiv:1701.05130](https://arxiv.org/abs/1701.05130) [find](https://scholar.google.com/scholar?q=L%2E+Ericson+and+R%2E+Mbuvha%2E+2017%2E+On+the+Performance+of+Network+Parallel+Training+in+Artificial+Neural+Networks%2E+%282017%29%2E+arXiv%3A1701%2E05130+%E2%80%94+%5BarXiv%3A1701%2E05130%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1701%2E05130%29)
73. P. Farber and K. Asanovic. 1997. Parallel neural network training on Multi-Spert. In Proceedings of 3rd International Conference on Algorithms and Architectures for Parallel Processing. 659–666. [find](https://scholar.google.com/scholar?q=P%2E+Farber+and+K%2E+Asanovic%2E+1997%2E+Parallel+neural+network+training+on+Multi%2DSpert%2E+In+Proceedings+of+3rd+International+Conference+on+Algorithms+and+Architectures+for+Parallel+Processing%2E+659%E2%80%93666%2E)
74. K. Frans, J. Ho, X. Chen, P. Abbeel, and J. Schulman. 2017. Meta Learning Shared Hierarchies. (2017). arXiv:1710.09767 [find](https://scholar.google.com/scholar?q=K%2E+Frans%2C+J%2E+Ho%2C+X%2E+Chen%2C+P%2E+Abbeel%2C+and+J%2E+Schulman%2E+2017%2E+Meta+Learning+Shared+Hierarchies%2E+%282017%29%2E+arXiv%3A1710%2E09767)
75. M. P. Friedlander and M. W. Schmidt. 2011. Hybrid Deterministic-Stochastic Methods for Data Fitting. (2011). arXiv:1104.2373 — [arXiv:1104.2373](https://arxiv.org/abs/1104.2373) [find](https://scholar.google.com/scholar?q=M%2E+P%2E+Friedlander+and+M%2E+W%2E+Schmidt%2E+2011%2E+Hybrid+Deterministic%2DStochastic+Methods+for+Data+Fitting%2E+%282011%29%2E+arXiv%3A1104%2E2373+%E2%80%94+%5BarXiv%3A1104%2E2373%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1104%2E2373%29)
76. A. Gaunt, M. Johnson, M. Riechert, D. Tarlow, R. Tomioka, D. Vytiniotis, and S. Webster. 2017. AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks. (2017). arXiv:1705.09786 [find](https://scholar.google.com/scholar?q=A%2E+Gaunt%2C+M%2E+Johnson%2C+M%2E+Riechert%2C+D%2E+Tarlow%2C+R%2E+Tomioka%2C+D%2E+Vytiniotis%2C+and+S%2E+Webster%2E+2017%2E+AMPNet%3A+Asynchronous+Model%2DParallel+Training+for+Dynamic+Neural+Networks%2E+%282017%29%2E+arXiv%3A1705%2E09786)
77. A. Gholami, A. Azad, P. H. Jin, K. Keutzer, and A. Buluç. 2018. Integrated Model, Batch, and Domain Parallelism in Training Neural Networks. In Proc. 30th on Symp. on Parallelism in Algorithms and Architectures, SPAA 2018. 77–86. [find](https://scholar.google.com/scholar?q=A%2E+Gholami%2C+A%2E+Azad%2C+P%2E+H%2E+Jin%2C+K%2E+Keutzer%2C+and+A%2E+Bulu%C3%A7%2E+2018%2E+Integrated+Model%2C+Batch%2C+and+Domain+Parallelism+in+Training+Neural+Networks%2E+In+Proc%2E+30th+on+Symp%2E+on+Parallelism+in+Algorithms+and+Arc)
78. A. Gibiansky. 2017. Bringing HPC Techniques to Deep Learning. (2017). [http://research.baidu.com/bringing-hpc-techniques-deep-learning/](http://research.baidu.com/bringing-hpc-techniques-deep-learning/) [↗](http://research.baidu.com/bringing-hpc-techniques-deep-learning/)
79. X. Glorot and Y. Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proc. Thirteenth International Conference on Artificial Intelligence and Statistics, Vol. 9. 249–256. [find](https://scholar.google.com/scholar?q=X%2E+Glorot+and+Y%2E+Bengio%2E+2010%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+Proc%2E+Thirteenth+International+Conference+on+Artificial+Intelligence+and+Statistics%2C+Vol%2E+9)
80. D. E. Goldberg and K. Deb. 1991. A Comparative Analysis of Selection Schemes Used in Genetic Algorithms. Foundations of Genetic Algorithms, Vol. 1. 69–93. [find](https://scholar.google.com/scholar?q=D%2E+E%2E+Goldberg+and+K%2E+Deb%2E+1991%2E+A+Comparative+Analysis+of+Selection+Schemes+Used+in+Genetic+Algorithms%2E+Foundations+of+Genetic+Algorithms%2C+Vol%2E+1%2E+69%E2%80%9393%2E)
81. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. [Generative Adversarial Nets](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-generative-adversarial-nets/). In Advances in Neural Information Processing Systems 27. 2672–2680.
82. Google. 2017. TensorFlow XLA Overview. (2017). [https://www.tensorflow.org/performance/xla](https://www.tensorflow.org/performance/xla) [↗](https://www.tensorflow.org/performance/xla)
83. P. Goyal, P. Dollár, R. B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. 2017. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. (2017). arXiv:1706.02677 [find](https://scholar.google.com/scholar?q=P%2E+Goyal%2C+P%2E+Doll%C3%A1r%2C+R%2E+B%2E+Girshick%2C+P%2E+Noordhuis%2C+L%2E+Wesolowski%2C+A%2E+Kyrola%2C+A%2E+Tulloch%2C+Y%2E+Jia%2C+and+K%2E+He%2E+2017%2E+Accurate%2C+Large+Minibatch+SGD%3A+Training+ImageNet+in+1+Hour%2E+%282017%29%2E+arXiv%3A1706%2E02677)
84. A. Graves et al. 2016. Hybrid computing using a neural network with dynamic external memory. Nature 538, 7626 (2016), 471–476. [find](https://scholar.google.com/scholar?q=A%2E+Graves+et+al%2E+2016%2E+Hybrid+computing+using+a+neural+network+with+dynamic+external+memory%2E+Nature+538%2C+7626+%282016%29%2C+471%E2%80%93476%2E)
85. W. Gropp, T. Hoefler, R. Thakur, and E. Lusk. 2014. Using Advanced MPI: Modern Features of the Message-Passing Interface. MIT Press. [find](https://scholar.google.com/scholar?q=W%2E+Gropp%2C+T%2E+Hoefler%2C+R%2E+Thakur%2C+and+E%2E+Lusk%2E+2014%2E+Using+Advanced+MPI%3A+Modern+Features+of+the+Message%2DPassing+Interface%2E+MIT+Press%2E)
86. A. Gruslys, R. Munos, I. Danihelka, M. Lanctot, and A. Graves. 2016. Memory-Efficient Backpropagation Through Time. In Advances in Neural Information Processing Systems 29. 4125–4133. [find](https://scholar.google.com/scholar?q=A%2E+Gruslys%2C+R%2E+Munos%2C+I%2E+Danihelka%2C+M%2E+Lanctot%2C+and+A%2E+Graves%2E+2016%2E+Memory%2DEfficient+Backpropagation+Through+Time%2E+In+Advances+in+Neural+Information+Processing+Systems+29%2E+4125%E2%80%934133%2E)
87. S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. 2015. Deep Learning with Limited Numerical Precision. In Proc. 32nd International Conference on Machine Learning, Vol. 37. 1737–1746. [find](https://scholar.google.com/scholar?q=S%2E+Gupta%2C+A%2E+Agrawal%2C+K%2E+Gopalakrishnan%2C+and+P%2E+Narayanan%2E+2015%2E+Deep+Learning+with+Limited+Numerical+Precision%2E+In+Proc%2E+32nd+International+Conference+on+Machine+Learning%2C+Vol%2E+37%2E+1737%E2%80%931746%2E)
88. S. Gupta, W. Zhang, and F. Wang. 2016. Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study. In IEEE 16th International Conference on Data Mining (ICDM). 171–180. [find](https://scholar.google.com/scholar?q=S%2E+Gupta%2C+W%2E+Zhang%2C+and+F%2E+Wang%2E+2016%2E+Model+Accuracy+and+Runtime+Tradeoff+in+Distributed+Deep+Learning%3A+A+Systematic+Study%2E+In+IEEE+16th+International+Conference+on+Data+Mining+%28ICDM%29%2E+171%E2%80%93180%2E)
89. S. Hadjis, C. Zhang, I. Mitliagkas, and C. Ré. 2016. Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs. (2016). arXiv:1606.04487 [find](https://scholar.google.com/scholar?q=S%2E+Hadjis%2C+C%2E+Zhang%2C+I%2E+Mitliagkas%2C+and+C%2E+R%C3%A9%2E+2016%2E+Omnivore%3A+An+Optimizer+for+Multi%2Ddevice+Deep+Learning+on+CPUs+and+GPUs%2E+%282016%29%2E+arXiv%3A1606%2E04487)
90. S. Han, H. Mao, and W. J. Dally. 2016. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. International Conference on Learning Representations (ICLR) (2016). [find](https://scholar.google.com/scholar?q=S%2E+Han%2C+H%2E+Mao%2C+and+W%2E+J%2E+Dally%2E+2016%2E+Deep+Compression%3A+Compressing+Deep+Neural+Network+with+Pruning%2C+Trained+Quantization+and+Huffman+Coding%2E+International+Conference+on+Learning+Representations+%28IC)
91. E. Hazan, A. Klivans, and Y. Yuan. 2018. Hyperparameter optimization: a spectral approach. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=E%2E+Hazan%2C+A%2E+Klivans%2C+and+Y%2E+Yuan%2E+2018%2E+Hyperparameter+optimization%3A+a+spectral+approach%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
92. K. He, X. Zhang, S. Ren, and J. Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proc. IEEE International Conference on Computer Vision (ICCV ’15). 1026–1034. [find](https://scholar.google.com/scholar?q=K%2E+He%2C+X%2E+Zhang%2C+S%2E+Ren%2C+and+J%2E+Sun%2E+2015%2E+Delving+Deep+into+Rectifiers%3A+Surpassing+Human%2DLevel+Performance+on+ImageNet+Classification%2E+In+Proc%2E+IEEE+International+Conference+on+Computer+Vision+%28ICCV+)
93. K. He, X. Zhang, S. Ren, and J. Sun. 2016. [Deep Residual Learning for Image Recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770–778.
94. X. He, D. Mudigere, M. Smelyanskiy, and M. Takac. 2017. Distributed Hessian-Free Optimization for Deep Neural Network. In AAAI Workshops. [find](https://scholar.google.com/scholar?q=X%2E+He%2C+D%2E+Mudigere%2C+M%2E+Smelyanskiy%2C+and+M%2E+Takac%2E+2017%2E+Distributed+Hessian%2DFree+Optimization+for+Deep+Neural+Network%2E+In+AAAI+Workshops%2E)
95. G. Hinton. 2012. Neural Networks for Machine Learning, Lecture 6a: Overview of Mini-batch Gradient Descent. (2012). [find](https://scholar.google.com/scholar?q=G%2E+Hinton%2E+2012%2E+Neural+Networks+for+Machine+Learning%2C+Lecture+6a%3A+Overview+of+Mini%2Dbatch+Gradient+Descent%2E+%282012%29%2E)
96. G. Hinton, O. Vinyals, and J. Dean. 2015. [Distilling the Knowledge in a Neural Network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). In NIPS Deep Learning and Representation Learning Workshop.
97. G. E. Hinton, S. Osindero, and Y. W. Teh. 2006. A Fast Learning Algorithm for Deep Belief Nets. Neural Computation 18, 7 (2006), 1527–1554. [find](https://scholar.google.com/scholar?q=G%2E+E%2E+Hinton%2C+S%2E+Osindero%2C+and+Y%2E+W%2E+Teh%2E+2006%2E+A+Fast+Learning+Algorithm+for+Deep+Belief+Nets%2E+Neural+Computation+18%2C+7+%282006%29%2C+1527%E2%80%931554%2E)
98. Q. Ho et al. 2013. More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server. In Proc. 26th International Conference on Neural Information Processing Systems - Volume 1 (NIPS’13). 1223–1231. [find](https://scholar.google.com/scholar?q=Q%2E+Ho+et+al%2E+2013%2E+More+Effective+Distributed+ML+via+a+Stale+Synchronous+Parallel+Parameter+Server%2E+In+Proc%2E+26th+International+Conference+on+Neural+Information+Processing+Systems+%2D+Volume+1+%28NIPS%E2%80%9913%29)
99. S. Hochreiter and J. Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (1997), 1735–1780. [find](https://scholar.google.com/scholar?q=S%2E+Hochreiter+and+J%2E+Schmidhuber%2E+1997%2E+Long+Short%2DTerm+Memory%2E+Neural+Comput%2E+9%2C+8+%281997%29%2C+1735%E2%80%931780%2E)
100. T. Hoefler, A. Barak, A. Shiloh, and Z. Drezner. 2017. Corrected Gossip Algorithms for Fast Reliable Broadcast on Unreliable Systems. In Proc. 31st IEEE Int’l Parallel & Distributed Processing Symposium (IPDPS’17). [find](https://scholar.google.com/scholar?q=T%2E+Hoefler%2C+A%2E+Barak%2C+A%2E+Shiloh%2C+and+Z%2E+Drezner%2E+2017%2E+Corrected+Gossip+Algorithms+for+Fast+Reliable+Broadcast+on+Unreliable+Systems%2E+In+Proc%2E+31st+IEEE+Int%E2%80%99l+Parallel+%26+Distributed+Processing+Symposi)
101. T. Hoefler and D. Moor. 2014. Energy, Memory, and Runtime Tradeoffs for Implementing Collective Communication Operations. Journal of Supercomputing Frontiers and Innovations 1, 2 (2014), 58–75. [find](https://scholar.google.com/scholar?q=T%2E+Hoefler+and+D%2E+Moor%2E+2014%2E+Energy%2C+Memory%2C+and+Runtime+Tradeoffs+for+Implementing+Collective+Communication+Operations%2E+Journal+of+Supercomputing+Frontiers+and+Innovations+1%2C+2+%282014%29%2C+58%E2%80%9375%2E)
102. T. Hoefler and T. Schneider. 2012. Optimization Principles for Collective Neighborhood Communications. In Proc. International Conference on High Performance Computing, Networking, Storage and Analysis. 98:1–98:10. [find](https://scholar.google.com/scholar?q=T%2E+Hoefler+and+T%2E+Schneider%2E+2012%2E+Optimization+Principles+for+Collective+Neighborhood+Communications%2E+In+Proc%2E+International+Conference+on+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis)
103. T. Hoefler and M. Snir. 2011. Generic Topology Mapping Strategies for Large-scale Parallel Architectures. In Proc. 2011 ACM International Conference on Supercomputing (ICS’11). 75–85. [find](https://scholar.google.com/scholar?q=T%2E+Hoefler+and+M%2E+Snir%2E+2011%2E+Generic+Topology+Mapping+Strategies+for+Large%2Dscale+Parallel+Architectures%2E+In+Proc%2E+2011+ACM+International+Conference+on+Supercomputing+%28ICS%E2%80%9911%29%2E+75%E2%80%9385%2E)
104. T. Hoefler and J. L. Traeff. 2009. Sparse Collective Operations for MPI. In Proc. 23rd IEEE Int’l Parallel & Distributed Processing Symposium, HIPS’09 Workshop. [find](https://scholar.google.com/scholar?q=T%2E+Hoefler+and+J%2E+L%2E+Traeff%2E+2009%2E+Sparse+Collective+Operations+for+MPI%2E+In+Proc%2E+23rd+IEEE+Int%E2%80%99l+Parallel+%26+Distributed+Processing+Symposium%2C+HIPS%E2%80%9909+Workshop%2E)
105. E. Hoffer, I. Hubara, and D. Soudry. 2017. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In Advances in Neural Information Processing Systems 30. 1729–1739. [find](https://scholar.google.com/scholar?q=E%2E+Hoffer%2C+I%2E+Hubara%2C+and+D%2E+Soudry%2E+2017%2E+Train+longer%2C+generalize+better%3A+closing+the+generalization+gap+in+large+batch+training+of+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Syst)
106. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. 2017. [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://md.research.elamur.ai/free/en/papers/howard-2017-mobilenets-efficient-convolutional-neural-networks/). (2017). arXiv:1704.04861
107. K. Hsieh, A. Harlap, N. Vijaykumar, D. Konomis, G. R. Ganger, P. B. Gibbons, and O. Mutlu. 2017. Gaia: Geo-distributed Machine Learning Approaching LAN Speeds. In Proc. 14th USENIX Conf. on NSDI. 629–647. [find](https://scholar.google.com/scholar?q=K%2E+Hsieh%2C+A%2E+Harlap%2C+N%2E+Vijaykumar%2C+D%2E+Konomis%2C+G%2E+R%2E+Ganger%2C+P%2E+B%2E+Gibbons%2C+and+O%2E+Mutlu%2E+2017%2E+Gaia%3A+Geo%2Ddistributed+Machine+Learning+Approaching+LAN+Speeds%2E+In+Proc%2E+14th+USENIX+Conf%2E+on+NSDI%2E+629%E2%80%93)
108. G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. 2017. Densely connected convolutional networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=G%2E+Huang%2C+Z%2E+Liu%2C+L%2E+van+der+Maaten%2C+and+K%2E+Q%2E+Weinberger%2E+2017%2E+Densely+connected+convolutional+networks%2E+In+Proc%2E+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2E)
109. I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. 2016. [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://md.research.elamur.ai/free/en/papers/hubara-2016-quantized-neural-networks-training-neural/). (2016). arXiv:1609.07061
110. D. A. Huffman. 1952. A Method for the Construction of Minimum-Redundancy Codes. Proc. IRE 40, 9 (1952), 1098–1101. [find](https://scholar.google.com/scholar?q=D%2E+A%2E+Huffman%2E+1952%2E+A+Method+for+the+Construction+of+Minimum%2DRedundancy+Codes%2E+Proc%2E+IRE+40%2C+9+%281952%29%2C+1098%E2%80%931101%2E)
111. F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and \\<1MB model size. (2016). arXiv:1602.07360 [find](https://scholar.google.com/scholar?q=F%2E+N%2E+Iandola%2C+M%2E+W%2E+Moskewicz%2C+K%2E+Ashraf%2C+S%2E+Han%2C+W%2E+J%2E+Dally%2C+and+K%2E+Keutzer%2E+2016%2E+SqueezeNet%3A+AlexNet%2Dlevel+accuracy+with+50x+fewer+parameters+and+%5C%3C1MB+model+size%2E+%282016%29%2E+arXiv%3A1602%2E07360)
112. F. N. Iandola, M. W. Moskewicz, K. Ashraf, and K. Keutzer. 2016. FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=F%2E+N%2E+Iandola%2C+M%2E+W%2E+Moskewicz%2C+K%2E+Ashraf%2C+and+K%2E+Keutzer%2E+2016%2E+FireCaffe%3A+Near%2DLinear+Acceleration+of+Deep+Neural+Network+Training+on+Compute+Clusters%2E+In+The+IEEE+Conference+on+Computer+Vision+and+)
113. P. Ienne. 1993. Architectures for Neuro-Computers: Review and Performance Evaluation. Technical Report. EPFL, Lausanne, Switzerland. [find](https://scholar.google.com/scholar?q=P%2E+Ienne%2E+1993%2E+Architectures+for+Neuro%2DComputers%3A+Review+and+Performance+Evaluation%2E+Technical+Report%2E+EPFL%2C+Lausanne%2C+Switzerland%2E)
114. D. J. Im, H. Ma, C. D. Kim, and G. W. Taylor. 2016. Generative Adversarial Parallelization. (2016). arXiv:1612.04021 — [arXiv:1612.04021](https://arxiv.org/abs/1612.04021) [find](https://scholar.google.com/scholar?q=D%2E+J%2E+Im%2C+H%2E+Ma%2C+C%2E+D%2E+Kim%2C+and+G%2E+W%2E+Taylor%2E+2016%2E+Generative+Adversarial+Parallelization%2E+%282016%29%2E+arXiv%3A1612%2E04021+%E2%80%94+%5BarXiv%3A1612%2E04021%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1612%2E04021%29)
115. Intel. 2009. Intel Math Kernel Library. Reference Manual. Intel Corporation. [find](https://scholar.google.com/scholar?q=Intel%2E+2009%2E+Intel+Math+Kernel+Library%2E+Reference+Manual%2E+Intel+Corporation%2E)
116. Intel. 2017. MKL-DNN. (2017). [https://01.org/mkl-dnn](https://01.org/mkl-dnn) [↗](https://01.org/mkl-dnn)
117. S. Ioffe and C. Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proc. 32nd International Conference on Machine Learning (ICML’15). 448–456. [find](https://scholar.google.com/scholar?q=S%2E+Ioffe+and+C%2E+Szegedy%2E+2015%2E+Batch+Normalization%3A+Accelerating+Deep+Network+Training+by+Reducing+Internal+Covariate+Shift%2E+In+Proc%2E+32nd+International+Conference+on+Machine+Learning+%28ICML%E2%80%9915%29%2E+448%E2%80%934)
118. M. Jaderberg et al. 2017. Population Based Training of Neural Networks. (2017). arXiv:1711.09846 [find](https://scholar.google.com/scholar?q=M%2E+Jaderberg+et+al%2E+2017%2E+Population+Based+Training+of+Neural+Networks%2E+%282017%29%2E+arXiv%3A1711%2E09846)
119. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. 2014. [Caffe: Convolutional architecture for fast feature embedding](https://md.research.elamur.ai/free/en/papers/jia-2014-caffe-convolutional-architecture-fast-feature/). In Proc. ACM Int’l Conf. on Multimedia. 675–678.
120. J. Jiang, B. Cui, C. Zhang, and L. Yu. 2017. Heterogeneity-aware Distributed Parameter Servers. In Proc. 2017 ACM International Conference on Management of Data (SIGMOD ’17). 463–478. [find](https://scholar.google.com/scholar?q=J%2E+Jiang%2C+B%2E+Cui%2C+C%2E+Zhang%2C+and+L%2E+Yu%2E+2017%2E+Heterogeneity%2Daware+Distributed+Parameter+Servers%2E+In+Proc%2E+2017+ACM+International+Conference+on+Management+of+Data+%28SIGMOD+%E2%80%9917%29%2E+463%E2%80%93478%2E)
121. P. H. Jin, Q. Yuan, F. N. Iandola, and K. Keutzer. 2016. How to scale distributed deep learning? ML Systems Workshop at NIPS (2016). [find](https://scholar.google.com/scholar?q=P%2E+H%2E+Jin%2C+Q%2E+Yuan%2C+F%2E+N%2E+Iandola%2C+and+K%2E+Keutzer%2E+2016%2E+How+to+scale+distributed+deep+learning%3F+ML+Systems+Workshop+at+NIPS+%282016%29%2E)
122. M. Johnson et al. 2016. [Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://md.research.elamur.ai/free/en/papers/johnson-2017-google-s-multilingual-neural-machine/). (2016). arXiv:1611.04558
123. R. Johnson and T. Zhang. 2013. Accelerating Stochastic Gradient Descent using Predictive Variance Reduction. In Advances in Neural Information Processing Systems 26. 315–323. [find](https://scholar.google.com/scholar?q=R%2E+Johnson+and+T%2E+Zhang%2E+2013%2E+Accelerating+Stochastic+Gradient+Descent+using+Predictive+Variance+Reduction%2E+In+Advances+in+Neural+Information+Processing+Systems+26%2E+315%E2%80%93323%2E)
124. N. P. Jouppi et al. 2017. [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://md.research.elamur.ai/free/en/papers/jouppi-2017-datacenter-performance-analysis-tensor-processing/). In Proc. 44th Annual International Symposium on Computer Architecture (ISCA ’17). 1–12.
125. L. Kaiser, A. N. Gomez, N. Shazeer, A. Vaswani, N. Parmar, L. Jones, and J. Uszkoreit. 2017. One Model To Learn Them All. (2017). arXiv:1706.05137 [find](https://scholar.google.com/scholar?q=L%2E+Kaiser%2C+A%2E+N%2E+Gomez%2C+N%2E+Shazeer%2C+A%2E+Vaswani%2C+N%2E+Parmar%2C+L%2E+Jones%2C+and+J%2E+Uszkoreit%2E+2017%2E+One+Model+To+Learn+Them+All%2E+%282017%29%2E+arXiv%3A1706%2E05137)
126. T. Karras, T. Aila, S. Laine, and J. Lehtinen. 2017. Progressive Growing of GANs for Improved Quality, Stability, and Variation. (2017). arXiv:1710.10196 — [arXiv:1710.10196](https://arxiv.org/abs/1710.10196) [find](https://scholar.google.com/scholar?q=T%2E+Karras%2C+T%2E+Aila%2C+S%2E+Laine%2C+and+J%2E+Lehtinen%2E+2017%2E+Progressive+Growing+of+GANs+for+Improved+Quality%2C+Stability%2C+and+Variation%2E+%282017%29%2E+arXiv%3A1710%2E10196+%E2%80%94+%5BarXiv%3A1710%2E10196%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F171)
127. J. Keuper and F. Pfreundt. 2015. Asynchronous Parallel Stochastic Gradient Descent: A Numeric Core for Scalable Distributed Machine Learning Algorithms. In Proc. Workshop on MLHPC. 1:1–1:11. [find](https://scholar.google.com/scholar?q=J%2E+Keuper+and+F%2E+Pfreundt%2E+2015%2E+Asynchronous+Parallel+Stochastic+Gradient+Descent%3A+A+Numeric+Core+for+Scalable+Distributed+Machine+Learning+Algorithms%2E+In+Proc%2E+Workshop+on+MLHPC%2E+1%3A1%E2%80%931%3A11%2E)
128. H. Kim et al. 2016. DeepSpark: Spark-Based Deep Learning Supporting Asynchronous Updates and Caffe Compatibility. (2016). arXiv:1602.08191 [find](https://scholar.google.com/scholar?q=H%2E+Kim+et+al%2E+2016%2E+DeepSpark%3A+Spark%2DBased+Deep+Learning+Supporting+Asynchronous+Updates+and+Caffe+Compatibility%2E+%282016%29%2E+arXiv%3A1602%2E08191)
129. Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin. 2016. Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Y%2E%2DD%2E+Kim%2C+E%2E+Park%2C+S%2E+Yoo%2C+T%2E+Choi%2C+L%2E+Yang%2C+and+D%2E+Shin%2E+2016%2E+Compression+of+Deep+Convolutional+Neural+Networks+for+Fast+and+Low+Power+Mobile+Applications%2E+In+International+Conference+on+Learning+R)
130. D. P. Kingma and J. Ba. 2015. [Adam: A Method for Stochastic Optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In Proc. International Conference on Learning Representations (ICLR).
131. A. Klein, S. Falkner, J. T. Springenberg, and F. Hutter. 2016. Learning curve prediction with Bayesian neural networks. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=A%2E+Klein%2C+S%2E+Falkner%2C+J%2E+T%2E+Springenberg%2C+and+F%2E+Hutter%2E+2016%2E+Learning+curve+prediction+with+Bayesian+neural+networks%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
132. U. Köster et al. 2017. Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks. In Advances in Neural Information Processing Systems 30. 1740–1750. [find](https://scholar.google.com/scholar?q=U%2E+K%C3%B6ster+et+al%2E+2017%2E+Flexpoint%3A+An+Adaptive+Numerical+Format+for+Efficient+Training+of+Deep+Neural+Networks%2E+In+Advances+in+Neural+Information+Processing+Systems+30%2E+1740%E2%80%931750%2E)
133. S. Krishnan, Y. Xiao, and R. A. Saurous. 2018. Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=S%2E+Krishnan%2C+Y%2E+Xiao%2C+and+R%2E+A%2E+Saurous%2E+2018%2E+Neumann+Optimizer%3A+A+Practical+Optimization+Algorithm+for+Deep+Neural+Networks%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
134. Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. Master’s thesis. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2E+2009%2E+Learning+Multiple+Layers+of+Features+from+Tiny+Images%2E+Master%E2%80%99s+thesis%2E)
135. A. Krizhevsky. 2014. [One weird trick for parallelizing convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2014-one-weird-trick-parallelizing-convolutional/). (2014). arXiv:1404.5997
136. A. Krizhevsky, I. Sutskever, and G. Hinton. 2012. [ImageNet Classification with Deep Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In Advances in Neural Information Processing Systems 25. 1097–1105.
137. T. Kurth et al. 2017. Deep Learning at 15PF: Supervised and Semi-supervised Classification for Scientific Data. In Proc. Int’l Conf. for High Performance Computing, Networking, Storage and Analysis (SC ’17). 7:1–7:11. [find](https://scholar.google.com/scholar?q=T%2E+Kurth+et+al%2E+2017%2E+Deep+Learning+at+15PF%3A+Supervised+and+Semi%2Dsupervised+Classification+for+Scientific+Data%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis+%28S)
138. G. Lacey, G. W. Taylor, and S. Areibi. 2016. Deep Learning on FPGAs: Past, Present, and Future. (2016). arXiv:1602.04283 [find](https://scholar.google.com/scholar?q=G%2E+Lacey%2C+G%2E+W%2E+Taylor%2C+and+S%2E+Areibi%2E+2016%2E+Deep+Learning+on+FPGAs%3A+Past%2C+Present%2C+and+Future%2E+%282016%29%2E+arXiv%3A1602%2E04283)
139. L. Lamport, R. Shostak, and M. Pease. 1982. The Byzantine Generals Problem. ACM Trans. Program. Lang. Syst. 4, 3 (1982), 382–401. [find](https://scholar.google.com/scholar?q=L%2E+Lamport%2C+R%2E+Shostak%2C+and+M%2E+Pease%2E+1982%2E+The+Byzantine+Generals+Problem%2E+ACM+Trans%2E+Program%2E+Lang%2E+Syst%2E+4%2C+3+%281982%29%2C+382%E2%80%93401%2E)
140. A. Lavin and S. Gray. 2016. [Fast Algorithms for Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/lavin-2015-fast-algorithms-convolutional-neural-networks/). In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
141. Q. V. Le, J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, and A. Y. Ng. 2011. On Optimization Methods for Deep Learning. In Proc. 28th International Conference on Machine Learning (ICML’11). 265–272. [find](https://scholar.google.com/scholar?q=Q%2E+V%2E+Le%2C+J%2E+Ngiam%2C+A%2E+Coates%2C+A%2E+Lahiri%2C+B%2E+Prochnow%2C+and+A%2E+Y%2E+Ng%2E+2011%2E+On+Optimization+Methods+for+Deep+Learning%2E+In+Proc%2E+28th+International+Conference+on+Machine+Learning+%28ICML%E2%80%9911%29%2E+265%E2%80%93272%2E)
142. Q. V. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. S. Corrado, J. Dean, and A. Y. Ng. 2012. Building High-level Features Using Large Scale Unsupervised Learning. In Proc. 29th Int’l Conf. on Machine Learning (ICML’12). 507–514. [find](https://scholar.google.com/scholar?q=Q%2E+V%2E+Le%2C+M%2E+Ranzato%2C+R%2E+Monga%2C+M%2E+Devin%2C+K%2E+Chen%2C+G%2E+S%2E+Corrado%2C+J%2E+Dean%2C+and+A%2E+Y%2E+Ng%2E+2012%2E+Building+High%2Dlevel+Features+Using+Large+Scale+Unsupervised+Learning%2E+In+Proc%2E+29th+Int%E2%80%99l+Conf%2E+on+Machin)
143. Y. LeCun, Y. Bengio, and G. Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436–444. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+Y%2E+Bengio%2C+and+G%2E+Hinton%2E+2015%2E+Deep+learning%2E+Nature+521%2C+7553+%282015%29%2C+436%E2%80%93444%2E)
144. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. 1989. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Comput. 1, 4 (1989), 541–551. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+B%2E+Boser%2C+J%2E+S%2E+Denker%2C+D%2E+Henderson%2C+R%2E+E%2E+Howard%2C+W%2E+Hubbard%2C+and+L%2E+D%2E+Jackel%2E+1989%2E+Backpropagation+Applied+to+Handwritten+Zip+Code+Recognition%2E+Neural+Comput%2E+1%2C+4+%281989%29%2C+541%E2%80%93551%2E)
145. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+L%2E+Bottou%2C+Y%2E+Bengio%2C+and+P%2E+Haffner%2E+1998%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proc%2E+IEEE+86%2C+11+%281998%29%2C+2278%E2%80%932324%2E)
146. Y. LeCun and C. Cortes. 1998. The MNIST database of handwritten digits. (1998). [http://yann.lecun.com/exdb/mnist](http://yann.lecun.com/exdb/mnist) [↗](http://yann.lecun.com/exdb/mnist)
147. H. Lee, P. Pham, Y. Largman, and A. Y. Ng. 2009. Unsupervised feature learning for audio classification using convolutional deep belief networks. In Advances in Neural Information Processing Systems 22. 1096–1104. [find](https://scholar.google.com/scholar?q=H%2E+Lee%2C+P%2E+Pham%2C+Y%2E+Largman%2C+and+A%2E+Y%2E+Ng%2E+2009%2E+Unsupervised+feature+learning+for+audio+classification+using+convolutional+deep+belief+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+2)
148. S. Lee, S. Purushwalkam, M. Cogswell, D. J. Crandall, and D. Batra. 2015. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks. (2015). arXiv:1511.06314 — [arXiv:1511.06314](https://arxiv.org/abs/1511.06314) [find](https://scholar.google.com/scholar?q=S%2E+Lee%2C+S%2E+Purushwalkam%2C+M%2E+Cogswell%2C+D%2E+J%2E+Crandall%2C+and+D%2E+Batra%2E+2015%2E+Why+M+Heads+are+Better+than+One%3A+Training+a+Diverse+Ensemble+of+Deep+Networks%2E+%282015%29%2E+arXiv%3A1511%2E06314+%E2%80%94+%5BarXiv%3A1511%2E06314%5D%28h)
149. C. Li, Y. Yang, M. Feng, S. Chakradhar, and H. Zhou. 2016. Optimizing Memory Efficiency for Deep Convolutional Neural Networks on GPUs. In Proc. Int’l Conf. for Supercomputing (SC ’16). 54:1–54:12. [find](https://scholar.google.com/scholar?q=C%2E+Li%2C+Y%2E+Yang%2C+M%2E+Feng%2C+S%2E+Chakradhar%2C+and+H%2E+Zhou%2E+2016%2E+Optimizing+Memory+Efficiency+for+Deep+Convolutional+Neural+Networks+on+GPUs%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+for+Supercomputing+%28SC+%E2%80%9916%29%2E+54%3A1%E2%80%9354%3A12%2E)
150. D. Li, X. Wang, and D. Kong. 2017. DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices. (2017). arXiv:1708.04728 — [arXiv:1708.04728](https://arxiv.org/abs/1708.04728) [find](https://scholar.google.com/scholar?q=D%2E+Li%2C+X%2E+Wang%2C+and+D%2E+Kong%2E+2017%2E+DeepRebirth%3A+Accelerating+Deep+Neural+Network+Execution+on+Mobile+Devices%2E+%282017%29%2E+arXiv%3A1708%2E04728+%E2%80%94+%5BarXiv%3A1708%2E04728%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1708%2E04728%29)
151. F. Li and B. Liu. 2016. Ternary Weight Networks. (2016). arXiv:1605.04711 — [arXiv:1605.04711](https://arxiv.org/abs/1605.04711) [find](https://scholar.google.com/scholar?q=F%2E+Li+and+B%2E+Liu%2E+2016%2E+Ternary+Weight+Networks%2E+%282016%29%2E+arXiv%3A1605%2E04711+%E2%80%94+%5BarXiv%3A1605%2E04711%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1605%2E04711%29)
152. M. Li et al. 2014. Scaling Distributed Machine Learning with the Parameter Server. In Proc. 11th USENIX Conference on Operating Systems Design and Implementation (OSDI’14). 583–598. [find](https://scholar.google.com/scholar?q=M%2E+Li+et+al%2E+2014%2E+Scaling+Distributed+Machine+Learning+with+the+Parameter+Server%2E+In+Proc%2E+11th+USENIX+Conference+on+Operating+Systems+Design+and+Implementation+%28OSDI%E2%80%9914%29%2E+583%E2%80%93598%2E)
153. T. Li, J. Zhong, J. Liu, W. Wu, and C. Zhang. 2017. Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads. (2017). arXiv:1708.07308 — [arXiv:1708.07308](https://arxiv.org/abs/1708.07308) [find](https://scholar.google.com/scholar?q=T%2E+Li%2C+J%2E+Zhong%2C+J%2E+Liu%2C+W%2E+Wu%2C+and+C%2E+Zhang%2E+2017%2E+Ease%2Eml%3A+Towards+Multi%2Dtenant+Resource+Sharing+for+Machine+Learning+Workloads%2E+%282017%29%2E+arXiv%3A1708%2E07308+%E2%80%94+%5BarXiv%3A1708%2E07308%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1)
154. Y. Li. 2017. Deep Reinforcement Learning: An Overview. (2017). arXiv:1701.07274 — [arXiv:1701.07274](https://arxiv.org/abs/1701.07274) [find](https://scholar.google.com/scholar?q=Y%2E+Li%2E+2017%2E+Deep+Reinforcement+Learning%3A+An+Overview%2E+%282017%29%2E+arXiv%3A1701%2E07274+%E2%80%94+%5BarXiv%3A1701%2E07274%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1701%2E07274%29)
155. X. Lian et al. 2017. Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent. In Advances in Neural Information Processing Systems 30. 5336–5346. [find](https://scholar.google.com/scholar?q=X%2E+Lian+et+al%2E+2017%2E+Can+Decentralized+Algorithms+Outperform+Centralized+Algorithms%3F+A+Case+Study+for+Decentralized+Parallel+Stochastic+Gradient+Descent%2E+In+Advances+in+Neural+Information+Processing+S)
156. X. Lian, Y. Huang, Y. Li, and J. Liu. 2015. Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization. In Proc. 28th Int’l Conf. on NIPS - Volume 2. 2737–2745. [find](https://scholar.google.com/scholar?q=X%2E+Lian%2C+Y%2E+Huang%2C+Y%2E+Li%2C+and+J%2E+Liu%2E+2015%2E+Asynchronous+Parallel+Stochastic+Gradient+for+Nonconvex+Optimization%2E+In+Proc%2E+28th+Int%E2%80%99l+Conf%2E+on+NIPS+%2D+Volume+2%2E+2737%E2%80%932745%2E)
157. M. Lin, Q. Chen, and S. Yan. 2014. Network In Network. In Proc. Int’l Conf. on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=M%2E+Lin%2C+Q%2E+Chen%2C+and+S%2E+Yan%2E+2014%2E+Network+In+Network%2E+In+Proc%2E+Int%E2%80%99l+Conf%2E+on+Learning+Representations+%28ICLR%29%2E)
158. Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally. 2018. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. In Proc. International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Y%2E+Lin%2C+S%2E+Han%2C+H%2E+Mao%2C+Y%2E+Wang%2C+and+W%2E+J%2E+Dally%2E+2018%2E+Deep+Gradient+Compression%3A+Reducing+the+Communication+Bandwidth+for+Distributed+Training%2E+In+Proc%2E+International+Conference+on+Learning+Represen)
159. C. Liu, B. Zoph, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy. 2017. Progressive Neural Architecture Search. (2017). arXiv:1712.00559 [find](https://scholar.google.com/scholar?q=C%2E+Liu%2C+B%2E+Zoph%2C+J%2E+Shlens%2C+W%2E+Hua%2C+L%2E%2DJ%2E+Li%2C+L%2E+Fei%2DFei%2C+A%2E+Yuille%2C+J%2E+Huang%2C+and+K%2E+Murphy%2E+2017%2E+Progressive+Neural+Architecture+Search%2E+%282017%29%2E+arXiv%3A1712%2E00559)
160. H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu. 2018. Hierarchical Representations for Efficient Architecture Search. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+K%2E+Simonyan%2C+O%2E+Vinyals%2C+C%2E+Fernando%2C+and+K%2E+Kavukcuoglu%2E+2018%2E+Hierarchical+Representations+for+Efficient+Architecture+Search%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
161. H. Liu, K. Simonyan, and Y. Yang. 2018. DARTS: Differentiable Architecture Search. (2018). arXiv:1806.09055 [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+K%2E+Simonyan%2C+and+Y%2E+Yang%2E+2018%2E+DARTS%3A+Differentiable+Architecture+Search%2E+%282018%29%2E+arXiv%3A1806%2E09055)
162. X. Liu, J. Pool, S. Han, and W. J. Dally. 2018. Efficient Sparse-Winograd Convolutional Neural Networks. International Conference on Learning Representations (ICLR) (2018). [find](https://scholar.google.com/scholar?q=X%2E+Liu%2C+J%2E+Pool%2C+S%2E+Han%2C+and+W%2E+J%2E+Dally%2E+2018%2E+Efficient+Sparse%2DWinograd+Convolutional+Neural+Networks%2E+International+Conference+on+Learning+Representations+%28ICLR%29+%282018%29%2E)
163. P. R. Lorenzo, J. Nalepa, L. S. Ramos, and J. R. Pastor. 2017. Hyper-parameter Selection in Deep Neural Networks Using Parallel Particle Swarm Optimization. In Proc. Genetic and Evolutionary Computation Conf. (GECCO ’17). 1864–1871. [find](https://scholar.google.com/scholar?q=P%2E+R%2E+Lorenzo%2C+J%2E+Nalepa%2C+L%2E+S%2E+Ramos%2C+and+J%2E+R%2E+Pastor%2E+2017%2E+Hyper%2Dparameter+Selection+in+Deep+Neural+Networks+Using+Parallel+Particle+Swarm+Optimization%2E+In+Proc%2E+Genetic+and+Evolutionary+Computati)
164. I. Loshchilov and F. Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=I%2E+Loshchilov+and+F%2E+Hutter%2E+2017%2E+SGDR%3A+Stochastic+Gradient+Descent+with+Warm+Restarts%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
165. J. Martens. 2010. Deep Learning via Hessian-free Optimization. In Proc. 27th International Conference on Machine Learning (ICML’10). 735–742. [find](https://scholar.google.com/scholar?q=J%2E+Martens%2E+2010%2E+Deep+Learning+via+Hessian%2Dfree+Optimization%2E+In+Proc%2E+27th+International+Conference+on+Machine+Learning+%28ICML%E2%80%9910%29%2E+735%E2%80%93742%2E)
166. M. Mathieu, M. Henaff, and Y. LeCun. 2014. Fast Training of Convolutional Networks through FFTs. International Conference on Learning Representations (ICLR) (2014). [find](https://scholar.google.com/scholar?q=M%2E+Mathieu%2C+M%2E+Henaff%2C+and+Y%2E+LeCun%2E+2014%2E+Fast+Training+of+Convolutional+Networks+through+FFTs%2E+International+Conference+on+Learning+Representations+%28ICLR%29+%282014%29%2E)
167. Message Passing Interface Forum. 2015. MPI: A Message-Passing Interface Standard Version 3.1. (2015). [find](https://scholar.google.com/scholar?q=Message+Passing+Interface+Forum%2E+2015%2E+MPI%3A+A+Message%2DPassing+Interface+Standard+Version+3%2E1%2E+%282015%29%2E)
168. Y. Miao, H. Zhang, and F. Metze. 2014. Distributed learning of multilingual DNN feature extractors using GPUs. In INTERSPEECH, 15th Annual Conference of the International Speech Communication Association. 830–834. [find](https://scholar.google.com/scholar?q=Y%2E+Miao%2C+H%2E+Zhang%2C+and+F%2E+Metze%2E+2014%2E+Distributed+learning+of+multilingual+DNN+feature+extractors+using+GPUs%2E+In+INTERSPEECH%2C+15th+Annual+Conference+of+the+International+Speech+Communication+Associat)
169. R. Miikkulainen et al. 2017. Evolving Deep Neural Networks. (2017). arXiv:1703.00548 — [arXiv:1703.00548](https://arxiv.org/abs/1703.00548) [find](https://scholar.google.com/scholar?q=R%2E+Miikkulainen+et+al%2E+2017%2E+Evolving+Deep+Neural+Networks%2E+%282017%29%2E+arXiv%3A1703%2E00548+%E2%80%94+%5BarXiv%3A1703%2E00548%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1703%2E00548%29)
170. V. Mnih et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529–533. [find](https://scholar.google.com/scholar?q=V%2E+Mnih+et+al%2E+2015%2E+Human%2Dlevel+control+through+deep+reinforcement+learning%2E+Nature+518%2C+7540+%282015%29%2C+529%E2%80%93533%2E)
171. V. Mnih et al. 2016. Asynchronous Methods for Deep Reinforcement Learning. In Proc. 33rd International Conference on Machine Learning, Vol. 48. 1928–1937. [find](https://scholar.google.com/scholar?q=V%2E+Mnih+et+al%2E+2016%2E+Asynchronous+Methods+for+Deep+Reinforcement+Learning%2E+In+Proc%2E+33rd+International+Conference+on+Machine+Learning%2C+Vol%2E+48%2E+1928%E2%80%931937%2E)
172. P. Moritz, R. Nishihara, and M. Jordan. 2016. A Linearly-Convergent Stochastic L-BFGS Algorithm. In Proc. 19th International Conference on Artificial Intelligence and Statistics, Vol. 51. 249–258. [find](https://scholar.google.com/scholar?q=P%2E+Moritz%2C+R%2E+Nishihara%2C+and+M%2E+Jordan%2E+2016%2E+A+Linearly%2DConvergent+Stochastic+L%2DBFGS+Algorithm%2E+In+Proc%2E+19th+International+Conference+on+Artificial+Intelligence+and+Statistics%2C+Vol%2E+51%2E+249%E2%80%93258%2E)
173. P. Moritz, R. Nishihara, I. Stoica, and M. I. Jordan. 2016. SparkNet: Training Deep Networks in Spark. In Proc. International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=P%2E+Moritz%2C+R%2E+Nishihara%2C+I%2E+Stoica%2C+and+M%2E+I%2E+Jordan%2E+2016%2E+SparkNet%3A+Training+Deep+Networks+in+Spark%2E+In+Proc%2E+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
174. U. A. Muller and A. Gunzinger. 1994. Neural net simulation on parallel computers. In Neural Networks, IEEE International Conference on, Vol. 6. 3961–3966. [find](https://scholar.google.com/scholar?q=U%2E+A%2E+Muller+and+A%2E+Gunzinger%2E+1994%2E+Neural+net+simulation+on+parallel+computers%2E+In+Neural+Networks%2C+IEEE+International+Conference+on%2C+Vol%2E+6%2E+3961%E2%80%933966%2E)
175. M. M. Najafabadi, F. Villanustre, T. M. Khoshgoftaar, N. Seliya, R. Wald, and E. Muharemagic. 2015. Deep learning applications and challenges in big data analytics. Journal of Big Data 2, 1 (2015), 1. [find](https://scholar.google.com/scholar?q=M%2E+M%2E+Najafabadi%2C+F%2E+Villanustre%2C+T%2E+M%2E+Khoshgoftaar%2C+N%2E+Seliya%2C+R%2E+Wald%2C+and+E%2E+Muharemagic%2E+2015%2E+Deep+learning+applications+and+challenges+in+big+data+analytics%2E+Journal+of+Big+Data+2%2C+1+%282015%29%2C+1%2E)
176. R. Negrinho and G. Gordon. 2017. DeepArchitect: Automatically Designing and Training Deep Architectures. (2017). arXiv:1704.08792 — [arXiv:1704.08792](https://arxiv.org/abs/1704.08792) [find](https://scholar.google.com/scholar?q=R%2E+Negrinho+and+G%2E+Gordon%2E+2017%2E+DeepArchitect%3A+Automatically+Designing+and+Training+Deep+Architectures%2E+%282017%29%2E+arXiv%3A1704%2E08792+%E2%80%94+%5BarXiv%3A1704%2E08792%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1704%2E08792%29)
177. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. 2009. Robust Stochastic Approximation Approach to Stochastic Programming. SIAM Journal on Optimization 19, 4 (2009), 1574–1609. [find](https://scholar.google.com/scholar?q=A%2E+Nemirovski%2C+A%2E+Juditsky%2C+G%2E+Lan%2C+and+A%2E+Shapiro%2E+2009%2E+Robust+Stochastic+Approximation+Approach+to+Stochastic+Programming%2E+SIAM+Journal+on+Optimization+19%2C+4+%282009%29%2C+1574%E2%80%931609%2E)
178. Y. Nesterov. 1983. A method of solving a convex programming problem with convergence rate O (1/k 2 ) . Soviet Mathematics Doklady 269 (1983), 543–547. [find](https://scholar.google.com/scholar?q=Y%2E+Nesterov%2E+1983%2E+A+method+of+solving+a+convex+programming+problem+with+convergence+rate+O+%281%2Fk+2+%29+%2E+Soviet+Mathematics+Doklady+269+%281983%29%2C+543%E2%80%93547%2E)
179. Netlib. 2017. Basic Linear Algebra Subprograms (BLAS). (2017). [http://www.netlib.org/blas](http://www.netlib.org/blas) [↗](http://www.netlib.org/blas)
180. J. Ngiam, Z. Chen, D. Chia, P. W. Koh, Q. V. Le, and A. Y. Ng. 2010. Tiled convolutional neural networks. In Advances in Neural Information Processing Systems 23. 1279–1287. [find](https://scholar.google.com/scholar?q=J%2E+Ngiam%2C+Z%2E+Chen%2C+D%2E+Chia%2C+P%2E+W%2E+Koh%2C+Q%2E+V%2E+Le%2C+and+A%2E+Y%2E+Ng%2E+2010%2E+Tiled+convolutional+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+23%2E+1279%E2%80%931287%2E)
181. J. Nocedal and S. Wright. 2006. Numerical Optimization. [find](https://scholar.google.com/scholar?q=J%2E+Nocedal+and+S%2E+Wright%2E+2006%2E+Numerical+Optimization%2E)
182. C. Noel and S. Osindero. 2014. Dogwild!-Distributed Hogwild for CPU & GPU. In NIPS Workshop on Distributed Machine Learning and Matrix Computations. [find](https://scholar.google.com/scholar?q=C%2E+Noel+and+S%2E+Osindero%2E+2014%2E+Dogwild%21%2DDistributed+Hogwild+for+CPU+%26+GPU%2E+In+NIPS+Workshop+on+Distributed+Machine+Learning+and+Matrix+Computations%2E)
183. E. Nurvitadhi et al. 2017. Can FPGAs Beat GPUs in Accelerating Next-Generation Deep Neural Networks?. In Proc. ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA ’17). 5–14. [find](https://scholar.google.com/scholar?q=E%2E+Nurvitadhi+et+al%2E+2017%2E+Can+FPGAs+Beat+GPUs+in+Accelerating+Next%2DGeneration+Deep+Neural+Networks%3F%2E+In+Proc%2E+ACM%2FSIGDA+International+Symposium+on+Field%2DProgrammable+Gate+Arrays+%28FPGA+%E2%80%9917%29%2E+5%E2%80%9314%2E)
184. NVIDIA. 2017. CUBLAS Library Documentation. (2017). [http://docs.nvidia.com/cuda/cublas](http://docs.nvidia.com/cuda/cublas) [↗](http://docs.nvidia.com/cuda/cublas)
185. NVIDIA. 2017. Programming Tensor Cores in CUDA 9. (2017). [https://devblogs.nvidia.com/programming-tensor-cores-cuda-9](https://devblogs.nvidia.com/programming-tensor-cores-cuda-9) [↗](https://devblogs.nvidia.com/programming-tensor-cores-cuda-9)
186. C. Olah. 2015. Understanding LSTM Networks. (2015). [http://colah.github.io/posts/2015-08-Understanding-LSTMs](http://colah.github.io/posts/2015-08-Understanding-LSTMs) [↗](http://colah.github.io/posts/2015-08-Understanding-LSTMs)
187. Y. Oyama et al. 2016. Predicting statistics of asynchronous SGD parameters for a large-scale distributed deep learning system on GPU supercomputers. In IEEE International Conference on Big Data (Big Data). 66–75. [find](https://scholar.google.com/scholar?q=Y%2E+Oyama+et+al%2E+2016%2E+Predicting+statistics+of+asynchronous+SGD+parameters+for+a+large%2Dscale+distributed+deep+learning+system+on+GPU+supercomputers%2E+In+IEEE+International+Conference+on+Big+Data+%28Big+D)
188. Y. Oyama, T. Ben-Nun, T. Hoefler, and S. Matsuoka. 2018. Accelerating Deep Learning Frameworks with Micro-batches. In 2018 IEEE CLUSTER. [find](https://scholar.google.com/scholar?q=Y%2E+Oyama%2C+T%2E+Ben%2DNun%2C+T%2E+Hoefler%2C+and+S%2E+Matsuoka%2E+2018%2E+Accelerating+Deep+Learning+Frameworks+with+Micro%2Dbatches%2E+In+2018+IEEE+CLUSTER%2E)
189. PaddlePaddle. 2017. Elastic Deep Learning. (2017). [https://github.com/PaddlePaddle/cloud/tree/develop/doc/edl](https://github.com/PaddlePaddle/cloud/tree/develop/doc/edl) [↗](https://github.com/PaddlePaddle/cloud/tree/develop/doc/edl)
190. T. Paine, H. Jin, J. Yang, Z. Lin, and T. S. Huang. 2013. GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training. (2013). arXiv:1312.6186 — [arXiv:1312.6186](https://arxiv.org/abs/1312.6186) [find](https://scholar.google.com/scholar?q=T%2E+Paine%2C+H%2E+Jin%2C+J%2E+Yang%2C+Z%2E+Lin%2C+and+T%2E+S%2E+Huang%2E+2013%2E+GPU+Asynchronous+Stochastic+Gradient+Descent+to+Speed+Up+Neural+Network+Training%2E+%282013%29%2E+arXiv%3A1312%2E6186+%E2%80%94+%5BarXiv%3A1312%2E6186%5D%28https%3A%2F%2Farxiv%2Eor)
191. S. J. Pan and Q. Yang. 2010. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering 22, 10 (2010), 1345–1359. [find](https://scholar.google.com/scholar?q=S%2E+J%2E+Pan+and+Q%2E+Yang%2E+2010%2E+A+Survey+on+Transfer+Learning%2E+IEEE+Transactions+on+Knowledge+and+Data+Engineering+22%2C+10+%282010%29%2C+1345%E2%80%931359%2E)
192. F. Petroski Such et al. 2017. Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning. (2017). arXiv:1712.06567 — [arXiv:1712.06567](https://arxiv.org/abs/1712.06567) [find](https://scholar.google.com/scholar?q=F%2E+Petroski+Such+et+al%2E+2017%2E+Deep+Neuroevolution%3A+Genetic+Algorithms+Are+a+Competitive+Alternative+for+Training+Deep+Neural+Networks+for+Reinforcement+Learning%2E+%282017%29%2E+arXiv%3A1712%2E06567+%E2%80%94+%5BarXiv%3A1712)
193. H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean. 2018. [Efficient Neural Architecture Search via Parameter Sharing](https://md.research.elamur.ai/free/en/papers/pham-2018-efficient-neural-architecture-search-parameter/). (2018). arXiv:1802.03268
194. B. T. Polyak and A. B. Juditsky. 1992. Acceleration of Stochastic Approximation by Averaging. SIAM J. Control Optim. 30, 4 (1992), 838–855. [find](https://scholar.google.com/scholar?q=B%2E+T%2E+Polyak+and+A%2E+B%2E+Juditsky%2E+1992%2E+Acceleration+of+Stochastic+Approximation+by+Averaging%2E+SIAM+J%2E+Control+Optim%2E+30%2C+4+%281992%29%2C+838%E2%80%93855%2E)
195. D. Povey, X. Zhang, and S. Khudanpur. 2014. Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging. (2014). arXiv:1410.7455 — [arXiv:1410.7455](https://arxiv.org/abs/1410.7455) [find](https://scholar.google.com/scholar?q=D%2E+Povey%2C+X%2E+Zhang%2C+and+S%2E+Khudanpur%2E+2014%2E+Parallel+training+of+Deep+Neural+Networks+with+Natural+Gradient+and+Parameter+Averaging%2E+%282014%29%2E+arXiv%3A1410%2E7455+%E2%80%94+%5BarXiv%3A1410%2E7455%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1)
196. H. Qi, E. R. Sparks, and A. Talwalkar. 2017. Paleo: A Performance Model for Deep Neural Networks. In Proc. International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=H%2E+Qi%2C+E%2E+R%2E+Sparks%2C+and+A%2E+Talwalkar%2E+2017%2E+Paleo%3A+A+Performance+Model+for+Deep+Neural+Networks%2E+In+Proc%2E+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
197. N. Qian. 1999. On the momentum term in gradient descent learning algorithms. Neural Networks 12, 1 (1999). [find](https://scholar.google.com/scholar?q=N%2E+Qian%2E+1999%2E+On+the+momentum+term+in+gradient+descent+learning+algorithms%2E+Neural+Networks+12%2C+1+%281999%29%2E)
198. R. Rabenseifner. 2004. Optimization of collective reduction operations. In Int’l Conf. on Computational Science. 1–9. [find](https://scholar.google.com/scholar?q=R%2E+Rabenseifner%2E+2004%2E+Optimization+of+collective+reduction+operations%2E+In+Int%E2%80%99l+Conf%2E+on+Computational+Science%2E+1%E2%80%939%2E)
199. A. Rahimi and B. Recht. 2017. Reflections on Random Kitchen Sinks. [http://www.argmin.net/2017/12/05/kitchen-sinks](http://www.argmin.net/2017/12/05/kitchen-sinks) NIPS Test of Time Award Talk. [↗](http://www.argmin.net/2017/12/05/kitchen-sinks)
200. R. Raina, A. Madhavan, and A. Y. Ng. 2009. Large-scale Deep Unsupervised Learning Using Graphics Processors. In Proc. 26th Annual International Conference on Machine Learning (ICML ’09). 873–880. [find](https://scholar.google.com/scholar?q=R%2E+Raina%2C+A%2E+Madhavan%2C+and+A%2E+Y%2E+Ng%2E+2009%2E+Large%2Dscale+Deep+Unsupervised+Learning+Using+Graphics+Processors%2E+In+Proc%2E+26th+Annual+International+Conference+on+Machine+Learning+%28ICML+%E2%80%9909%29%2E+873%E2%80%93880%2E)
201. S. Sundhar Ram, A. Nedic, and V. V. Veeravalli. 2009. Asynchronous gossip algorithms for stochastic optimization. In International Conference on Game Theory for Networks. 80–81. [find](https://scholar.google.com/scholar?q=S%2E+Sundhar+Ram%2C+A%2E+Nedic%2C+and+V%2E+V%2E+Veeravalli%2E+2009%2E+Asynchronous+gossip+algorithms+for+stochastic+optimization%2E+In+International+Conference+on+Game+Theory+for+Networks%2E+80%E2%80%9381%2E)
202. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. 2016. [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/rastegari-2016-xnor-net-imagenet-classification-using/). (2016). arXiv:1603.05279
203. E. Real, A. Aggarwal, Y. Huang, and Q. V Le. 2018. Regularized Evolution for Image Classifier Architecture Search. (2018). arXiv:1802.01548 [find](https://scholar.google.com/scholar?q=E%2E+Real%2C+A%2E+Aggarwal%2C+Y%2E+Huang%2C+and+Q%2E+V+Le%2E+2018%2E+Regularized+Evolution+for+Image+Classifier+Architecture+Search%2E+%282018%29%2E+arXiv%3A1802%2E01548)
204. E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin. 2017. Large-Scale Evolution of Image Classifiers. In Proc. 34th International Conference on Machine Learning. 2902–2911. [find](https://scholar.google.com/scholar?q=E%2E+Real%2C+S%2E+Moore%2C+A%2E+Selle%2C+S%2E+Saxena%2C+Y%2E+L%2E+Suematsu%2C+J%2E+Tan%2C+Q%2E+V%2E+Le%2C+and+A%2E+Kurakin%2E+2017%2E+Large%2DScale+Evolution+of+Image+Classifiers%2E+In+Proc%2E+34th+International+Conference+on+Machine+Learning%2E+)
205. B. Recht, C. Re, S. Wright, and F. Niu. 2011. Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent. In Advances in Neural Information Processing Systems 24. 693–701. [find](https://scholar.google.com/scholar?q=B%2E+Recht%2C+C%2E+Re%2C+S%2E+Wright%2C+and+F%2E+Niu%2E+2011%2E+Hogwild%3A+A+Lock%2DFree+Approach+to+Parallelizing+Stochastic+Gradient+Descent%2E+In+Advances+in+Neural+Information+Processing+Systems+24%2E+693%E2%80%93701%2E)
206. C. Renggli, D. Alistarh, and T. Hoefler. 2018. SparCML: High-Performance Sparse Communication for Machine Learning. (2018). arXiv:1802.08021 — [arXiv:1802.08021](https://arxiv.org/abs/1802.08021) [find](https://scholar.google.com/scholar?q=C%2E+Renggli%2C+D%2E+Alistarh%2C+and+T%2E+Hoefler%2E+2018%2E+SparCML%3A+High%2DPerformance+Sparse+Communication+for+Machine+Learning%2E+%282018%29%2E+arXiv%3A1802%2E08021+%E2%80%94+%5BarXiv%3A1802%2E08021%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1802%2E08021%29)
207. Herbert Robbins and Sutton Monro. 1951. A Stochastic Approximation Method. The Annals of Mathematical Statistics 22, 3 (1951), 400–407. [find](https://scholar.google.com/scholar?q=Herbert+Robbins+and+Sutton+Monro%2E+1951%2E+A+Stochastic+Approximation+Method%2E+The+Annals+of+Mathematical+Statistics+22%2C+3+%281951%29%2C+400%E2%80%93407%2E)
208. T. Salimans and D. P. Kingma. 2016. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In Advances in Neural Information Processing Systems 29. 901–909. [find](https://scholar.google.com/scholar?q=T%2E+Salimans+and+D%2E+P%2E+Kingma%2E+2016%2E+Weight+Normalization%3A+A+Simple+Reparameterization+to+Accelerate+Training+of+Deep+Neural+Networks%2E+In+Advances+in+Neural+Information+Processing+Systems+29%2E+901%E2%80%93909%2E)
209. J. Schmidhuber. 2015. Deep learning in neural networks: An overview. Neural Networks 61 (2015), 85–117. [find](https://scholar.google.com/scholar?q=J%2E+Schmidhuber%2E+2015%2E+Deep+learning+in+neural+networks%3A+An+overview%2E+Neural+Networks+61+%282015%29%2C+85%E2%80%93117%2E)
210. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 2014. 1-Bit Stochastic Gradient Descent and Application to Data-Parallel Distributed Training of Speech DNNs. In Interspeech. [find](https://scholar.google.com/scholar?q=F%2E+Seide%2C+H%2E+Fu%2C+J%2E+Droppo%2C+G%2E+Li%2C+and+D%2E+Yu%2E+2014%2E+1%2DBit+Stochastic+Gradient+Descent+and+Application+to+Data%2DParallel+Distributed+Training+of+Speech+DNNs%2E+In+Interspeech%2E)
211. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 2014. On parallelizability of stochastic gradient descent for speech DNNs. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 235–239. [find](https://scholar.google.com/scholar?q=F%2E+Seide%2C+H%2E+Fu%2C+J%2E+Droppo%2C+G%2E+Li%2C+and+D%2E+Yu%2E+2014%2E+On+parallelizability+of+stochastic+gradient+descent+for+speech+DNNs%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28IC)
212. S. Shalev-Shwartz and S. Ben-David. 2014. Understanding machine learning: From theory to algorithms. Cambridge university press. [find](https://scholar.google.com/scholar?q=S%2E+Shalev%2DShwartz+and+S%2E+Ben%2DDavid%2E+2014%2E+Understanding+machine+learning%3A+From+theory+to+algorithms%2E+Cambridge+university+press%2E)
213. O. Shamir. 2016. Without-Replacement Sampling for Stochastic Gradient Methods. In Advances in Neural Information Processing Systems 29. 46–54. [find](https://scholar.google.com/scholar?q=O%2E+Shamir%2E+2016%2E+Without%2DReplacement+Sampling+for+Stochastic+Gradient+Methods%2E+In+Advances+in+Neural+Information+Processing+Systems+29%2E+46%E2%80%9354%2E)
214. R. Shokri and V. Shmatikov. 2015. Privacy-Preserving Deep Learning. In Proc. 22Nd ACM SIGSAC Conference on Computer and Communications Security (CCS ’15). 1310–1321. [find](https://scholar.google.com/scholar?q=R%2E+Shokri+and+V%2E+Shmatikov%2E+2015%2E+Privacy%2DPreserving+Deep+Learning%2E+In+Proc%2E+22Nd+ACM+SIGSAC+Conference+on+Computer+and+Communications+Security+%28CCS+%E2%80%9915%29%2E+1310%E2%80%931321%2E)
215. D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. 2017. Mastering the game of go without human knowledge. Nature 550, 7676 (2017), 354. [find](https://scholar.google.com/scholar?q=D%2E+Silver%2C+J%2E+Schrittwieser%2C+K%2E+Simonyan%2C+I%2E+Antonoglou%2C+A%2E+Huang%2C+A%2E+Guez%2C+T%2E+Hubert%2C+L%2E+Baker%2C+M%2E+Lai%2C+A%2E+Bolton%2C+et+al%2E+2017%2E+Mastering+the+game+of+go+without+human+knowledge%2E+Nature+550%2C+7676+%28201)
216. K. Simonyan and A. Zisserman. 2015. [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). Int’l Conf. Learning Representations (ICLR).
217. A. J. R. Simpson. 2015. Instant Learning: Parallel Deep Neural Networks and Convolutional Bootstrapping. (2015). arXiv:1505.05972 — [arXiv:1505.05972](https://arxiv.org/abs/1505.05972) [find](https://scholar.google.com/scholar?q=A%2E+J%2E+R%2E+Simpson%2E+2015%2E+Instant+Learning%3A+Parallel+Deep+Neural+Networks+and+Convolutional+Bootstrapping%2E+%282015%29%2E+arXiv%3A1505%2E05972+%E2%80%94+%5BarXiv%3A1505%2E05972%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1505%2E05972%29)
218. S. L. Smith, P. Kindermans, and Q. V. Le. 2017. Don’t Decay the Learning Rate, Increase the Batch Size. (2017). arXiv:1711.00489 [find](https://scholar.google.com/scholar?q=S%2E+L%2E+Smith%2C+P%2E+Kindermans%2C+and+Q%2E+V%2E+Le%2E+2017%2E+Don%E2%80%99t+Decay+the+Learning+Rate%2C+Increase+the+Batch+Size%2E+%282017%29%2E+arXiv%3A1711%2E00489)
219. J. Snoek, H. Larochelle, and R. P Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms. In Advances in Neural Information Processing Systems 25. 2951–2959. [find](https://scholar.google.com/scholar?q=J%2E+Snoek%2C+H%2E+Larochelle%2C+and+R%2E+P+Adams%2E+2012%2E+Practical+Bayesian+Optimization+of+Machine+Learning+Algorithms%2E+In+Advances+in+Neural+Information+Processing+Systems+25%2E+2951%E2%80%932959%2E)
220. E. Solomonik and T. Hoefler. 2015. Sparse Tensor Algebra as a Parallel Programming Model. (2015). arXiv:1512.00066 — [arXiv:1512.00066](https://arxiv.org/abs/1512.00066) [find](https://scholar.google.com/scholar?q=E%2E+Solomonik+and+T%2E+Hoefler%2E+2015%2E+Sparse+Tensor+Algebra+as+a+Parallel+Programming+Model%2E+%282015%29%2E+arXiv%3A1512%2E00066+%E2%80%94+%5BarXiv%3A1512%2E00066%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1512%2E00066%29)
221. M. Song, Y. Hu, H. Chen, and T. Li. 2017. Towards Pervasive and User Satisfactory CNN across GPU Microarchitectures. In IEEE International Symposium on High Performance Computer Architecture (HPCA). 1–12. [find](https://scholar.google.com/scholar?q=M%2E+Song%2C+Y%2E+Hu%2C+H%2E+Chen%2C+and+T%2E+Li%2E+2017%2E+Towards+Pervasive+and+User+Satisfactory+CNN+across+GPU+Microarchitectures%2E+In+IEEE+International+Symposium+on+High+Performance+Computer+Architecture+%28HPCA%29%2E+1)
222. H. V. Sorensen and C. S. Burrus. 1993. Efficient computation of the DFT with only a subset of input or output points. IEEE Transactions on Signal Processing 41, 3 (1993), 1184–1200. [find](https://scholar.google.com/scholar?q=H%2E+V%2E+Sorensen+and+C%2E+S%2E+Burrus%2E+1993%2E+Efficient+computation+of+the+DFT+with+only+a+subset+of+input+or+output+points%2E+IEEE+Transactions+on+Signal+Processing+41%2C+3+%281993%29%2C+1184%E2%80%931200%2E)
223. V. Strassen. 1969. Gaussian Elimination is Not Optimal. Numer. Math. 13, 4 (1969), 354–356. [find](https://scholar.google.com/scholar?q=V%2E+Strassen%2E+1969%2E+Gaussian+Elimination+is+Not+Optimal%2E+Numer%2E+Math%2E+13%2C+4+%281969%29%2C+354%E2%80%93356%2E)
224. N. Strom. 2015. Scalable distributed DNN training using commodity GPU cloud computing. In Sixteenth Annual Conference of the International Speech Communication Association. [find](https://scholar.google.com/scholar?q=N%2E+Strom%2E+2015%2E+Scalable+distributed+DNN+training+using+commodity+GPU+cloud+computing%2E+In+Sixteenth+Annual+Conference+of+the+International+Speech+Communication+Association%2E)
225. V. Sze, Y. H. Chen, T. J. Yang, and J. S. Emer. 2017. [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://md.research.elamur.ai/free/en/papers/sze-2017-efficient-processing-deep-neural-networks/). Proc. IEEE 105, 12 (2017), 2295–2329.
226. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. Going Deeper with Convolutions. In Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=C%2E+Szegedy%2C+W%2E+Liu%2C+Y%2E+Jia%2C+P%2E+Sermanet%2C+S%2E+Reed%2C+D%2E+Anguelov%2C+D%2E+Erhan%2C+V%2E+Vanhoucke%2C+and+A%2E+Rabinovich%2E+2015%2E+Going+Deeper+with+Convolutions%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2E)
227. G. Taylor, R. Burmeister, Z. Xu, B. Singh, A. Patel, and T. Goldstein. 2016. Training Neural Networks Without Gradients: A Scalable ADMM Approach. (2016). arXiv:1605.02026 [find](https://scholar.google.com/scholar?q=G%2E+Taylor%2C+R%2E+Burmeister%2C+Z%2E+Xu%2C+B%2E+Singh%2C+A%2E+Patel%2C+and+T%2E+Goldstein%2E+2016%2E+Training+Neural+Networks+Without+Gradients%3A+A+Scalable+ADMM+Approach%2E+%282016%29%2E+arXiv%3A1605%2E02026)
228. L. Truong et al. 2016. Latte: A Language, Compiler, and Runtime for Elegant and Efficient Deep Neural Networks. In Proc. 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’16). 209–223. [find](https://scholar.google.com/scholar?q=L%2E+Truong+et+al%2E+2016%2E+Latte%3A+A+Language%2C+Compiler%2C+and+Runtime+for+Elegant+and+Efficient+Deep+Neural+Networks%2E+In+Proc%2E+37th+ACM+SIGPLAN+Conference+on+Programming+Language+Design+and+Implementation+%28)
229. J. Tsitsiklis, D. Bertsekas, and M. Athans. 1986. Distributed asynchronous deterministic and stochastic gradient optimization algorithms. IEEE Trans. Automat. Control 31, 9 (1986), 803–812. [find](https://scholar.google.com/scholar?q=J%2E+Tsitsiklis%2C+D%2E+Bertsekas%2C+and+M%2E+Athans%2E+1986%2E+Distributed+asynchronous+deterministic+and+stochastic+gradient+optimization+algorithms%2E+IEEE+Trans%2E+Automat%2E+Control+31%2C+9+%281986%29%2C+803%E2%80%93812%2E)
230. V. Vanhoucke, A. Senior, and M. Z. Mao. 2011. Improving the speed of neural networks on CPUs. In Deep Learning and Unsupervised Feature Learning Workshop, NIPS. [find](https://scholar.google.com/scholar?q=V%2E+Vanhoucke%2C+A%2E+Senior%2C+and+M%2E+Z%2E+Mao%2E+2011%2E+Improving+the+speed+of+neural+networks+on+CPUs%2E+In+Deep+Learning+and+Unsupervised+Feature+Learning+Workshop%2C+NIPS%2E)
231. N. Vasilache et al. 2018. [Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions](https://md.research.elamur.ai/free/en/papers/vasilache-2018-tensor-comprehensions-framework-agnostic-high/). (2018). arXiv:1802.04730
232. N. Vasilache, J. Johnson, M. Mathieu, S. Chintala, S. Piantino, and Y. LeCun. 2015. Fast Convolutional Nets With fbfft: A GPU Performance Evaluation. International Conference on Learning Representations (ICLR) (2015). [find](https://scholar.google.com/scholar?q=N%2E+Vasilache%2C+J%2E+Johnson%2C+M%2E+Mathieu%2C+S%2E+Chintala%2C+S%2E+Piantino%2C+and+Y%2E+LeCun%2E+2015%2E+Fast+Convolutional+Nets+With+fbfft%3A+A+GPU+Performance+Evaluation%2E+International+Conference+on+Learning+Representatio)
233. A. Vasudevan, A. Anderson, and D. Gregg. 2017. Parallel Multi Channel Convolution using General Matrix Multiplication. (2017). arXiv:1704.04428 — [arXiv:1704.04428](https://arxiv.org/abs/1704.04428) [find](https://scholar.google.com/scholar?q=A%2E+Vasudevan%2C+A%2E+Anderson%2C+and+D%2E+Gregg%2E+2017%2E+Parallel+Multi+Channel+Convolution+using+General+Matrix+Multiplication%2E+%282017%29%2E+arXiv%3A1704%2E04428+%E2%80%94+%5BarXiv%3A1704%2E04428%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1704%2E04428%29)
234. P. Verbancsics and J. Harguess. 2015. Image Classification Using Generative Neuro Evolution for Deep Learning. In IEEE Winter Conference on Applications of Computer Vision. 488–493. [find](https://scholar.google.com/scholar?q=P%2E+Verbancsics+and+J%2E+Harguess%2E+2015%2E+Image+Classification+Using+Generative+Neuro+Evolution+for+Deep+Learning%2E+In+IEEE+Winter+Conference+on+Applications+of+Computer+Vision%2E+488%E2%80%93493%2E)
235. A. Viebke, S. Memeti, S. Pllana, and A. Abraham. 2017. CHAOS: a parallelization scheme for training convolutional neural networks on Intel Xeon Phi. The Journal of Supercomputing (2017). [find](https://scholar.google.com/scholar?q=A%2E+Viebke%2C+S%2E+Memeti%2C+S%2E+Pllana%2C+and+A%2E+Abraham%2E+2017%2E+CHAOS%3A+a+parallelization+scheme+for+training+convolutional+neural+networks+on+Intel+Xeon+Phi%2E+The+Journal+of+Supercomputing+%282017%29%2E)
236. H. Wang, B. Raj, and E. P. Xing. 2017. On the Origin of Deep Learning. (2017). arXiv:1702.07800 — [arXiv:1702.07800](https://arxiv.org/abs/1702.07800) [find](https://scholar.google.com/scholar?q=H%2E+Wang%2C+B%2E+Raj%2C+and+E%2E+P%2E+Xing%2E+2017%2E+On+the+Origin+of+Deep+Learning%2E+%282017%29%2E+arXiv%3A1702%2E07800+%E2%80%94+%5BarXiv%3A1702%2E07800%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1702%2E07800%29)
237. W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. 2017. TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning. In Advances in Neural Information Processing Systems 30. 1509–1519. [find](https://scholar.google.com/scholar?q=W%2E+Wen%2C+C%2E+Xu%2C+F%2E+Yan%2C+C%2E+Wu%2C+Y%2E+Wang%2C+Y%2E+Chen%2C+and+H%2E+Li%2E+2017%2E+TernGrad%3A+Ternary+Gradients+to+Reduce+Communication+in+Distributed+Deep+Learning%2E+In+Advances+in+Neural+Information+Processing+Systems+)
238. P. J. Werbos. 1990. Backpropagation through time: what it does and how to do it. Proc. IEEE 78, 10 (1990), 1550–1560. [find](https://scholar.google.com/scholar?q=P%2E+J%2E+Werbos%2E+1990%2E+Backpropagation+through+time%3A+what+it+does+and+how+to+do+it%2E+Proc%2E+IEEE+78%2C+10+%281990%29%2C+1550%E2%80%931560%2E)
239. J. H. Wilkinson. 1994. Rounding Errors in Algebraic Processes. Dover Publications, Inc. [find](https://scholar.google.com/scholar?q=J%2E+H%2E+Wilkinson%2E+1994%2E+Rounding+Errors+in+Algebraic+Processes%2E+Dover+Publications%2C+Inc%2E)
240. R. J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8, 3 (1992), 229–256. [find](https://scholar.google.com/scholar?q=R%2E+J%2E+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+Learning+8%2C+3+%281992%29%2C+229%E2%80%93256%2E)
241. S. Winograd. 1980. Arithmetic Complexity of Computations. Society for Industrial and Applied Mathematics. [find](https://scholar.google.com/scholar?q=S%2E+Winograd%2E+1980%2E+Arithmetic+Complexity+of+Computations%2E+Society+for+Industrial+and+Applied+Mathematics%2E)
242. L. Xie and A. Yuille. 2017. Genetic CNN. In IEEE International Conference on Computer Vision (ICCV). 1388–1397. [find](https://scholar.google.com/scholar?q=L%2E+Xie+and+A%2E+Yuille%2E+2017%2E+Genetic+CNN%2E+In+IEEE+International+Conference+on+Computer+Vision+%28ICCV%29%2E+1388%E2%80%931397%2E)
243. P. Xie, J. K. Kim, Y. Zhou, Q. Ho, A. Kumar, Y. Yu, and E. Xing. 2016. Lighter-communication Distributed Machine Learning via Sufficient Factor Broadcasting. In Proc. 32nd Conf. on Uncertainty in Artificial Intelligence (UAI’16). 795–804. [find](https://scholar.google.com/scholar?q=P%2E+Xie%2C+J%2E+K%2E+Kim%2C+Y%2E+Zhou%2C+Q%2E+Ho%2C+A%2E+Kumar%2C+Y%2E+Yu%2C+and+E%2E+Xing%2E+2016%2E+Lighter%2Dcommunication+Distributed+Machine+Learning+via+Sufficient+Factor+Broadcasting%2E+In+Proc%2E+32nd+Conf%2E+on+Uncertainty+in+Arti)
244. E. P. Xing, Q. Ho, W. Dai, J. K. Kim, J. Wei, S. Lee, X. Zheng, P. Xie, A. Kumar, and Y. Yu. 2015. Petuum: A New Platform for Distributed Machine Learning on Big Data. IEEE Transactions on Big Data 1, 2 (2015), 49–67. [find](https://scholar.google.com/scholar?q=E%2E+P%2E+Xing%2C+Q%2E+Ho%2C+W%2E+Dai%2C+J%2E+K%2E+Kim%2C+J%2E+Wei%2C+S%2E+Lee%2C+X%2E+Zheng%2C+P%2E+Xie%2C+A%2E+Kumar%2C+and+Y%2E+Yu%2E+2015%2E+Petuum%3A+A+New+Platform+for+Distributed+Machine+Learning+on+Big+Data%2E+IEEE+Transactions+on+Big+Data+1%2C)
245. K. Xu et al. 2015. [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). (2015). arXiv:1502.03044
246. O. Yadan, K. Adams, Y. Taigman, and M. Ranzato. 2013. Multi-GPU Training of ConvNets. (2013). arXiv:1312.5853 — [arXiv:1312.5853](https://arxiv.org/abs/1312.5853) [find](https://scholar.google.com/scholar?q=O%2E+Yadan%2C+K%2E+Adams%2C+Y%2E+Taigman%2C+and+M%2E+Ranzato%2E+2013%2E+Multi%2DGPU+Training+of+ConvNets%2E+%282013%29%2E+arXiv%3A1312%2E5853+%E2%80%94+%5BarXiv%3A1312%2E5853%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1312%2E5853%29)
247. F. Yan, O. Ruwase, Y. He, and T. Chilimbi. 2015. Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems. In Proc. 21st ACM Int’l Conf. on Knowledge Discovery and Data Mining (KDD ’15). 1355–1364. [find](https://scholar.google.com/scholar?q=F%2E+Yan%2C+O%2E+Ruwase%2C+Y%2E+He%2C+and+T%2E+Chilimbi%2E+2015%2E+Performance+Modeling+and+Scalability+Optimization+of+Distributed+Deep+Learning+Systems%2E+In+Proc%2E+21st+ACM+Int%E2%80%99l+Conf%2E+on+Knowledge+Discovery+and+Data+M)
248. Y. You, A. Buluç, and J. Demmel. 2017. Scaling Deep Learning on GPU and Knights Landing Clusters. In Proc. International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’17). 9:1–9:12. [find](https://scholar.google.com/scholar?q=Y%2E+You%2C+A%2E+Bulu%C3%A7%2C+and+J%2E+Demmel%2E+2017%2E+Scaling+Deep+Learning+on+GPU+and+Knights+Landing+Clusters%2E+In+Proc%2E+International+Conference+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis+%28SC)
249. Y. You, I. Gitman, and B. Ginsburg. 2017. Large Batch Training of Convolutional Networks. (2017). arXiv:1708.03888 — [arXiv:1708.03888](https://arxiv.org/abs/1708.03888) [find](https://scholar.google.com/scholar?q=Y%2E+You%2C+I%2E+Gitman%2C+and+B%2E+Ginsburg%2E+2017%2E+Large+Batch+Training+of+Convolutional+Networks%2E+%282017%29%2E+arXiv%3A1708%2E03888+%E2%80%94+%5BarXiv%3A1708%2E03888%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1708%2E03888%29)
250. Y. You, Z. Zhang, C. Hsieh, and J. Demmel. 2017. [100-epoch ImageNet Training with AlexNet in 24 Minutes](https://md.research.elamur.ai/free/en/papers/you-2017-imagenet-training-minutes/). (2017). arXiv:1709.05011
251. S. R. Young et al. 2017. Evolving Deep Networks Using HPC. In Proc. Workshop on Machine Learning in HPC Environments (MLHPC’17). 7:1–7:7. [find](https://scholar.google.com/scholar?q=S%2E+R%2E+Young+et+al%2E+2017%2E+Evolving+Deep+Networks+Using+HPC%2E+In+Proc%2E+Workshop+on+Machine+Learning+in+HPC+Environments+%28MLHPC%E2%80%9917%29%2E+7%3A1%E2%80%937%3A7%2E)
252. F. Yu and V. Koltun. 2016. Multi-Scale Context Aggregation by Dilated Convolutions. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=F%2E+Yu+and+V%2E+Koltun%2E+2016%2E+Multi%2DScale+Context+Aggregation+by+Dilated+Convolutions%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
253. Y. Yu, J. Jiang, and X. Chi. 2016. Using Supercomputer to Speed up Neural Network Training. In IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS). 942–947. [find](https://scholar.google.com/scholar?q=Y%2E+Yu%2C+J%2E+Jiang%2C+and+X%2E+Chi%2E+2016%2E+Using+Supercomputer+to+Speed+up+Neural+Network+Training%2E+In+IEEE+22nd+International+Conference+on+Parallel+and+Distributed+Systems+%28ICPADS%29%2E+942%E2%80%93947%2E)
254. H. Zhang, Z. Hu, J. Wei, P. Xie, G. Kim, Q. Ho, and E. P. Xing. 2015. Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines. (2015). arXiv:1512.06216 [find](https://scholar.google.com/scholar?q=H%2E+Zhang%2C+Z%2E+Hu%2C+J%2E+Wei%2C+P%2E+Xie%2C+G%2E+Kim%2C+Q%2E+Ho%2C+and+E%2E+P%2E+Xing%2E+2015%2E+Poseidon%3A+A+System+Architecture+for+Efficient+GPU%2Dbased+Deep+Learning+on+Multiple+Machines%2E+%282015%29%2E+arXiv%3A1512%2E06216)
255. H. Zhang, Z. Zheng, S. Xu, W. Dai, Q. Ho, X. Liang, Z. Hu, J. Wei, P. Xie, and E. P. Xing. 2017. Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters. In USENIX ATC. 181–193. [find](https://scholar.google.com/scholar?q=H%2E+Zhang%2C+Z%2E+Zheng%2C+S%2E+Xu%2C+W%2E+Dai%2C+Q%2E+Ho%2C+X%2E+Liang%2C+Z%2E+Hu%2C+J%2E+Wei%2C+P%2E+Xie%2C+and+E%2E+P%2E+Xing%2E+2017%2E+Poseidon%3A+An+Efficient+Communication+Architecture+for+Distributed+Deep+Learning+on+GPU+Clusters%2E+In+USE)
256. J. Zhang, I. Mitliagkas, and C. Ré. 2017. YellowFin and the Art of Momentum Tuning. (2017). arXiv:1706.03471 — [arXiv:1706.03471](https://arxiv.org/abs/1706.03471) [find](https://scholar.google.com/scholar?q=J%2E+Zhang%2C+I%2E+Mitliagkas%2C+and+C%2E+R%C3%A9%2E+2017%2E+YellowFin+and+the+Art+of+Momentum+Tuning%2E+%282017%29%2E+arXiv%3A1706%2E03471+%E2%80%94+%5BarXiv%3A1706%2E03471%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1706%2E03471%29)
257. K. Zhang and X. W. Chen. 2014. Large-Scale Deep Belief Nets With MapReduce. IEEE Access 2 (2014), 395–403. [find](https://scholar.google.com/scholar?q=K%2E+Zhang+and+X%2E+W%2E+Chen%2E+2014%2E+Large%2DScale+Deep+Belief+Nets+With+MapReduce%2E+IEEE+Access+2+%282014%29%2C+395%E2%80%93403%2E)
258. S. Zhang, A. Choromanska, and Y. LeCun. 2015. Deep Learning with Elastic Averaging SGD. In Proc. 28th International Conference on Neural Information Processing Systems - Volume 1 (NIPS’15). 685–693. [find](https://scholar.google.com/scholar?q=S%2E+Zhang%2C+A%2E+Choromanska%2C+and+Y%2E+LeCun%2E+2015%2E+Deep+Learning+with+Elastic+Averaging+SGD%2E+In+Proc%2E+28th+International+Conference+on+Neural+Information+Processing+Systems+%2D+Volume+1+%28NIPS%E2%80%9915%29%2E+685%E2%80%93693%2E)
259. S. Zhang, Z. Du, L. Zhang, H. Lan, S. Liu, L. Li, Q. Guo, T. Chen, and Y. Chen. 2016. Cambricon-X: An accelerator for sparse neural networks. In 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 1–12. [find](https://scholar.google.com/scholar?q=S%2E+Zhang%2C+Z%2E+Du%2C+L%2E+Zhang%2C+H%2E+Lan%2C+S%2E+Liu%2C+L%2E+Li%2C+Q%2E+Guo%2C+T%2E+Chen%2C+and+Y%2E+Chen%2E+2016%2E+Cambricon%2DX%3A+An+accelerator+for+sparse+neural+networks%2E+In+49th+Annual+IEEE%2FACM+International+Symposium+on+Microar)
260. S. Zhang, C. Zhang, Z. You, R. Zheng, and B. Xu. 2013. Asynchronous stochastic gradient descent for DNN training. In IEEE International Conference on Acoustics, Speech and Signal Processing. 6660–6663. [find](https://scholar.google.com/scholar?q=S%2E+Zhang%2C+C%2E+Zhang%2C+Z%2E+You%2C+R%2E+Zheng%2C+and+B%2E+Xu%2E+2013%2E+Asynchronous+stochastic+gradient+descent+for+DNN+training%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing%2E+6660%E2%80%936663)
261. W. Zhang, S. Gupta, X. Lian, and J. Liu. 2016. Staleness-aware async-SGD for Distributed Deep Learning. In Proc. Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI’16). 2350–2356. [find](https://scholar.google.com/scholar?q=W%2E+Zhang%2C+S%2E+Gupta%2C+X%2E+Lian%2C+and+J%2E+Liu%2E+2016%2E+Staleness%2Daware+async%2DSGD+for+Distributed+Deep+Learning%2E+In+Proc%2E+Twenty%2DFifth+International+Joint+Conference+on+Artificial+Intelligence+%28IJCAI%E2%80%9916%29%2E+2350)
262. X. Zhang, M. McKenna, J. P. Mesirov, and D. L. Waltz. 1990. An Efficient Implementation of the Back-propagation Algorithm on the Connection Machine CM-2. In Advances in Neural Information Processing Systems 2. 801–809. [find](https://scholar.google.com/scholar?q=X%2E+Zhang%2C+M%2E+McKenna%2C+J%2E+P%2E+Mesirov%2C+and+D%2E+L%2E+Waltz%2E+1990%2E+An+Efficient+Implementation+of+the+Back%2Dpropagation+Algorithm+on+the+Connection+Machine+CM%2D2%2E+In+Advances+in+Neural+Information+Processing+S)
263. H. Zhao and J. Canny. 2014. Kylix: A Sparse Allreduce for Commodity Clusters. In 43rd International Conference on Parallel Processing. 273–282. [find](https://scholar.google.com/scholar?q=H%2E+Zhao+and+J%2E+Canny%2E+2014%2E+Kylix%3A+A+Sparse+Allreduce+for+Commodity+Clusters%2E+In+43rd+International+Conference+on+Parallel+Processing%2E+273%E2%80%93282%2E)
264. Z. Zhong, J. Yan, and C.-L. Liu. 2017. Practical Network Blocks Design with Q-Learning. (2017). arXiv:1708.05552 [find](https://scholar.google.com/scholar?q=Z%2E+Zhong%2C+J%2E+Yan%2C+and+C%2E%2DL%2E+Liu%2E+2017%2E+Practical+Network+Blocks+Design+with+Q%2DLearning%2E+%282017%29%2E+arXiv%3A1708%2E05552)
265. S. Zhou, Z. Ni, X. Zhou, H. Wen, Y. Wu, and Y. Zou. 2016. [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://md.research.elamur.ai/free/en/papers/zhou-2016-dorefa-net-training-low-bitwidth/). (2016). arXiv:1606.06160
266. M. A. Zinkevich, M. Weimer, A. Smola, and L. Li. 2010. Parallelized Stochastic Gradient Descent. In Proc. 23rd International Conference on Neural Information Processing Systems - Volume 2. 2595–2603. [find](https://scholar.google.com/scholar?q=M%2E+A%2E+Zinkevich%2C+M%2E+Weimer%2C+A%2E+Smola%2C+and+L%2E+Li%2E+2010%2E+Parallelized+Stochastic+Gradient+Descent%2E+In+Proc%2E+23rd+International+Conference+on+Neural+Information+Processing+Systems+%2D+Volume+2%2E+2595%E2%80%932603%2E)
267. A. Zlateski, K. Lee, and H. S. Seung. 2016. ZNNi: Maximizing the Inference Throughput of 3D Convolutional Networks on CPUs and GPUs. In Int’l Conf. for High Performance Computing, Networking, Storage and Analysis. 854–865. [find](https://scholar.google.com/scholar?q=A%2E+Zlateski%2C+K%2E+Lee%2C+and+H%2E+S%2E+Seung%2E+2016%2E+ZNNi%3A+Maximizing+the+Inference+Throughput+of+3D+Convolutional+Networks+on+CPUs+and+GPUs%2E+In+Int%E2%80%99l+Conf%2E+for+High+Performance+Computing%2C+Networking%2C+Storage+)
268. B. Zoph and Q. V. Le. 2017. [Neural Architecture Search with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/). In International Conference on Learning Representations (ICLR).
269. B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. 2017. [Learning Transferable Architectures for Scalable Image Recognition](https://md.research.elamur.ai/free/en/papers/zoph-2017-learning-transferable-architectures-scalable-image/). (2017). arXiv:1707.07012

*Updated 2 November 2026*