---
title: "PipeDream: Fast and Efficient Pipeline Parallel DNN Training"
description: "Data-parallel DNN training gives every GPU a full model copy and synchronizes weights after each… 47 references, cited by 74 papers here."
authors: ["Aaron Harlap", "Deepak Narayanan", "Amar Phanishayee", "Vivek Seshadri", "Nikhil Devanur", "Greg Ganger", "Phil Gibbons"]
year: 2018
venue: "arXiv:1806.03377"
source: "https://arxiv.org/abs/1806.03377"
doi: "10.48550/arxiv.1806.03377"
arxiv: "1806.03377"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/harlap-2018-pipedream-fast-efficient-pipeline-parallel/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/harlap-2018-pipedream-fast-efficient-pipeline-parallel/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / PipeDream: Fast and Efficient Pipeline Parallel DNN Training

# PipeDream: Fast and Efficient Pipeline Parallel DNN Training

**Authors:** Aaron Harlap, [Deepak Narayanan](https://md.research.elamur.ai/free/en/authors/deepak-narayanan/), [Amar Phanishayee](https://md.research.elamur.ai/free/en/authors/amar-phanishayee/), [Vivek Seshadri](https://md.research.elamur.ai/free/en/authors/vivek-seshadri/), Nikhil Devanur, Greg Ganger and Phil Gibbons · **Year:** 2018 · **Venue:** arXiv:1806.03377 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1806.03377) · [PDF](https://arxiv.org/pdf/1806.03377) · [DOI 10.48550/arxiv.1806.03377](https://doi.org/10.48550/arxiv.1806.03377) · [arXiv 1806.03377](https://arxiv.org/abs/1806.03377) · [OpenAlex](https://openalex.org/W2807147113)

**Citations:** 88 citations (OpenAlex, 2026-10-14) · [74](https://md.research.elamur.ai/free/en/papers/harlap-2018-pipedream-fast-efficient-pipeline-parallel/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Data-parallel DNN training gives every GPU a full model copy and synchronizes weights after each step. As models grow and GPUs get faster, network traffic increasingly dominates runtime. For VGG16, it can take up to 85% of training time. Classic model parallelism avoids exchanging all weights but leaves most workers idle. PipeDream splits a network's layers into stages on different machines and feeds in several minibatches at once so every stage stays busy. A short profiling run drives an automatic partitioner that balances compute and keeps inter-stage traffic low. Overloaded stages can be replicated in data-parallel fashion. Each worker alternates forward and backward work in round-robin order. It also stores weight versions per in-flight minibatch, so backward passes reuse the parameters from their forward pass. The authors tested four CNNs on ILSVRC12 and the S2VT video-captioning RNN on MSVD, using two GPU clusters. PipeDream cut communication by up to 95% and reached target accuracy up to 5x sooner than data-parallel training. This makes pipelining a practical option when bandwidth limits scaling.

## Key points
- PipeDream combines pipelining, model parallelism and data parallelism automatically. Only activations and gradients cross stage boundaries, which reduces traffic by up to 95% compared with data-parallel BSP.
- Speedups over data-parallel BSP on ILSVRC12: Inception-v3 1.45x, VGG16 5.12x, ResNet-50 1.21x, AlexNet 6.76x. S2VT on MSVD trains about 3x faster.
- VGG16 on Cluster-A with 4, 8 and 16 machines: BSP gives 1.47x, 2.35x and 3.28x over a single machine, while PipeDream gives 3.14x, 7.04x and 9.86x.
- PipeDream reached 48% VGG16 accuracy 7.4x faster than 4-machine ASP. ASP avoids communication stalls but suffers from stale weights.
- Ablation on VGG16: simple model parallelism is slower than one machine. Straight pipelines without stage replication reach 2.56x and 3.49x on 4 and 8 machines, and the full PipeDream design performs best.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations
- [Microsoft Research (United Kingdom)](https://md.research.elamur.ai/free/en/companies/microsoft-research-united-kingdom/) — Company, Cambridge, United Kingdom · 11 works · 3,215 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

## Cite

Cite as: Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger and Phil Gibbons. 2018. PipeDream: Fast and Efficient Pipeline Parallel DNN Training. arXiv:1806.03377.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{harlap2018pipedream,
  title = {{PipeDream: Fast and Efficient Pipeline Parallel DNN Training}},
  author = {Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger and Phil Gibbons},
  year = {2018},
  doi = {10.48550/arxiv.1806.03377},
  eprint = {1806.03377},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1806.03377}
}
```

- [← previous LCQMC:A Large-scale Chinese Question Matching Corpus](https://md.research.elamur.ai/free/en/papers/liu-2018-lcqmc-large-scale-chinese-question/)
- [next → Extreme Adaptation for Personalized Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/michel-2018-extreme-adaptation-personalized-neural-machine/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

47 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. [Tensorflow: A system for large-scale machine learning](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-system-large-scale-machine/). In 12…
2. Baidu Inc. Bringing HPC Techniques to Deep Learning, 2017. URL [http://research.baidu.com/bringing-hpc-techniques-deep-learning/](http://research.baidu.com/bringing-hpc-techniques-deep-learning/). [↗](http://research.baidu.com/bringing-hpc-techniques-deep-learning/)
3. D. L. Chen and W. B. Dolan. [Collecting highly parallel data for paraphrase evaluation](https://md.research.elamur.ai/free/en/papers/chen-2011-collecting-highly-parallel-data-paraphrase/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 190–200. Association for Computational Linguistics, 2011.
4. J. Chen, R. Monga, S. Bengio, and R. Jozefowicz. Revisiting distributed synchronous sgd. In International Conference on Learning Representations Workshop Track, 2016. URL https://arxiv.org/abs/1604.00981. — [arXiv:1604.00981](https://arxiv.org/abs/1604.00981) [↗](https://arxiv.org/abs/1604.00981)
5. T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang. [Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems](https://md.research.elamur.ai/free/en/papers/chen-2015-mxnet-flexible-efficient-machine-learning/). CoRR, abs/1512.01274, 2015. URL http://arxiv.org/abs/1512.01274. [↗](http://arxiv.org/abs/1512.01274)
6. X. Chen, A. Eversole, G. Li, D. Yu, and F. Seide. Pipelined Back-propagation for Context-dependent Deep Neural Networks. In Interspeech, 2012. [find](https://scholar.google.com/scholar?q=X%2E+Chen%2C+A%2E+Eversole%2C+G%2E+Li%2C+D%2E+Yu%2C+and+F%2E+Seide%2E+Pipelined+Back%2Dpropagation+for+Context%2Ddependent+Deep+Neural+Networks%2E+In+Interspeech%2C+2012%2E)
7. T. M. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. Project adam: Building an efficient and scalable deep learning training system. In OSDI, volume 14, pages 571–582, 2014. [find](https://scholar.google.com/scholar?q=T%2E+M%2E+Chilimbi%2C+Y%2E+Suzue%2C+J%2E+Apacible%2C+and+K%2E+Kalyanaraman%2E+Project+adam%3A+Building+an+efficient+and+scalable+deep+learning+training+system%2E+In+OSDI%2C+volume+14%2C+pages+571%E2%80%93582%2C+2014%2E)
8. H. Cui, J. Cipar, Q. Ho, J. K. Kim, S. Lee, A. Kumar, J. Wei, W. Dai, G. R. Ganger, P. B. Gibbons, et al. Exploiting bounded staleness to speed up big data analytics. In USENIX Annual Technical Conference, pages 37–48, 2014. [find](https://scholar.google.com/scholar?q=H%2E+Cui%2C+J%2E+Cipar%2C+Q%2E+Ho%2C+J%2E+K%2E+Kim%2C+S%2E+Lee%2C+A%2E+Kumar%2C+J%2E+Wei%2C+W%2E+Dai%2C+G%2E+R%2E+Ganger%2C+P%2E+B%2E+Gibbons%2C+et+al%2E+Exploiting+bounded+staleness+to+speed+up+big+data+analytics%2E+In+USENIX+Annual+Technical+Confer)
9. H. Cui, H. Zhang, G. R. Ganger, P. B. Gibbons, and E. P. Xing. Geeps: Scalable deep learning on distributed gpus with a gpu-specialized parameter server. In Proceedings of the Eleventh European Conference on Computer Systems, page 4. ACM, 2016. [find](https://scholar.google.com/scholar?q=H%2E+Cui%2C+H%2E+Zhang%2C+G%2E+R%2E+Ganger%2C+P%2E+B%2E+Gibbons%2C+and+E%2E+P%2E+Xing%2E+Geeps%3A+Scalable+deep+learning+on+distributed+gpus+with+a+gpu%2Dspecialized+parameter+server%2E+In+Proceedings+of+the+Eleventh+European+Confer)
10. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pages 1223–1231, 2012. [find](https://scholar.google.com/scholar?q=J%2E+Dean%2C+G%2E+Corrado%2C+R%2E+Monga%2C+K%2E+Chen%2C+M%2E+Devin%2C+M%2E+Mao%2C+A%2E+Senior%2C+P%2E+Tucker%2C+K%2E+Yang%2C+Q%2E+V%2E+Le%2C+et+al%2E+Large+scale+distributed+deep+networks%2E+In+Advances+in+neural+information+processing+systems%2C+p)
11. M. Denkowski and A. Lavie. [Meteor universal: Language specific translation evaluation for any target language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/). In Proceedings of the ninth workshop on statistical machine translation, pages 376–380, 2014.
12. DGX-1. NVIDIA DGX-1. [https://www.nvidia.com/en-us/data-center/dgx-1/](https://www.nvidia.com/en-us/data-center/dgx-1/). [↗](https://www.nvidia.com/en-us/data-center/dgx-1/)
13. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul): 2121–2159, 2011. [find](https://scholar.google.com/scholar?q=J%2E+Duchi%2C+E%2E+Hazan%2C+and+Y%2E+Singer%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+Journal+of+Machine+Learning+Research%2C+12%28Jul%29%3A+2121%E2%80%932159%2C+2011%2E)
14. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint arXiv:1706.02677, 2017. [find](https://scholar.google.com/scholar?q=P%2E+Goyal%2C+P%2E+Doll%C3%A1r%2C+R%2E+Girshick%2C+P%2E+Noordhuis%2C+L%2E+Wesolowski%2C+A%2E+Kyrola%2C+A%2E+Tulloch%2C+Y%2E+Jia%2C+and+K%2E+He%2E+Accurate%2C+Large+Minibatch+SGD%3A+Training+ImageNet+in+1+Hour%2E+arXiv+preprint+arXiv%3A1706%2E02677%2C+20)
15. S. Hadjis, C. Zhang, I. Mitliagkas, D. Iter, and C. Ré. Omnivore: An optimizer for multi-device deep learning on cpus and gpus. arXiv preprint arXiv:1606.04487, 2016. [find](https://scholar.google.com/scholar?q=S%2E+Hadjis%2C+C%2E+Zhang%2C+I%2E+Mitliagkas%2C+D%2E+Iter%2C+and+C%2E+R%C3%A9%2E+Omnivore%3A+An+optimizer+for+multi%2Ddevice+deep+learning+on+cpus+and+gpus%2E+arXiv+preprint+arXiv%3A1606%2E04487%2C+2016%2E)
16. K. He, X. Zhang, S. Ren, and J. Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). CoRR, abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385. [↗](http://arxiv.org/abs/1512.03385)
17. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82–97, Nov 2012. [find](https://scholar.google.com/scholar?q=G%2E+Hinton%2C+L%2E+Deng%2C+D%2E+Yu%2C+G%2E+E%2E+Dahl%2C+A%2E+r%2E+Mohamed%2C+N%2E+Jaitly%2C+A%2E+Senior%2C+V%2E+Vanhoucke%2C+P%2E+Nguyen%2C+T%2E+N%2E+Sainath%2C+and+B%2E+Kingsbury%2E+Deep+neural+networks+for+acoustic+modeling+in+speech+recognition%3A+)
18. Q. Ho, J. Cipar, H. Cui, S. Lee, J. K. Kim, P. B. Gibbons, G. A. Gibson, G. Ganger, and E. P. Xing. More effective distributed ml via a stale synchronous parallel parameter server. In Advances in neural information processing systems, pages 1223–1231, 2013. [find](https://scholar.google.com/scholar?q=Q%2E+Ho%2C+J%2E+Cipar%2C+H%2E+Cui%2C+S%2E+Lee%2C+J%2E+K%2E+Kim%2C+P%2E+B%2E+Gibbons%2C+G%2E+A%2E+Gibson%2C+G%2E+Ganger%2C+and+E%2E+P%2E+Xing%2E+More+effective+distributed+ml+via+a+stale+synchronous+parallel+parameter+server%2E+In+Advances+in+neur)
19. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448–456, 2015. [find](https://scholar.google.com/scholar?q=S%2E+Ioffe+and+C%2E+Szegedy%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+In+International+Conference+on+Machine+Learning%2C+pages+448%E2%80%93456%2C+2015%2E)
20. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. [Caffe: Convolutional architecture for fast feature embedding](https://md.research.elamur.ai/free/en/papers/jia-2014-caffe-convolutional-architecture-fast-feature/). arXiv preprint arXiv:1408.5093, 2014.
21. N. Kalchbrenner, E. Grefenstette, and P. Blunsom. [A convolutional neural network for modelling sentences](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2014-convolutional-neural-network-modelling-sentences/). CoRR, abs/1404.2188, 2014. URL http://arxiv.org/abs/1404.2188. [↗](http://arxiv.org/abs/1404.2188)
22. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 1725–1732, June 2014. [find](https://scholar.google.com/scholar?q=A%2E+Karpathy%2C+G%2E+Toderici%2C+S%2E+Shetty%2C+T%2E+Leung%2C+R%2E+Sukthankar%2C+and+L%2E+Fei%2DFei%2E+Large%2Dscale+video+classification+with+convolutional+neural+networks%2E+In+2014+IEEE+Conference+on+Computer+Vision+and+Patter)
23. J. K. Kim, Q. Ho, S. Lee, X. Zheng, W. Dai, G. A. Gibson, and E. P. Xing. STRADS: a distributed framework for scheduled model parallel machine learning. In EuroSys, pages 5:1–5:16, 2016. [find](https://scholar.google.com/scholar?q=J%2E+K%2E+Kim%2C+Q%2E+Ho%2C+S%2E+Lee%2C+X%2E+Zheng%2C+W%2E+Dai%2C+G%2E+A%2E+Gibson%2C+and+E%2E+P%2E+Xing%2E+STRADS%3A+a+distributed+framework+for+scheduled+model+parallel+machine+learning%2E+In+EuroSys%2C+pages+5%3A1%E2%80%935%3A16%2C+2016%2E)
24. D. Kingma and J. Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014.
25. A. Krizhevsky. [One weird trick for parallelizing convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2014-one-weird-trick-parallelizing-convolutional/). arXiv preprint arXiv:1404.5997, 2014.
26. A. Krizhevsky, I. Sutskever, and G. E. Hinton. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In Advances in neural information processing systems, pages 1097–1105, 2012.
27. S. Lee, J. K. Kim, X. Zheng, Q. Ho, G. A. Gibson, and E. P. Xing. On model parallelization and scheduling strategies for distributed machine learning. In NIPS, pages 2834– 2842, 2014. [find](https://scholar.google.com/scholar?q=S%2E+Lee%2C+J%2E+K%2E+Kim%2C+X%2E+Zheng%2C+Q%2E+Ho%2C+G%2E+A%2E+Gibson%2C+and+E%2E+P%2E+Xing%2E+On+model+parallelization+and+scheduling+strategies+for+distributed+machine+learning%2E+In+NIPS%2C+pages+2834%E2%80%93+2842%2C+2014%2E)
28. M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su. Scaling distributed machine learning with the parameter server. In OSDI, volume 1, page 3, 2014. [find](https://scholar.google.com/scholar?q=M%2E+Li%2C+D%2E+G%2E+Andersen%2C+J%2E+W%2E+Park%2C+A%2E+J%2E+Smola%2C+A%2E+Ahmed%2C+V%2E+Josifovski%2C+J%2E+Long%2C+E%2E+J%2E+Shekita%2C+and+B%2E%2DY%2E+Su%2E+Scaling+distributed+machine+learning+with+the+parameter+server%2E+In+OSDI%2C+volume+1%2C+page+3)
29. Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. M. Hellerstein. Distributed graphlab: A framework for machine learning in the cloud. PVLDB, 5(8):716–727, 2012. [find](https://scholar.google.com/scholar?q=Y%2E+Low%2C+J%2E+Gonzalez%2C+A%2E+Kyrola%2C+D%2E+Bickson%2C+C%2E+Guestrin%2C+and+J%2E+M%2E+Hellerstein%2E+Distributed+graphlab%3A+A+framework+for+machine+learning+in+the+cloud%2E+PVLDB%2C+5%288%29%3A716%E2%80%93727%2C+2012%2E)
30. A. Mirhoseini, H. Pham, Q. Le, M. Norouzi, S. Bengio, B. Steiner, Y. Zhou, N. Kumar, R. Larsen, and J. Dean. Device placement optimization with reinforcement learning. 2017. URL https://arxiv.org/abs/1706.04972. [↗](https://arxiv.org/abs/1706.04972)
31. J. Močkus. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference, pages 400–404. Springer, 1975. [find](https://scholar.google.com/scholar?q=J%2E+Mo%C4%8Dkus%2E+On+bayesian+methods+for+seeking+the+extremum%2E+In+Optimization+Techniques+IFIP+Technical+Conference%2C+pages+400%E2%80%93404%2E+Springer%2C+1975%2E)
32. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. [Imagenet large scale visual recognition challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/). International Journal of Computer Vision, 115(3):211– 252, 2015.
33. F. Seide and A. Agarwal. CNTK: Microsoft’s open-source deep-learning toolkit. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 2135–2135, New York, NY, USA, 2016. URL [https://github.com/Microsoft/CNTK](https://github.com/Microsoft/CNTK). [↗](https://github.com/Microsoft/CNTK)
34. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. On parallelizability of stochastic gradient descent for speech dnns. In International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE SPS, May 2014. [find](https://scholar.google.com/scholar?q=F%2E+Seide%2C+H%2E+Fu%2C+J%2E+Droppo%2C+G%2E+Li%2C+and+D%2E+Yu%2E+On+parallelizability+of+stochastic+gradient+descent+for+speech+dnns%2E+In+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2E+IEEE)
35. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In Fifteenth Annual Conference of the International Speech Communication Association, 2014. [find](https://scholar.google.com/scholar?q=F%2E+Seide%2C+H%2E+Fu%2C+J%2E+Droppo%2C+G%2E+Li%2C+and+D%2E+Yu%2E+1%2Dbit+stochastic+gradient+descent+and+its+application+to+data%2Dparallel+distributed+training+of+speech+dnns%2E+In+Fifteenth+Annual+Conference+of+the+Internat)
36. K. Simonyan and A. Zisserman. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). arXiv preprint arXiv:1409.1556, 2014.
37. J. Snoek, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pages 2951–2959, 2012. [find](https://scholar.google.com/scholar?q=J%2E+Snoek%2C+H%2E+Larochelle%2C+and+R%2E+P%2E+Adams%2E+Practical+bayesian+optimization+of+machine+learning+algorithms%2E+In+Advances+in+neural+information+processing+systems%2C+pages+2951%E2%80%932959%2C+2012%2E)
38. E. R. Sparks, A. Talwalkar, D. Haas, M. J. Franklin, M. I. Jordan, and T. Kraska. Automating model search for large scale machine learning. In Proceedings of the Sixth ACM Symposium on Cloud Computing, pages 368–380. ACM, 2015. [find](https://scholar.google.com/scholar?q=E%2E+R%2E+Sparks%2C+A%2E+Talwalkar%2C+D%2E+Haas%2C+M%2E+J%2E+Franklin%2C+M%2E+I%2E+Jordan%2C+and+T%2E+Kraska%2E+Automating+model+search+for+large+scale+machine+learning%2E+In+Proceedings+of+the+Sixth+ACM+Symposium+on+Cloud+Computing)
39. R. Thakur, R. Rabenseifner, and W. Gropp. Optimization of collective communication operations in mpich. The International Journal of High Performance Computing Applications, 19(1):49–66, 2005. [find](https://scholar.google.com/scholar?q=R%2E+Thakur%2C+R%2E+Rabenseifner%2C+and+W%2E+Gropp%2E+Optimization+of+collective+communication+operations+in+mpich%2E+The+International+Journal+of+High+Performance+Computing+Applications%2C+19%281%29%3A49%E2%80%9366%2C+2005%2E)
40. T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2): 26–31, 2012. [find](https://scholar.google.com/scholar?q=T%2E+Tieleman+and+G%2E+Hinton%2E+Lecture+6%2E5%2Drmsprop%3A+Divide+the+gradient+by+a+running+average+of+its+recent+magnitude%2E+COURSERA%3A+Neural+networks+for+machine+learning%2C+4%282%29%3A+26%E2%80%9331%2C+2012%2E)
41. Uber Technologies Inc. Meet Horovod: Uber’s Open Source Distributed Deep Learning Framework for Tensor-Flow, 2017. URL [https://eng.uber.com/horovod/](https://eng.uber.com/horovod/). [↗](https://eng.uber.com/horovod/)
42. L. G. Valiant. A bridging model for parallel computation. Commun. ACM, 33(8), Aug. 1990. [find](https://scholar.google.com/scholar?q=L%2E+G%2E+Valiant%2E+A+bridging+model+for+parallel+computation%2E+Commun%2E+ACM%2C+33%288%29%2C+Aug%2E+1990%2E)
43. S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko. Sequence to sequence-video to text. In Proceedings of the IEEE international conference on computer vision, pages 4534–4542, 2015. [find](https://scholar.google.com/scholar?q=S%2E+Venugopalan%2C+M%2E+Rohrbach%2C+J%2E+Donahue%2C+R%2E+Mooney%2C+T%2E+Darrell%2C+and+K%2E+Saenko%2E+Sequence+to+sequence%2Dvideo+to+text%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pages+4534%E2%80%9345)
44. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). CoRR, abs/1411.4555, 2014. URL http://arxiv.org/abs/1411.4555. [↗](http://arxiv.org/abs/1411.4555)
45. J. Yuan, F. Gao, Q. Ho, W. Dai, J. Wei, X. Zheng, E. P. Xing, T. Liu, and W. Ma. Lightlda: Big topic models on modest computer clusters. In WWW, pages 1351–1361. ACM, 2015. [find](https://scholar.google.com/scholar?q=J%2E+Yuan%2C+F%2E+Gao%2C+Q%2E+Ho%2C+W%2E+Dai%2C+J%2E+Wei%2C+X%2E+Zheng%2C+E%2E+P%2E+Xing%2C+T%2E+Liu%2C+and+W%2E+Ma%2E+Lightlda%3A+Big+topic+models+on+modest+computer+clusters%2E+In+WWW%2C+pages+1351%E2%80%931361%2E+ACM%2C+2015%2E)
46. ZeroMQ. Distributed Messaging. [http://zeromq.org/](http://zeromq.org/). [↗](http://zeromq.org/)
47. H. Zhang, Z. Zheng, S. Xu, W. Dai, Q. Ho, X. Liang, Z. Hu, J. Wei, P. Xie, and E. P. Xing. Poseidon: An efficient communication architecture for distributed deep learning on GPU clusters. In 2017 USENIX Annual Technical Conference (USENIX ATC 17), pages 181–193, Santa Clara, CA, 2017. USENIX Associa… [find](https://scholar.google.com/scholar?q=H%2E+Zhang%2C+Z%2E+Zheng%2C+S%2E+Xu%2C+W%2E+Dai%2C+Q%2E+Ho%2C+X%2E+Liang%2C+Z%2E+Hu%2C+J%2E+Wei%2C+P%2E+Xie%2C+and+E%2E+P%2E+Xing%2E+Poseidon%3A+An+efficient+communication+architecture+for+distributed+deep+learning+on+GPU+clusters%2E+In+2017+USEN)

*Updated 2 November 2026*