---
title: "DAPPLE — paper by Fan et al. (2020) — summary and ideas"
description: "The authors from Alibaba and HKU build a synchronous framework with three parts. 59 references, cited by 81 papers here."
authors: ["Shiqing Fan", "Yi Rong", "Chen Meng", "Zongyan Cao", "Siyu Wang", "Zhen Zheng", "Chuan Wu", "Guoping Long", "Jun Yang", "Lixue Xia", "Lansong Diao", "Xiaoyong Liu", "Wei Lin"]
year: 2020
venue: "arXiv:2007.01045"
source: "https://arxiv.org/abs/2007.01045"
doi: "10.48550/arxiv.2007.01045"
arxiv: "2007.01045"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fan-2020-dapple-pipelined-data-parallel-approach/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fan-2020-dapple-pipelined-data-parallel-approach/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

# DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

**Authors:** Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, [Siyu Wang](https://md.research.elamur.ai/free/en/authors/siyu-wang/), [Zhen Zheng](https://md.research.elamur.ai/free/en/authors/zhen-zheng/), [Chuan Wu](https://md.research.elamur.ai/free/en/authors/chuan-wu/), Guoping Long, [Jun Yang](https://md.research.elamur.ai/free/en/authors/jun-yang/), Lixue Xia, Lansong Diao, Xiaoyong Liu and [Wei Lin](https://md.research.elamur.ai/free/en/authors/wei-lin/) · **Year:** 2020 · **Venue:** arXiv:2007.01045 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2007.01045) · [PDF](https://arxiv.org/pdf/2007.01045) · [DOI 10.48550/arxiv.2007.01045](https://doi.org/10.48550/arxiv.2007.01045) · [arXiv 2007.01045](https://arxiv.org/abs/2007.01045) · [OpenAlex](https://openalex.org/W3038581078)

**Citations:** 26 citations (OpenAlex, 2026-10-14) · [81](https://md.research.elamur.ai/free/en/papers/fan-2020-dapple-pipelined-data-parallel-approach/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
DAPPLE tackles the problem of training very large DNNs on GPU clusters whose interconnects vary, such as NVLink inside servers and slower Ethernet between them. Pure data parallelism breaks down when a model no longer fits on one device or when gradient synchronization dominates. Existing pipeline approaches have their own costs: asynchronous ones (PipeDream) raise convergence concerns and store several weight versions, while synchronous GPipe needs a lot of activation memory or about 20% extra recomputation. The authors from Alibaba and HKU build a synchronous framework with three parts. A profiler collects per-layer compute time, activation size and parameter size. A planner then searches hybrid strategies that split layers into stages, replicate stages data-parallel style, and place them on devices with awareness of memory and topology. Finally, a runtime interleaves forward and backward work so activations are freed early. Across image classification, translation and language modeling, DAPPLE beats DP, the PipeDream planner and GPipe in throughput while using less memory. This makes synchronous large-model training more practical on everyday industrial hardware.

## Key points
- Benchmarks: GNMT-16, BERT-48, XLNET-36, AmoebaNet-36 and VGG-19, tested on three cluster setups: hierarchical NVLink + Ethernet, 25 Gbps Ethernet and 10 Gbps Ethernet.
- Reported gains:
  - up to 2.32× over the best DP baseline;
  - up to 3.23× over strategies from the PipeDream planner in synchronous settings;
  - 1.6× higher throughput than GPipe, with 12% less memory.
- The scheduling method is independent of re-computation. It saves roughly another 20% of device memory even when recomputation is already turned on.
- The efficiency model treats bubble overhead as P = (1+α)(S−1)/M and efficiency as 1/(1+P), where S is the number of stages, M the number of micro-batches and α the communication-to-computation ratio. More micro-batches and fewer stages give better utilization.
- Activations crossing partition boundaries are tiny next to gradients, for example 8.8MB vs 2.8GB for BERT-48 and 26MB vs 1.1GB for GNMT-16. This explains why pipelining can cut traffic compared with full DP.

## Interesting ideas
- **Model communication as a pipeline stage.** Folding communication into the analytical pipeline model gives a simple formula that predicts when a split pays off. The same trick could help capacity planning for any staged data-processing job.
- **Compare traffic volumes before choosing a strategy.** Measuring boundary-activation size against gradient size is a cheap first check for whether pipelining or plain DP suits a model and network. This is handy before committing to costly infrastructure.
- **Profile first, then plan automatically.** Collecting per-layer statistics and letting a planner search hybrid layouts, including stage replication, replaces manual tuning with a reproducible search. It also adapts when the hardware changes.
- **Interleave forward and backward work.** Running backward passes in a fixed order alongside forward ones lets memory for each micro-batch be released as early as possible. Peak memory drops without recomputing anything, and the idea stacks with other memory-saving techniques.
- **Topology-aware placement.** Assigning stages to devices with the interconnect hierarchy in mind matters on mixed NVLink/Ethernet clusters. It is a useful design rule for anyone training on heterogeneous or rented GPU setups.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 201 works · 17,173 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 118 works · 13,496 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

## Cite

Cite as: Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu and Wei Lin. 2020. DAPPLE: A Pipelined Data Parallel Approach for Training Large Models. arXiv:2007.01045.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{fan2020dapple,
  title = {{DAPPLE: A Pipelined Data Parallel Approach for Training Large Models}},
  author = {Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu and Wei Lin},
  year = {2020},
  doi = {10.48550/arxiv.2007.01045},
  eprint = {2007.01045},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2007.01045}
}
```

- [← previous An Empirical Study of Pre-trained Transformers for Arabic Information Extraction](https://md.research.elamur.ai/free/en/papers/lan-2020-empirical-study-pre-trained-transformers/)
- [next → LEDGAR: A Large-Scale Multi-label Corpus for Text Classification of Legal Provisions in Contracts](https://md.research.elamur.ai/free/en/papers/tuggener-2020-ledgar-large-scale-multi-label/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

59 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Sutton, The Bitter Lesson, 2019, [http://www.incompleteideas.net/IncIdeas/BitterLesson.html](http://www.incompleteideas.net/IncIdeas/BitterLesson.html). [↗](http://www.incompleteideas.net/IncIdeas/BitterLesson.html)
2. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and J. P. Liu, “[Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/),” https://arxiv.org/abs/1910.10683, 2019. [↗](https://arxiv.org/abs/1910.10683)
3. J. Wang, P. Huang, H. Zhao, Z. Zhang, B. Zhao, and D. L. Lee, “Billion-scale commodity embedding for e-commerce recommendation in alibaba,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2018, pp. 839–848. [find](https://scholar.google.com/scholar?q=J%2E+Wang%2C+P%2E+Huang%2C+H%2E+Zhao%2C+Z%2E+Zhang%2C+B%2E+Zhao%2C+and+D%2E+L%2E+Lee%2C+%E2%80%9CBillion%2Dscale+commodity+embedding+for+e%2Dcommerce+recommendation+in+alibaba%2C%E2%80%9D+in+Proceedings+of+the+24th+ACM+SIGKDD+International+Conferen)
4. P. Covington, J. Adams, and E. Sargin, “Deep neural networks for youtube recommendations,” in Proceedings of the 10th ACM Conference on Recommender Systems, ACM, New York, NY, USA. ACM, 2016. [find](https://scholar.google.com/scholar?q=P%2E+Covington%2C+J%2E+Adams%2C+and+E%2E+Sargin%2C+%E2%80%9CDeep+neural+networks+for+youtube+recommendations%2C%E2%80%9D+in+Proceedings+of+the+10th+ACM+Conference+on+Recommender+Systems%2C+ACM%2C+New+York%2C+NY%2C+USA%2E+ACM%2C+2016%2E)
5. Gradients Accumulation-Tensorflow, 2019, [https://github.com/tensorflow/tensorflow/pull/32576](https://github.com/tensorflow/tensorflow/pull/32576). [↗](https://github.com/tensorflow/tensorflow/pull/32576)
6. Gradients Accumulation-PyTorch, 2019, [https://gist.github.com/thomwolf/ac7a7da6b1888c2eeac8ac8b9b05d3d3](https://gist.github.com/thomwolf/ac7a7da6b1888c2eeac8ac8b9b05d3d3). [↗](https://gist.github.com/thomwolf/ac7a7da6b1888c2eeac8ac8b9b05d3d3)
7. A New Lightweight, Modular, and Scalable Deep Learning Framework, 2016, [https://caffe2.ai/](https://caffe2.ai/). [↗](https://caffe2.ai/)
8. A. Jayarajan, J. Wei, G. Gibson, A. Fedorova, and G. Pekhimenko, “Priority-based parameter propagation for distributed dnn training,” arXiv preprint arXiv:1905.03960, 2019. — [arXiv:1905.03960](https://arxiv.org/abs/1905.03960) [find](https://scholar.google.com/scholar?q=A%2E+Jayarajan%2C+J%2E+Wei%2C+G%2E+Gibson%2C+A%2E+Fedorova%2C+and+G%2E+Pekhimenko%2C+%E2%80%9CPriority%2Dbased+parameter+propagation+for+distributed+dnn+training%2C%E2%80%9D+arXiv+preprint+arXiv%3A1905%2E03960%2C+2019%2E+%E2%80%94+%5BarXiv%3A1905%2E03960%5D%28https%3A)
9. A. Sergeev and M. Del Balso, “[Horovod: fast and easy distributed deep learning in tensorflow](https://md.research.elamur.ai/free/en/papers/sergeev-2018-horovod-fast-easy-distributed-deep/),” arXiv preprint arXiv:1802.05799, 2018.
10. Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu et al., “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” in Advances in Neural Information Processing Systems, 2019, pp. 103–112. [find](https://scholar.google.com/scholar?q=Y%2E+Huang%2C+Y%2E+Cheng%2C+A%2E+Bapna%2C+O%2E+Firat%2C+D%2E+Chen%2C+M%2E+Chen%2C+H%2E+Lee%2C+J%2E+Ngiam%2C+Q%2E+V%2E+Le%2C+Y%2E+Wu+et+al%2E%2C+%E2%80%9CGpipe%3A+Efficient+training+of+giant+neural+networks+using+pipeline+parallelism%2C%E2%80%9D+in+Advances+in+Neur)
11. J. Zhan and J. Zhang, “Pipe-torch: Pipeline-based distributed deep learning in a gpu cluster with heterogeneous networking,” in 2019 Seventh International Conference on Advanced Cloud and Big Data (CBD). IEEE, 2019, pp. 55–60. [find](https://scholar.google.com/scholar?q=J%2E+Zhan+and+J%2E+Zhang%2C+%E2%80%9CPipe%2Dtorch%3A+Pipeline%2Dbased+distributed+deep+learning+in+a+gpu+cluster+with+heterogeneous+networking%2C%E2%80%9D+in+2019+Seventh+International+Conference+on+Advanced+Cloud+and+Big+Data+%28CB)
12. D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles. ACM, 2019, pp. 1–15. [find](https://scholar.google.com/scholar?q=D%2E+Narayanan%2C+A%2E+Harlap%2C+A%2E+Phanishayee%2C+V%2E+Seshadri%2C+N%2E+R%2E+Devanur%2C+G%2E+R%2E+Ganger%2C+P%2E+B%2E+Gibbons%2C+and+M%2E+Zaharia%2C+%E2%80%9CPipedream%3A+generalized+pipeline+parallelism+for+dnn+training%2C%E2%80%9D+in+Proceedings+of+the+)
13. T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,” arXiv preprint arXiv:1604.06174, 2016. [find](https://scholar.google.com/scholar?q=T%2E+Chen%2C+B%2E+Xu%2C+C%2E+Zhang%2C+and+C%2E+Guestrin%2C+%E2%80%9CTraining+deep+nets+with+sublinear+memory+cost%2C%E2%80%9D+arXiv+preprint+arXiv%3A1604%2E06174%2C+2016%2E)
14. A. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. Devanur, G. Ganger, and P. Gibbons, “[Pipedream: Fast and efficient pipeline parallel dnn training](https://md.research.elamur.ai/free/en/papers/harlap-2018-pipedream-fast-efficient-pipeline-parallel/),” arXiv preprint arXiv:1806.03377, 2018.
15. M. Wang, C. Meng, G. Long, C. Wu, J. Yang, W. Lin, and Y. Jia, “Characterizing deep learning training workloads on alibaba-pai,” arXiv preprint arXiv:1910.05930, 2019. [find](https://scholar.google.com/scholar?q=M%2E+Wang%2C+C%2E+Meng%2C+G%2E+Long%2C+C%2E+Wu%2C+J%2E+Yang%2C+W%2E+Lin%2C+and+Y%2E+Jia%2C+%E2%80%9CCharacterizing+deep+learning+training+workloads+on+alibaba%2Dpai%2C%E2%80%9D+arXiv+preprint+arXiv%3A1910%2E05930%2C+2019%2E)
16. GpipeTalk, 2019, [https://www.youtube.com/watch?v=9s2cum25Kkc](https://www.youtube.com/watch?v=9s2cum25Kkc). [↗](https://www.youtube.com/watch?v=9s2cum25Kkc)
17. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,” arXiv preprint arXiv:1706.02677, 2017. [find](https://scholar.google.com/scholar?q=P%2E+Goyal%2C+P%2E+Dolla%CC%81r%2C+R%2E+Girshick%2C+P%2E+Noordhuis%2C+L%2E+Wesolowski%2C+A%2E+Kyrola%2C+A%2E+Tulloch%2C+Y%2E+Jia%2C+and+K%2E+He%2C+%E2%80%9CAccurate%2C+large+minibatch+sgd%3A+Training+imagenet+in+1+hour%2C%E2%80%9D+arXiv+preprint+arXiv%3A1706%2E02677%2C)
18. G. Long, J. Yang, K. Zhu, and W. Lin, “Fusionstitching: Deep fusion and code generation for tensorflow computations on gpus,” https://arxiv.org/abs/1811.05213, 2018. [↗](https://arxiv.org/abs/1811.05213)
19. G. Long, J. Yang, and W. Lin, “Fusionstitching: Boosting execution efficiency of memory intensive computations for dl workloads,” https://arxiv.org/abs/1911.11576, 2019. [↗](https://arxiv.org/abs/1911.11576)
20. H. Zhang, Z. Zheng, S. Xu, W. Dai, Q. Ho, X. Liang, Z. Hu, J. Wei, P. Xi, and E. P. Xing, “Poseidon: An efficient communication architecture for distributed deep learning on GPU clusters,” in 2017 USENIX Annual Technical Conference (USENIX ATC 17). Santa Clara, CA: USENIX Association, Jul. 2017, pp.… [find](https://scholar.google.com/scholar?q=H%2E+Zhang%2C+Z%2E+Zheng%2C+S%2E+Xu%2C+W%2E+Dai%2C+Q%2E+Ho%2C+X%2E+Liang%2C+Z%2E+Hu%2C+J%2E+Wei%2C+P%2E+Xi%2C+and+E%2E+P%2E+Xing%2C+%E2%80%9CPoseidon%3A+An+efficient+communication+architecture+for+distributed+deep+learning+on+GPU+clusters%2C%E2%80%9D+in+2017+USE)
21. B. Yang, J. Zhang, J. Li, C. Ré, C. R. Aberger, and C. De Sa, “Pipemare: Asynchronous pipeline parallel dnn training,” arXiv preprint arXiv:1910.05124, 2019. — [arXiv:1910.05124](https://arxiv.org/abs/1910.05124) [find](https://scholar.google.com/scholar?q=B%2E+Yang%2C+J%2E+Zhang%2C+J%2E+Li%2C+C%2E+Re%CC%81%2C+C%2E+R%2E+Aberger%2C+and+C%2E+De+Sa%2C+%E2%80%9CPipemare%3A+Asynchronous+pipeline+parallel+dnn+training%2C%E2%80%9D+arXiv+preprint+arXiv%3A1910%2E05124%2C+2019%2E+%E2%80%94+%5BarXiv%3A1910%2E05124%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
22. NVLink, 2019, [https://www.nvidia.com/en-us/data-center/nvlink/](https://www.nvidia.com/en-us/data-center/nvlink/). [↗](https://www.nvidia.com/en-us/data-center/nvlink/)
23. H. Lee, M. Jeong, C. Kim, S. Lim, I. Kim, W. Baek, and B. Yoon, “torchgpipe, A GPipe implementation in PyTorch,” [https://github.com/kakaobrain/torchgpipe](https://github.com/kakaobrain/torchgpipe), 2019. [↗](https://github.com/kakaobrain/torchgpipe)
24. I. Bárány and V. S. Grinberg, “Block partitions of sequences,” Israel Journal of Mathematics, vol. 206, no. 1, pp. 155–164, 2015. [find](https://scholar.google.com/scholar?q=I%2E+Ba%CC%81ra%CC%81ny+and+V%2E+S%2E+Grinberg%2C+%E2%80%9CBlock+partitions+of+sequences%2C%E2%80%9D+Israel+Journal+of+Mathematics%2C+vol%2E+206%2C+no%2E+1%2C+pp%2E+155%E2%80%93164%2C+2015%2E)
25. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B.… [find](https://scholar.google.com/scholar?q=M%2E+Abadi%2C+A%2E+Agarwal%2C+P%2E+Barham%2C+E%2E+Brevdo%2C+Z%2E+Chen%2C+C%2E+Citro%2C+G%2E+S%2E+Corrado%2C+A%2E+Davis%2C+J%2E+Dean%2C+M%2E+Devin%2C+S%2E+Ghemawat%2C+I%2E+Goodfellow%2C+A%2E+Harp%2C+G%2E+Irving%2C+M%2E+Isard%2C+Y%2E+Jia%2C+R%2E+Jozefowicz%2C+L%2E+Kaiser%2C+M)
26. J. Demouth, “Cuda pro tip: Minimize the tail effect,” 2015. [Online]. Available: [https://devblogs.nvidia.com/cuda-pro-tip-minimize-the-tail-effect/](https://devblogs.nvidia.com/cuda-pro-tip-minimize-the-tail-effect/) [↗](https://devblogs.nvidia.com/cuda-pro-tip-minimize-the-tail-effect/)
27. R. Sennrich, B. Haddow, and A. Birch, “[Edinburgh neural machine translation systems for wmt 16](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/),” arXiv preprint arXiv:1606.02891, 2016.
28. P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” arXiv preprint arXiv:1806.[03822, 2018](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/).
29. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “[Imagenet large scale visual recognition challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/),” International journal of computer vision, vol. 115, no. 3, pp. 211–252, 2015.
30. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al., “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144, 2016. [find](https://scholar.google.com/scholar?q=Y%2E+Wu%2C+M%2E+Schuster%2C+Z%2E+Chen%2C+Q%2E+V%2E+Le%2C+M%2E+Norouzi%2C+W%2E+Macherey%2C+M%2E+Krikun%2C+Y%2E+Cao%2C+Q%2E+Gao%2C+K%2E+Macherey+et+al%2E%2C+%E2%80%9CGoogle%E2%80%99s+neural+machine+translation+system%3A+Bridging+the+gap+between+human+and+machine+t)
31. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “[Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/),” arXiv preprint arXiv:1810.04805, 2018.
32. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, “[Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/),” arXiv preprint arXiv:1906.08237, 2019.
33. K. He, X. Zhang, S. Ren, and J. Sun, “[Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/),” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
34. K. Simonyan and A. Zisserman, “[Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/),” arXiv preprint arXiv:1409.1556, 2014.
35. S. A. R. Shah, W. Wu, Q. Lu, L. Zhang, S. Sasidharan, P. DeMar, C. Guok, J. Macauley, E. Pouyoul, J. Kim et al., “Amoebanet: An sdn-enabled network service for big data science,” Journal of Network and Computer Applications, vol. 119, pp. 70–82, 2018. [find](https://scholar.google.com/scholar?q=S%2E+A%2E+R%2E+Shah%2C+W%2E+Wu%2C+Q%2E+Lu%2C+L%2E+Zhang%2C+S%2E+Sasidharan%2C+P%2E+DeMar%2C+C%2E+Guok%2C+J%2E+Macauley%2C+E%2E+Pouyoul%2C+J%2E+Kim+et+al%2E%2C+%E2%80%9CAmoebanet%3A+An+sdn%2Denabled+network+service+for+big+data+science%2C%E2%80%9D+Journal+of+Network+an)
36. NCCL, 2019, [https://developer.nvidia.com/nccl](https://developer.nvidia.com/nccl). [↗](https://developer.nvidia.com/nccl)
37. D. P. Kingma and J. Ba, “[Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/),” arXiv preprint arXiv:1412.6980, 2014.
38. T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural Networks for Machine Learning 4, 2012. [find](https://scholar.google.com/scholar?q=T%2E+Tieleman+and+G%2E+Hinton%2C+%E2%80%9CLecture+6%2E5%2Drmsprop%3A+Divide+the+gradient+by+a+running+average+of+its+recent+magnitude%2C%E2%80%9D+COURSERA%3A+Neural+Networks+for+Machine+Learning+4%2C+2012%2E)
39. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang et al., “Large scale distributed deep networks,” in Advances in neural information processing systems, 2012, pp. 1223–1231. [find](https://scholar.google.com/scholar?q=J%2E+Dean%2C+G%2E+Corrado%2C+R%2E+Monga%2C+K%2E+Chen%2C+M%2E+Devin%2C+M%2E+Mao%2C+M%2E+Ranzato%2C+A%2E+Senior%2C+P%2E+Tucker%2C+K%2E+Yang+et+al%2E%2C+%E2%80%9CLarge+scale+distributed+deep+networks%2C%E2%80%9D+in+Advances+in+neural+information+processing+system)
40. S. Pal, E. Ebrahimi, A. Zulfiqar, Y. Fu, V. Zhang, S. Migacz, D. Nellans, and P. Gupta, “Optimizing multi-gpu parallelization strategies for deep learning training,” IEEE Micro, vol. 39, no. 5, pp. 91–101, 2019. [find](https://scholar.google.com/scholar?q=S%2E+Pal%2C+E%2E+Ebrahimi%2C+A%2E+Zulfiqar%2C+Y%2E+Fu%2C+V%2E+Zhang%2C+S%2E+Migacz%2C+D%2E+Nellans%2C+and+P%2E+Gupta%2C+%E2%80%9COptimizing+multi%2Dgpu+parallelization+strategies+for+deep+learning+training%2C%E2%80%9D+IEEE+Micro%2C+vol%2E+39%2C+no%2E+5%2C+pp%2E+91)
41. Z. Jia, S. Lin, C. R. Qi, and A. Aiken, “Exploring the hidden dimension in accelerating convolutional neural networks,” 2018. [find](https://scholar.google.com/scholar?q=Z%2E+Jia%2C+S%2E+Lin%2C+C%2E+R%2E+Qi%2C+and+A%2E+Aiken%2C+%E2%80%9CExploring+the+hidden+dimension+in+accelerating+convolutional+neural+networks%2C%E2%80%9D+2018%2E)
42. J. Geng, D. Li, and S. Wang, “Horizontal or vertical?: A hybrid approach to large-scale distributed machine learning,” in Proceedings of the 10th Workshop on Scientific Cloud Computing. ACM, 2019, pp. 1–4. [find](https://scholar.google.com/scholar?q=J%2E+Geng%2C+D%2E+Li%2C+and+S%2E+Wang%2C+%E2%80%9CHorizontal+or+vertical%3F%3A+A+hybrid+approach+to+large%2Dscale+distributed+machine+learning%2C%E2%80%9D+in+Proceedings+of+the+10th+Workshop+on+Scientific+Cloud+Computing%2E+ACM%2C+2019%2C+pp%2E)
43. A. Krizhevsky, “[One weird trick for parallelizing convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2014-one-weird-trick-parallelizing-convolutional/),” arXiv preprint arXiv:1404.5997, 2014.
44. Baidu-allreduce, 2018, [https://github.com/baidu-research/baidu-allreduce](https://github.com/baidu-research/baidu-allreduce). [↗](https://github.com/baidu-research/baidu-allreduce)
45. X. Jia, S. Song, W. He, Y. Wang, H. Rong, F. Zhou, L. Xie, Z. Guo, Y. Yang, L. Yu et al., “[Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes](https://md.research.elamur.ai/free/en/papers/jia-2018-highly-scalable-deep-learning-training/),” arXiv preprint arXiv:1807.11205, 2018.
46. N. Arivazhagan, A. Bapna, O. Firat, D. Lepikhin, M. Johnson, M. Krikun, M. X. Chen, Y. Cao, G. Foster, C. Cherry et al., “[Massively multilingual neural machine translation in the wild: Findings and challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/),” arXiv preprint arXiv:1907.05019, 2019.
47. Byteps, A high performance and generic framework for distributed DNN training, 2019, [https://github.com/bytedance/byteps](https://github.com/bytedance/byteps). [↗](https://github.com/bytedance/byteps)
48. Y. You, J. Hseu, C. Ying, J. Demmel, K. Keutzer, and C.-J. Hsieh, “Large-batch training for lstm and beyond,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2019, pp. 1–16. [find](https://scholar.google.com/scholar?q=Y%2E+You%2C+J%2E+Hseu%2C+C%2E+Ying%2C+J%2E+Demmel%2C+K%2E+Keutzer%2C+and+C%2E%2DJ%2E+Hsieh%2C+%E2%80%9CLarge%2Dbatch+training+for+lstm+and+beyond%2C%E2%80%9D+in+Proceedings+of+the+International+Conference+for+High+Performance+Computing%2C+Networking%2C)
49. T. D. Le, T. Sekiyama, Y. Negishi, H. Imai, and K. Kawachiya, “Involving cpus into multi-gpu deep learning,” in Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering. ACM, 2018, pp. 56–67. [find](https://scholar.google.com/scholar?q=T%2E+D%2E+Le%2C+T%2E+Sekiyama%2C+Y%2E+Negishi%2C+H%2E+Imai%2C+and+K%2E+Kawachiya%2C+%E2%80%9CInvolving+cpus+into+multi%2Dgpu+deep+learning%2C%E2%80%9D+in+Proceedings+of+the+2018+ACM%2FSPEC+International+Conference+on+Performance+Engineering%2E+AC)
50. Z. Jia, M. Zaharia, and A. Aiken, “[Beyond data and model parallelism for deep neural networks](https://md.research.elamur.ai/free/en/papers/jia-2018-beyond-data-model-parallelism-deep/),” arXiv preprint arXiv:1807.05358, 2018.
51. Z. Huo, B. Gu, Q. Yang, and H. Huang, “Decoupled parallel backpropagation with convergence guarantee,” arXiv preprint arXiv:1804.10574, 2018. — [arXiv:1804.10574](https://arxiv.org/abs/1804.10574) [find](https://scholar.google.com/scholar?q=Z%2E+Huo%2C+B%2E+Gu%2C+Q%2E+Yang%2C+and+H%2E+Huang%2C+%E2%80%9CDecoupled+parallel+backpropagation+with+convergence+guarantee%2C%E2%80%9D+arXiv+preprint+arXiv%3A1804%2E10574%2C+2018%2E+%E2%80%94+%5BarXiv%3A1804%2E10574%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1804%2E10574%29)
52. J. Geng, D. Li, and S. Wang, “Rima: An rdma-accelerated model-parallelized solution to large-scale matrix factorization,” in 2019 IEEE 35th International Conference on Data Engineering (ICDE). IEEE, 2019, pp. 100–111. [find](https://scholar.google.com/scholar?q=J%2E+Geng%2C+D%2E+Li%2C+and+S%2E+Wang%2C+%E2%80%9CRima%3A+An+rdma%2Daccelerated+model%2Dparallelized+solution+to+large%2Dscale+matrix+factorization%2C%E2%80%9D+in+2019+IEEE+35th+International+Conference+on+Data+Engineering+%28ICDE%29%2E+IEEE%2C+2)
53. C.-C. Chen, C.-L. Yang, and H.-Y. Cheng, “Efficient and robust parallel dnn training through model parallelism on multi-gpu platform,” arXiv preprint arXiv:1809.02839, 2018. [find](https://scholar.google.com/scholar?q=C%2E%2DC%2E+Chen%2C+C%2E%2DL%2E+Yang%2C+and+H%2E%2DY%2E+Cheng%2C+%E2%80%9CEfficient+and+robust+parallel+dnn+training+through+model+parallelism+on+multi%2Dgpu+platform%2C%E2%80%9D+arXiv+preprint+arXiv%3A1809%2E02839%2C+2018%2E)
54. N. Dryden, N. Maruyama, T. Moon, T. Benson, M. Snir, and B. Van Essen, “Channel and filter parallelism for large-scale cnn training,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2019, pp. 1–20. [find](https://scholar.google.com/scholar?q=N%2E+Dryden%2C+N%2E+Maruyama%2C+T%2E+Moon%2C+T%2E+Benson%2C+M%2E+Snir%2C+and+B%2E+Van+Essen%2C+%E2%80%9CChannel+and+filter+parallelism+for+large%2Dscale+cnn+training%2C%E2%80%9D+in+Proceedings+of+the+International+Conference+for+High+Performanc)
55. J. Geng, D. Li, and S. Wang, “Elasticpipe: An efficient and dynamic model-parallel solution to dnn training,” in Proceedings of the 10th Workshop on Scientific Cloud Computing. ACM, 2019, pp. 5–9. [find](https://scholar.google.com/scholar?q=J%2E+Geng%2C+D%2E+Li%2C+and+S%2E+Wang%2C+%E2%80%9CElasticpipe%3A+An+efficient+and+dynamic+model%2Dparallel+solution+to+dnn+training%2C%E2%80%9D+in+Proceedings+of+the+10th+Workshop+on+Scientific+Cloud+Computing%2E+ACM%2C+2019%2C+pp%2E+5%E2%80%939%2E)
56. L. Guan, W. Yin, D. Li, and X. Lu, “Xpipe: Efficient pipeline model parallelism for multi-gpu dnn training,” arXiv preprint arXiv:1911.04610, 2019. [find](https://scholar.google.com/scholar?q=L%2E+Guan%2C+W%2E+Yin%2C+D%2E+Li%2C+and+X%2E+Lu%2C+%E2%80%9CXpipe%3A+Efficient+pipeline+model+parallelism+for+multi%2Dgpu+dnn+training%2C%E2%80%9D+arXiv+preprint+arXiv%3A1911%2E04610%2C+2019%2E)
57. A. Mirhoseini, H. Pham, Q. V. Le, B. Steiner, R. Larsen, Y. Zhou, N. Kumar, M. Norouzi, S. Bengio, and J. Dean, “Device placement optimization with reinforcement learning,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017, pp. 2430–2439. [find](https://scholar.google.com/scholar?q=A%2E+Mirhoseini%2C+H%2E+Pham%2C+Q%2E+V%2E+Le%2C+B%2E+Steiner%2C+R%2E+Larsen%2C+Y%2E+Zhou%2C+N%2E+Kumar%2C+M%2E+Norouzi%2C+S%2E+Bengio%2C+and+J%2E+Dean%2C+%E2%80%9CDevice+placement+optimization+with+reinforcement+learning%2C%E2%80%9D+in+Proceedings+of+the+34th+)
58. J. Gu, M. Chowdhury, K. G. Shin, Y. Zhu, M. Jeon, J. Qian, H. Liu, and C. Guo, “Tiresias: A {GPU} cluster manager for distributed deep learning,” in 16th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 19), 2019, pp. 485–500. [find](https://scholar.google.com/scholar?q=J%2E+Gu%2C+M%2E+Chowdhury%2C+K%2E+G%2E+Shin%2C+Y%2E+Zhu%2C+M%2E+Jeon%2C+J%2E+Qian%2C+H%2E+Liu%2C+and+C%2E+Guo%2C+%E2%80%9CTiresias%3A+A+%7BGPU%7D+cluster+manager+for+distributed+deep+learning%2C%E2%80%9D+in+16th+%7BUSENIX%7D+Symposium+on+Networked+Systems+Design)
59. M. Wang, C.-c. Huang, and J. Li, “Supporting very large models using automatic dataflow graph partitioning,” in Proceedings of the Fourteenth EuroSys Conference 2019, 2019, pp. 1–17. [find](https://scholar.google.com/scholar?q=M%2E+Wang%2C+C%2E%2Dc%2E+Huang%2C+and+J%2E+Li%2C+%E2%80%9CSupporting+very+large+models+using+automatic+dataflow+graph+partitioning%2C%E2%80%9D+in+Proceedings+of+the+Fourteenth+EuroSys+Conference+2019%2C+2019%2C+pp%2E+1%E2%80%9317%2E)

*Updated 2 November 2026*