---
title: "Alpa — paper by Zheng et al. (2022) · OSDI 2022"
description: "Training models with billions of parameters requires combining data, operator and pipeline parallelism. 62 references, cited by 208 papers here."
authors: ["Lianmin Zheng", "Zhuohan Li", "Hao Zhang", "Yonghao Zhuang", "Zhifeng Chen", "Yanping Huang", "Yida Wang", "Yuanzhong Xu", "Danyang Zhuo", "Eric P. Xing"]
year: 2022
venue: "OSDI 2022"
source: "https://arxiv.org/abs/2201.12023"
arxiv: "2201.12023"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zheng-2022-alpa-automating-inter-intra-operator/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zheng-2022-alpa-automating-inter-intra-operator/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

# Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

**Authors:** [Lianmin Zheng](https://md.research.elamur.ai/free/en/authors/lianmin-zheng/), [Zhuohan Li](https://md.research.elamur.ai/free/en/authors/zhuohan-li/), [Hao Zhang](https://md.research.elamur.ai/free/en/authors/hao-zhang/), [Yonghao Zhuang](https://md.research.elamur.ai/free/en/authors/yonghao-zhuang/), [Zhifeng Chen](https://md.research.elamur.ai/free/en/authors/zhifeng-chen/), [Yanping Huang](https://md.research.elamur.ai/free/en/authors/yanping-huang/), Yida Wang, [Yuanzhong Xu](https://md.research.elamur.ai/free/en/authors/yuanzhong-xu/), [Danyang Zhuo](https://md.research.elamur.ai/free/en/authors/danyang-zhuo/) and [Eric P. Xing](https://md.research.elamur.ai/free/en/authors/eric-p-xing/) · **Year:** 2022 · **Venue:** OSDI 2022 · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2201.12023) · [PDF](https://arxiv.org/pdf/2201.12023) · [arXiv 2201.12023](https://arxiv.org/abs/2201.12023) · [OpenAlex](https://openalex.org/W4226479682)

**Citations:** 71 citations (OpenAlex, 2026-10-14) · [208](https://md.research.elamur.ai/free/en/papers/zheng-2022-alpa-automating-inter-intra-operator/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training models with billions of parameters requires combining data, operator and pipeline parallelism. Until now, engineers either designed that combination by hand for a specific model and cluster, as with Megatron-LM's "3D parallelism", or used automatic tools that search only one kind of parallelism. Alpa takes a different view. It groups all techniques into two levels:

- **Intra-operator parallelism:** splits individual tensor operators across devices.
- **Inter-operator parallelism:** cuts the graph into pipelined stages.

The compiler divides the cluster into device meshes with fast internal links, partitions the model into stages, and maps each stage to a mesh. It then optimizes each level as a separate tractable sub-problem. A runtime coordinates execution, with SPMD inside a mesh and MPMD across meshes.

The authors tested Alpa on 64 GPUs:

- On GPT, it matched Megatron-LM.
- On GShard MoE, it clearly outperformed DeepSpeed.
- On Wide-ResNet, a model with no known manual plan, it scaled well.

This matters because it makes large-model training far less dependent on rare expert tuning.

## Key points
- **Test cluster:** Amazon EC2, 8 p3.16xlarge instances, 64 GPUs in total.
- **GShard MoE vs. DeepSpeed:** 3.5× faster on 2 nodes and 9.7× faster on 4 nodes.
- **GPT vs. Megatron-LM:** throughput on par with this specialized system.
- **Wide-ResNet:** about 80% linear scaling efficiency on 4 nodes.
  - In the discovered 16-GPU plan, stages use unequal mesh sizes (4, 4 and 8 GPUs).
  - The plan includes unusual convolution partitionings found by an ILP solver.
- **Data parallelism:** treated as a special case of intra-op parallelism, namely splitting along the batch axis.
- **Supporting techniques:** Alpa also uses rematerialization and mixed precision.
- **Availability:** the code is open source (github.com/alpa-projects/alpa).

## Interesting ideas
- **Match the parallelism to the network.** Communication-heavy operator splitting is placed on tightly connected devices, while pipeline stages, which only exchange data with their neighbours, span the slower links. The same matching could guide placement of any distributed workload where bandwidth varies across the cluster.
- **Split the search into two levels.** Instead of searching one huge combinatorial space, Alpa optimizes two nested sub-problems, each solvable near-optimally. The result is not guaranteed to be globally optimal, but it works well in practice, which makes this a useful template for other combinatorial planning tasks.
- **Unify concepts to shrink the design space.** Recasting data parallelism as a batch-axis operator partition removes one separate category. Looking for such unifications can simplify other system designs.
- **Test on models without a hand-made strategy.** Wide-ResNet has no expert plan to copy, so it checks whether an automatic system generalizes rather than just reproducing tuned baselines. This is a good evaluation habit for any auto-tuning tool.
- **Inspect the discovered plans.** Visualizing the chosen strategy, as in Figure 12, shows non-obvious choices that a human expert would be unlikely to make. Such views help build trust in automated optimizers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 209 works · 244,782 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations
- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 128 works · 41,700 citations
- [Duke University](https://md.research.elamur.ai/free/en/universities/duke-university/) — University, Durham, United States · 37 works · 15,861 citations
- [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 138 works · 6,590 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 100 works

## Cite

Cite as: Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo and Eric P. Xing. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. OSDI 2022.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zheng2022alpa,
  title = {{Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning}},
  author = {Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo and Eric P. Xing},
  booktitle = {OSDI 2022},
  year = {2022},
  eprint = {2201.12023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.12023}
}
```

- [← previous Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://md.research.elamur.ai/free/en/papers/yu-2022-generate-rather-than-retrieve-large/)
- [next → OpenPrompt: An Open-source Framework for Prompt-learning](https://md.research.elamur.ai/free/en/papers/ding-2022-openprompt-open-source-framework-prompt/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/)

## References

62 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. [Tensorflow: a system for large-scale machine learning](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-system-large-scale-machine/). In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 265…
2. Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, and Nipun Kwatra. Varuna: scalable, low-cost training of massive deep learning models. In Proceedings of the Seventeenth European Conference on Computer Systems, pages 472–487, 2022. [find](https://scholar.google.com/scholar?q=Sanjith+Athlur%2C+Nitika+Saran%2C+Muthian+Sivathanu%2C+Ramachandran+Ramjee%2C+and+Nipun+Kwatra%2E+Varuna%3A+scalable%2C+low%2Dcost+training+of+massive+deep+learning+models%2E+In+Proceedings+of+the+Seventeenth+European+)
3. AWS Cluster Configuratoins. [https://aws.amazon.com/ec2/instance-types/p3/](https://aws.amazon.com/ec2/instance-types/p3/). [↗](https://aws.amazon.com/ec2/instance-types/p3/)
4. Kevin Aydin, MohammadHossein Bateni, and Vahab Mirrokni. Distributed balanced partitioning via linear embedding. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, pages 387–396, 2016. [find](https://scholar.google.com/scholar?q=Kevin+Aydin%2C+MohammadHossein+Bateni%2C+and+Vahab+Mirrokni%2E+Distributed+balanced+partitioning+via+linear+embedding%2E+In+Proceedings+of+the+Ninth+ACM+International+Conference+on+Web+Search+and+Data+Mining%2C)
5. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450, 2016.
6. Youhui Bai, Cheng Li, Quan Zhou, Jun Yi, Ping Gong, Feng Yan, Ruichuan Chen, and Yinlong Xu. Gradient compression supercharged high-performance data parallel dnn training. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles CD-ROM, pages 359–375, 2021. [find](https://scholar.google.com/scholar?q=Youhui+Bai%2C+Cheng+Li%2C+Quan+Zhou%2C+Jun+Yi%2C+Ping+Gong%2C+Feng+Yan%2C+Ruichuan+Chen%2C+and+Yinlong+Xu%2E+Gradient+compression+supercharged+high%2Dperformance+data+parallel+dnn+training%2E+In+Proceedings+of+the+ACM+SI)
7. Paul Barham, Aakanksha Chowdhery, Jeff Dean, Sanjay Ghemawat, Steven Hand, Daniel Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy, et al. Pathways: Asynchronous distributed dataflow for ml. Proceedings of Machine Learning and Systems, 4, 2022. [find](https://scholar.google.com/scholar?q=Paul+Barham%2C+Aakanksha+Chowdhery%2C+Jeff+Dean%2C+Sanjay+Ghemawat%2C+Steven+Hand%2C+Daniel+Hurt%2C+Michael+Isard%2C+Hyeontaek+Lim%2C+Ruoming+Pang%2C+Sudip+Roy%2C+et+al%2E+Pathways%3A+Asynchronous+distributed+dataflow+for+ml)
8. L Susan Blackford, Jaeyoung Choi, Andy Cleary, Eduardo D’Azevedo, James Demmel, Inderjit Dhillon, Jack Dongarra, Sven Hammarling, Greg Henry, Antoine Petitet, et al. ScaLAPACK users’ guide. SIAM, 1997. [find](https://scholar.google.com/scholar?q=L+Susan+Blackford%2C+Jaeyoung+Choi%2C+Andy+Cleary%2C+Eduardo+D%E2%80%99Azevedo%2C+James+Demmel%2C+Inderjit+Dhillon%2C+Jack+Dongarra%2C+Sven+Hammarling%2C+Greg+Henry%2C+Antoine+Petitet%2C+et+al%2E+ScaLAPACK+users%E2%80%99+guide%2E+SIAM%2C+1997)
9. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. [find](https://scholar.google.com/scholar?q=James+Bradbury%2C+Roy+Frostig%2C+Peter+Hawkins%2C+Matthew+James+Johnson%2C+Chris+Leary%2C+Dougal+Maclaurin%2C+George+Necula%2C+Adam+Paszke%2C+Jake+VanderPlas%2C+Skye+Wanderman%2DMilne%2C+and+Qiao+Zhang%2E+JAX%3A+composable+tra)
10. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 2020.
11. Zhenkun Cai, Xiao Yan, Kaihao Ma, Yidi Wu, Yuzhen Huang, James Cheng, Teng Su, and Fan Yu. Tensoropt: Exploring the tradeoffs in distributed dnn training with auto-parallelism. IEEE Transactions on Parallel and Distributed Systems, 33(8):1967–1981, 2021. [find](https://scholar.google.com/scholar?q=Zhenkun+Cai%2C+Xiao+Yan%2C+Kaihao+Ma%2C+Yidi+Wu%2C+Yuzhen+Huang%2C+James+Cheng%2C+Teng+Su%2C+and+Fan+Yu%2E+Tensoropt%3A+Exploring+the+tradeoffs+in+distributed+dnn+training+with+auto%2Dparallelism%2E+IEEE+Transactions+on+Pa)
12. Jianfei Chen, Lianmin Zheng, Zhewei Yao, Dequan Wang, Ion Stoica, Michael W Mahoney, and Joseph E Gonzalez. Actnn: Reducing training memory footprint via 2-bit activation compressed training. In International Conference on Machine Learning, 2021. [find](https://scholar.google.com/scholar?q=Jianfei+Chen%2C+Lianmin+Zheng%2C+Zhewei+Yao%2C+Dequan+Wang%2C+Ion+Stoica%2C+Michael+W+Mahoney%2C+and+Joseph+E+Gonzalez%2E+Actnn%3A+Reducing+training+memory+footprint+via+2%2Dbit+activation+compressed+training%2E+In+Inter)
13. Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. [Tvm: An automated end-to-end optimizing compiler for deep learning](https://md.research.elamur.ai/free/en/papers/chen-2018-tvm-automated-end-end-optimizing/). In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 578–5…
14. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. [find](https://scholar.google.com/scholar?q=Tianqi+Chen%2C+Bing+Xu%2C+Chiyuan+Zhang%2C+and+Carlos+Guestrin%2E+Training+deep+nets+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1604%2E06174%2C+2016%2E)
15. Tyler Denniston, Shoaib Kamil, and Saman Amarasinghe. Distributed halide. ACM SIGPLAN Notices, 51(8):1–12, 2016. [find](https://scholar.google.com/scholar?q=Tyler+Denniston%2C+Shoaib+Kamil%2C+and+Saman+Amarasinghe%2E+Distributed+halide%2E+ACM+SIGPLAN+Notices%2C+51%288%29%3A1%E2%80%9312%2C+2016%2E)
16. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Di… — [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
17. Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al. [Dapple: A pipelined data parallel approach for training large models](https://md.research.elamur.ai/free/en/papers/fan-2020-dapple-pipelined-data-parallel-approach/). In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 43…
18. John Forrest and Robin Lougee-Heimer. Cbc user guide. In Emerging theory, methods, and applications, pages 257–277. INFORMS, 2005. [find](https://scholar.google.com/scholar?q=John+Forrest+and+Robin+Lougee%2DHeimer%2E+Cbc+user+guide%2E+In+Emerging+theory%2C+methods%2C+and+applications%2C+pages+257%E2%80%93277%2E+INFORMS%2C+2005%2E)
19. Richard J Forrester and Noah Hunt-Isaak. Computational comparison of exact solution methods for 0-1 quadratic programs: Recommendations for practitioners. Journal of Applied Mathematics, 2020, 2020. [find](https://scholar.google.com/scholar?q=Richard+J+Forrester+and+Noah+Hunt%2DIsaak%2E+Computational+comparison+of+exact+solution+methods+for+0%2D1+quadratic+programs%3A+Recommendations+for+practitioners%2E+Journal+of+Applied+Mathematics%2C+2020%2C+2020%2E)
20. Google Clould TPU Cluster Configurations. [https://cloud.google.com/tpu](https://cloud.google.com/tpu). [↗](https://cloud.google.com/tpu)
21. Chien-Chin Huang, Gu Jin, and Jinyang Li. Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 1341–1355, 2020. [find](https://scholar.google.com/scholar?q=Chien%2DChin+Huang%2C+Gu+Jin%2C+and+Jinyang+Li%2E+Swapadvisor%3A+Pushing+deep+learning+beyond+the+gpu+memory+limit+via+smart+swapping%2E+In+Proceedings+of+the+Twenty%2DFifth+International+Conference+on+Architectura)
22. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32:103–112, 2019. [find](https://scholar.google.com/scholar?q=Yanping+Huang%2C+Youlong+Cheng%2C+Ankur+Bapna%2C+Orhan+Firat%2C+Dehao+Chen%2C+Mia+Chen%2C+HyoukJoong+Lee%2C+Jiquan+Ngiam%2C+Quoc+V+Le%2C+Yonghui+Wu%2C+et+al%2E+Gpipe%3A+Efficient+training+of+giant+neural+networks+using+pipel)
23. Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E Gonzalez. Checkmate: Breaking the memory wall with optimal tensor rematerialization. arXiv preprint arXiv:1910.02653, 2019. [find](https://scholar.google.com/scholar?q=Paras+Jain%2C+Ajay+Jain%2C+Aniruddha+Nrusimha%2C+Amir+Gholami%2C+Pieter+Abbeel%2C+Kurt+Keutzer%2C+Ion+Stoica%2C+and+Joseph+E+Gonzalez%2E+Checkmate%3A+Breaking+the+memory+wall+with+optimal+tensor+rematerialization%2E+arXi)
24. Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken. [Taso: optimizing deep learning computation with automatic generation of graph substitutions](https://md.research.elamur.ai/free/en/papers/jia-2019-taso-optimizing-deep-learning-computation/). In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 47– 62, 2019.
25. Zhihao Jia, Matei Zaharia, and Alex Aiken. [Beyond data and model parallelism for deep neural networks](https://md.research.elamur.ai/free/en/papers/jia-2018-beyond-data-model-parallelism-deep/). arXiv preprint arXiv:1807.05358, 2018.
26. Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. A unified architecture for accelerating distributed dnn training in heterogeneous gpu/cpu clusters. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 463–479, 2020. [find](https://scholar.google.com/scholar?q=Yimin+Jiang%2C+Yibo+Zhu%2C+Chang+Lan%2C+Bairen+Yi%2C+Yong+Cui%2C+and+Chuanxiong+Guo%2E+A+unified+architecture+for+accelerating+distributed+dnn+training+in+heterogeneous+gpu%2Fcpu+clusters%2E+In+14th+USENIX+Symposium+)
27. Soojeong Kim, Gyeong-In Yu, Hojin Park, Sungwoo Cho, Eunji Jeong, Hyeonmin Ha, Sanha Lee, Joo Seong Jeong, and Byung-Gon Chun. Parallax: Sparsity-aware data parallel training of deep neural networks. In Proceedings of the Fourteenth EuroSys Conference 2019, pages 1–15, 2019. [find](https://scholar.google.com/scholar?q=Soojeong+Kim%2C+Gyeong%2DIn+Yu%2C+Hojin+Park%2C+Sungwoo+Cho%2C+Eunji+Jeong%2C+Hyeonmin+Ha%2C+Sanha+Lee%2C+Joo+Seong+Jeong%2C+and+Byung%2DGon+Chun%2E+Parallax%3A+Sparsity%2Daware+data+parallel+training+of+deep+neural+networks%2E+)
28. Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He, Jared Roesch, Tianqi Chen, and Zachary Tatlock. Dynamic tensor rematerialization. arXiv preprint arXiv:2006.09616, 2020. [find](https://scholar.google.com/scholar?q=Marisa+Kirisame%2C+Steven+Lyubomirsky%2C+Altan+Haan%2C+Jennifer+Brennan%2C+Mike+He%2C+Jared+Roesch%2C+Tianqi+Chen%2C+and+Zachary+Tatlock%2E+Dynamic+tensor+rematerialization%2E+arXiv+preprint+arXiv%3A2006%2E09616%2C+2020%2E)
29. Alex Krizhevsky. [One weird trick for parallelizing convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2014-one-weird-trick-parallelizing-convolutional/). arXiv preprint arXiv:1404.5997, 2014.
30. Woo-Yeon Lee, Yunseong Lee, Joo Seong Jeong, Gyeong-In Yu, Joo Yeon Kim, Ho Jin Park, Beomyeol Jeon, Wonwook Song, Gunhee Kim, Markus Weimer, et al. Automating system configuration of distributed machine learning. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), p… [find](https://scholar.google.com/scholar?q=Woo%2DYeon+Lee%2C+Yunseong+Lee%2C+Joo+Seong+Jeong%2C+Gyeong%2DIn+Yu%2C+Joo+Yeon+Kim%2C+Ho+Jin+Park%2C+Beomyeol+Jeon%2C+Wonwook+Song%2C+Gunhee+Kim%2C+Markus+Weimer%2C+et+al%2E+Automating+system+configuration+of+distributed+mach)
31. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020. [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and+autom)
32. Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. [Pytorch distributed: Experiences on accelerating data parallel training](https://md.research.elamur.ai/free/en/papers/li-2020-pytorch-distributed-experiences-accelerating-data/). arXiv preprint arXiv:2006.15704, 2020.
33. Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. Terapipe: Token-level pipeline parallelism for training large-scale language models. arXiv preprint arXiv:2102.07988, 2021. [find](https://scholar.google.com/scholar?q=Zhuohan+Li%2C+Siyuan+Zhuang%2C+Shiyuan+Guo%2C+Danyang+Zhuo%2C+Hao+Zhang%2C+Dawn+Song%2C+and+Ion+Stoica%2E+Terapipe%3A+Token%2Dlevel+pipeline+parallelism+for+training+large%2Dscale+language+models%2E+arXiv+preprint+arXiv%3A21)
34. Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. Rammer: Enabling holistic deep learning compiler optimizations with rtasks. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 881–897,… [find](https://scholar.google.com/scholar?q=Lingxiao+Ma%2C+Zhiqiang+Xie%2C+Zhi+Yang%2C+Jilong+Xue%2C+Youshan+Miao%2C+Wei+Cui%2C+Wenxiang+Hu%2C+Fan+Yang%2C+Lintao+Zhang%2C+and+Lidong+Zhou%2E+Rammer%3A+Enabling+holistic+deep+learning+compiler+optimizations+with+rtasks)
35. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017. [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Sharan+Narang%2C+Jonah+Alben%2C+Gregory+Diamos%2C+Erich+Elsen%2C+David+Garcia%2C+Boris+Ginsburg%2C+Michael+Houston%2C+Oleksii+Kuchaiev%2C+Ganesh+Venkatesh%2C+et+al%2E+Mixed+precision+training%2E+arXiv)
36. Azalia Mirhoseini, Hieu Pham, Quoc V Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. Device placement optimization with reinforcement learning. In International Conference on Machine Learning, pages 2430–2439. PMLR, 2017. [find](https://scholar.google.com/scholar?q=Azalia+Mirhoseini%2C+Hieu+Pham%2C+Quoc+V+Le%2C+Benoit+Steiner%2C+Rasmus+Larsen%2C+Yuefeng+Zhou%2C+Naveen+Kumar%2C+Mohammad+Norouzi%2C+Samy+Bengio%2C+and+Jeff+Dean%2E+Device+placement+optimization+with+reinforcement+learn)
37. Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging ai applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)… [find](https://scholar.google.com/scholar?q=Philipp+Moritz%2C+Robert+Nishihara%2C+Stephanie+Wang%2C+Alexey+Tumanov%2C+Richard+Liaw%2C+Eric+Liang%2C+Melih+Elibol%2C+Zongheng+Yang%2C+William+Paul%2C+Michael+I+Jordan%2C+et+al%2E+Ray%3A+A+distributed+framework+for+emergin)
38. Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1–15, 2019. [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Aaron+Harlap%2C+Amar+Phanishayee%2C+Vivek+Seshadri%2C+Nikhil+R+Devanur%2C+Gregory+R+Ganger%2C+Phillip+B+Gibbons%2C+and+Matei+Zaharia%2E+Pipedream%3A+generalized+pipeline+parallelism+for+dnn+training)
39. Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. [Memory-efficient pipeline-parallel dnn training](https://md.research.elamur.ai/free/en/papers/narayanan-2020-memory-efficient-pipeline-parallel-dnn/). In International Conference on Machine Learning, pages 7937–7947. PMLR, 2021.
40. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+et+al%2E+Efficient+large%2D)
41. NVIDIA. The nvidia collective communication library, 2018. [find](https://scholar.google.com/scholar?q=NVIDIA%2E+The+nvidia+collective+communication+library%2C+2018%2E)
42. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. [Pytorch: an imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). In Advances in Neural Information Processing Systems, pages 8024–8035, 2019.
43. Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, and Chuanxiong Guo. A generic communication scheduler for distributed dnn training acceleration. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 16– 29, 2019. [find](https://scholar.google.com/scholar?q=Yanghua+Peng%2C+Yibo+Zhu%2C+Yangrui+Chen%2C+Yixin+Bao%2C+Bairen+Yi%2C+Chang+Lan%2C+Chuan+Wu%2C+and+Chuanxiong+Guo%2E+A+generic+communication+scheduler+for+distributed+dnn+training+acceleration%2E+In+Proceedings+of+the+)
44. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing)
45. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26th+ACM+)
46. Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. [Zero-offload: Democratizing billion-scale model training](https://md.research.elamur.ai/free/en/papers/ren-2021-zero-offload-democratizing-billion-scale/). arXiv preprint arXiv:2101.06840, 2021.
47. Alexander Sergeev and Mike Del Balso. [Horovod: fast and easy distributed deep learning in tensorflow](https://md.research.elamur.ai/free/en/papers/sergeev-2018-horovod-fast-easy-distributed-deep/). arXiv preprint arXiv:1802.05799, 2018.
48. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. [Mesh-tensorflow: Deep learning for supercomputers](https://md.research.elamur.ai/free/en/papers/shazeer-2018-mesh-tensorflow-deep-learning-supercomputers/). arXiv preprint arXiv:1811.02084, 2018.
49. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprin)
50. Jakub M Tarnawski, Deepak Narayanan, and Amar Phanishayee. Piper: Multidimensional planner for dnn parallelization. Advances in Neural Information Processing Systems, 34, 2021. [find](https://scholar.google.com/scholar?q=Jakub+M+Tarnawski%2C+Deepak+Narayanan%2C+and+Amar+Phanishayee%2E+Piper%3A+Multidimensional+planner+for+dnn+parallelization%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%2C+2021%2E)
51. Google XLA Team. Xla: Optimizing compiler for machine learning, 2017. [find](https://scholar.google.com/scholar?q=Google+XLA+Team%2E+Xla%3A+Optimizing+compiler+for+machine+learning%2C+2017%2E)
52. The Theano Development Team, Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Angermueller, Dzmitry Bahdanau, Nicolas Ballas, Frédéric Bastien, Justin Bayer, Anatoly Belikov, et al. [Theano: A python framework for fast computation of mathematical expressions](https://md.research.elamur.ai/free/en/papers/the-theano-development-team-2016-theano-python-framework-fast-computation/). arXiv preprint arXiv:1605.02688,…
53. Thijs Vogels, Sai Praneeth Karinireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances In Neural Information Processing Systems 32 (Nips 2019), 32(CONF), 2019. [find](https://scholar.google.com/scholar?q=Thijs+Vogels%2C+Sai+Praneeth+Karinireddy%2C+and+Martin+Jaggi%2E+Powersgd%3A+Practical+low%2Drank+gradient+compression+for+distributed+optimization%2E+Advances+In+Neural+Information+Processing+Systems+32+%28Nips+201)
54. Haojie Wang, Jidong Zhai, Mingyu Gao, Zixuan Ma, Shizhi Tang, Liyan Zheng, Yuanzhi Li, Kaiyuan Rong, Yuanyong Chen, and Zhihao Jia. Pet: Optimizing tensor programs with partially equivalent transformations and automated corrections. In 15th USENIX Symposium on Operating Systems Design and Implementa… [find](https://scholar.google.com/scholar?q=Haojie+Wang%2C+Jidong+Zhai%2C+Mingyu+Gao%2C+Zixuan+Ma%2C+Shizhi+Tang%2C+Liyan+Zheng%2C+Yuanzhi+Li%2C+Kaiyuan+Rong%2C+Yuanyong+Chen%2C+and+Zhihao+Jia%2E+Pet%3A+Optimizing+tensor+programs+with+partially+equivalent+transforma)
55. Minjie Wang, Chien-chin Huang, and Jinyang Li. Supporting very large models using automatic dataflow graph partitioning. In Proceedings of the Fourteenth EuroSys Conference 2019, pages 1–17, 2019. [find](https://scholar.google.com/scholar?q=Minjie+Wang%2C+Chien%2Dchin+Huang%2C+and+Jinyang+Li%2E+Supporting+very+large+models+using+automatic+dataflow+graph+partitioning%2E+In+Proceedings+of+the+Fourteenth+EuroSys+Conference+2019%2C+pages+1%E2%80%9317%2C+2019%2E)
56. Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Hongjun Choi, Blake Hechtman, and Shibo Wang. Automatic cross-replica sharding of weight update in data-parallel training. arXiv preprint arXiv:2004.13336, 2020. — [arXiv:2004.13336](https://arxiv.org/abs/2004.13336) [find](https://scholar.google.com/scholar?q=Yuanzhong+Xu%2C+HyoukJoong+Lee%2C+Dehao+Chen%2C+Hongjun+Choi%2C+Blake+Hechtman%2C+and+Shibo+Wang%2E+Automatic+cross%2Dreplica+sharding+of+weight+update+in+data%2Dparallel+training%2E+arXiv+preprint+arXiv%3A2004%2E13336%2C+20)
57. Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, et al. Gspmd: General and scalable parallelization for ml computation graphs. arXiv preprint arXiv:2105.04663, 2021. [find](https://scholar.google.com/scholar?q=Yuanzhong+Xu%2C+HyoukJoong+Lee%2C+Dehao+Chen%2C+Blake+Hechtman%2C+Yanping+Huang%2C+Rahul+Joshi%2C+Maxim+Krikun%2C+Dmitry+Lepikhin%2C+Andy+Ly%2C+Marcello+Maggioni%2C+et+al%2E+Gspmd%3A+General+and+scalable+parallelization+for+)
58. Jinhui Yuan, Xinqi Li, Cheng Cheng, Juncheng Liu, Ran Guo, Shenghang Cai, Chi Yao, Fei Yang, Xiaodong Yi, Chuan Wu, et al. Oneflow: Redesign the distributed deep learning framework from scratch. arXiv preprint arXiv:2110.15032, 2021. — [arXiv:2110.15032](https://arxiv.org/abs/2110.15032) [find](https://scholar.google.com/scholar?q=Jinhui+Yuan%2C+Xinqi+Li%2C+Cheng+Cheng%2C+Juncheng+Liu%2C+Ran+Guo%2C+Shenghang+Cai%2C+Chi+Yao%2C+Fei+Yang%2C+Xiaodong+Yi%2C+Chuan+Wu%2C+et+al%2E+Oneflow%3A+Redesign+the+distributed+deep+learning+framework+from+scratch%2E+arXiv)
59. Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016. — [arXiv:1605.07146](https://arxiv.org/abs/1605.07146) [find](https://scholar.google.com/scholar?q=Sergey+Zagoruyko+and+Nikos+Komodakis%2E+Wide+residual+networks%2E+arXiv+preprint+arXiv%3A1605%2E07146%2C+2016%2E+%E2%80%94+%5BarXiv%3A1605%2E07146%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1605%2E07146%29)
60. Hao Zhang, Yuan Li, Zhijie Deng, Xiaodan Liang, Lawrence Carin, and Eric Xing. Autosync: Learning to synchronize for data-parallel distributed deep learning. Advances in Neural Information Processing Systems, 33, 2020. [find](https://scholar.google.com/scholar?q=Hao+Zhang%2C+Yuan+Li%2C+Zhijie+Deng%2C+Xiaodan+Liang%2C+Lawrence+Carin%2C+and+Eric+Xing%2E+Autosync%3A+Learning+to+synchronize+for+data%2Dparallel+distributed+deep+learning%2E+Advances+in+Neural+Information+Processing+)
61. Zhen Zhang, Shuai Zheng, Yida Wang, Justin Chiu, George Karypis, Trishul Chilimbi, Mu Li, and Xin Jin. Mics: Near-linear scaling for training gigantic model on public cloud. arXiv preprint arXiv:2205.00119, 2022. [find](https://scholar.google.com/scholar?q=Zhen+Zhang%2C+Shuai+Zheng%2C+Yida+Wang%2C+Justin+Chiu%2C+George+Karypis%2C+Trishul+Chilimbi%2C+Mu+Li%2C+and+Xin+Jin%2E+Mics%3A+Near%2Dlinear+scaling+for+training+gigantic+model+on+public+cloud%2E+arXiv+preprint+arXiv%3A2205%2E)
62. Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, et al. Ansor: Generating high-performance tensor programs for deep learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 863–879,… [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Chengfan+Jia%2C+Minmin+Sun%2C+Zhao+Wu%2C+Cody+Hao+Yu%2C+Ameer+Haj%2DAli%2C+Yida+Wang%2C+Jun+Yang%2C+Danyang+Zhuo%2C+Koushik+Sen%2C+et+al%2E+Ansor%3A+Generating+high%2Dperformance+tensor+programs+for+deep+learnin)

*Updated 2 November 2026*