---
title: "Memory-Efficient Pipeline-Parallel DNN Training (2020)"
description: "Very large language models often cannot fit their weights and activations on one accelerator, so… 23 references, cited by 98 papers here."
authors: ["Deepak Narayanan", "Amar Phanishayee", "Kaiyu Shi", "Xie Chen", "Matei Zaharia"]
year: 2020
venue: "ICML 2021"
source: "https://arxiv.org/abs/2006.09503"
arxiv: "2006.09503"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/narayanan-2020-memory-efficient-pipeline-parallel-dnn/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/narayanan-2020-memory-efficient-pipeline-parallel-dnn/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / Memory-Efficient Pipeline-Parallel DNN Training

# Memory-Efficient Pipeline-Parallel DNN Training

**Authors:** [Deepak Narayanan](https://md.research.elamur.ai/free/en/authors/deepak-narayanan/), [Amar Phanishayee](https://md.research.elamur.ai/free/en/authors/amar-phanishayee/), Kaiyu Shi, [Xie Chen](https://md.research.elamur.ai/free/en/authors/xie-chen/) and [Matei Zaharia](https://md.research.elamur.ai/free/en/authors/matei-zaharia/) · **Year:** 2020 · **Venue:** [ICML 2021](https://md.research.elamur.ai/free/en/venues/international-conference-on-machine-learning/) ([ICML 2020](https://md.research.elamur.ai/free/en/venues/international-conference-on-machine-learning/2020/)) · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2006.09503) · [PDF](https://arxiv.org/pdf/2006.09503) · [arXiv 2006.09503](https://arxiv.org/abs/2006.09503) · [OpenAlex](https://openalex.org/W3036879053)

**Citations:** 55 citations (OpenAlex, 2026-10-14) · [98](https://md.research.elamur.ai/free/en/papers/narayanan-2020-memory-efficient-pipeline-parallel-dnn/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Very large language models often cannot fit their weights and activations on one accelerator, so training must be spread across devices. Earlier pipeline-parallel systems force a choice. GPipe keeps one weight copy but stalls by periodically draining the pipeline. PipeDream avoids the stalls but may store as many weight copies as there are stages. PipeDream-2BW resolves this with double-buffered weight updates (2BW). Gradients from many microbatches are summed into one update. Microbatches already in the pipeline finish on the previous "shadow" weights, while newly admitted ones use the fresh weights, so each worker holds only two versions. The update uses gradients that are exactly one step stale. A variant, PipeDream-Flush, gives up some speed for even less memory and standard update semantics. A memory-aware planner chooses stage sizes, replication, batch size and activation recomputation, and keeps heavy traffic inside servers. On GPT and BERT models, training runs up to 20× faster than a tuned model-parallel baseline with comparable accuracy. This matters because it makes billion-parameter training feasible on commodity GPU clusters.

## Key points
- **Memory per worker:** 2BW keeps two weight versions per worker, versus up to d versions in PipeDream for a pipeline of depth d. It also removes GPipe's pipeline flushes.
- **Speedups:**
  - 1.3× to 20× faster end-to-end on GPT models than an optimized model-parallel (Megatron-style) baseline. The largest gains appear on AWS multi-server setups with limited inter-node bandwidth.
  - Up to 3.2× faster than GPipe.
- **Model size:** about 30 billion parameters trained on 64 GPUs with 16 GB each. For comparison, Megatron-LM trained 8.3B parameters on 8 GPUs with 32 GB each.
- **Convergence:** Adam with 2BW tracks the training loss of vanilla Adam at equal batch size, and downstream finetuning accuracy is similar.
- **Planner cost:** the planner considers only O(m²) (width, depth) configurations for m workers, compared with O(n³m²) for PipeDream's planner. It finishes in under a second, whereas PipeDream's partitioner needs roughly 30 minutes on large models.
- **Limitation:** accumulating gradients inside the pipeline raises the minimum batch size the system supports.

## Interesting ideas
- **Delay updates by a fixed amount.** Accept a constant, uniform one-step staleness in exchange for bounded memory. Because every stage sees the same delay, no new hyperparameter tuning is needed. This could apply to any asynchronous or streaming training setup.
- **Exploit repetition to shrink the search.** Models built from repeated blocks, such as stacks of transformer layers, can be split into equal, equally replicated stages. This turns an exponential partitioning problem into a small exhaustive search, a useful trick for any placement or scheduling task over regular structures.
- **Treat memory savers as tunable options.** Activation recomputation should be chosen jointly with throughput rather than always switched on, since always recomputing cost about 1.33× in speed. This is a reminder to put memory/speed trade-offs inside the cost model.
- **Plan around network topology.** Placing communication-heavy splits on fast intra-server links, and only lighter traffic across servers, explains much of the gain on cheap cloud clusters. It is relevant for teams training on rented, unevenly connected hardware.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations
- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations
- [Databricks](https://md.research.elamur.ai/free/en/companies/databricks/) — Company, San Francisco, United States · 2 works · 217 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen and Matei Zaharia. 2020. Memory-Efficient Pipeline-Parallel DNN Training. ICML 2021.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{narayanan2020memory,
  title = {{Memory-Efficient Pipeline-Parallel DNN Training}},
  author = {Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen and Matei Zaharia},
  booktitle = {ICML 2021},
  year = {2020},
  eprint = {2006.09503},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.09503}
}
```

- [← previous Generating Counter Narratives against Online Hate Speech: Data and Strategies](https://md.research.elamur.ai/free/en/papers/tekiroglu-2020-generating-counter-narratives-against-online/)
- [next → Continual Relation Learning via Episodic Memory Activation and Reconsolidation](https://md.research.elamur.ai/free/en/papers/han-2020-continual-relation-learning-episodic-memory/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

23 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Megatron Repository. — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/) [https://github.com/nvidia/megatron-lm](https://github.com/nvidia/megatron-lm). NVIDIA Deep Learning Examples, BERT. https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/LanguageModeling/BERT/README. md\#results. OpenWebText Dataset. https://github.com/jcpeterson/openwebtext. Brown, T., Mann, B., Ryder, N.… [↗](https://github.com/nvidia/megatron-lm)
2. Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training Deep Nets with Sublinear Memory Cost. arXiv preprint arXiv:1604.06174, 2016. [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E%2C+Xu%2C+B%2E%2C+Zhang%2C+C%2E%2C+and+Guestrin%2C+C%2E+Training+Deep+Nets+with+Sublinear+Memory+Cost%2E+arXiv+preprint+arXiv%3A1604%2E06174%2C+2016%2E)
3. Chilimbi, T. M., Suzue, Y., Apacible, J., and Kalyanaraman, K. Project Adam: Building an Efficient and Scalable Deep Learning Training System. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14), volume 14, pp. 571–582, 2014. [find](https://scholar.google.com/scholar?q=Chilimbi%2C+T%2E+M%2E%2C+Suzue%2C+Y%2E%2C+Apacible%2C+J%2E%2C+and+Kalyanaraman%2C+K%2E+Project+Adam%3A+Building+an+Efficient+and+Scalable+Deep+Learning+Training+System%2E+In+11th+USENIX+Symposium+on+Operating+Systems+Design+and+)
4. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., et al. Large Scale Distributed Deep Networks. In Advances in Neural Information Processing Systems, pp. 1223–1231, 2012. [find](https://scholar.google.com/scholar?q=Dean%2C+J%2E%2C+Corrado%2C+G%2E%2C+Monga%2C+R%2E%2C+Chen%2C+K%2E%2C+Devin%2C+M%2E%2C+Mao%2C+M%2E%2C+Senior%2C+A%2E%2C+Tucker%2C+P%2E%2C+Yang%2C+K%2E%2C+Le%2C+Q%2E+V%2E%2C+et+al%2E+Large+Scale+Distributed+Deep+Networks%2E+In+Advances+in+Neural+Information+Processing+)
5. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. [Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
6. Fan, S., Rong, Y., Meng, C., Cao, Z., Wang, S., Zheng, Z., Wu, C., Long, G., Yang, J., Xia, L., et al. DAP-PLE: A Pipelined Data Parallel Approach for Training Large Models. In Proceedings of the 26th ACM SIG-PLAN Symposium on Principles and Practice of Parallel Programming, pp. 431–445, 2021. [find](https://scholar.google.com/scholar?q=Fan%2C+S%2E%2C+Rong%2C+Y%2E%2C+Meng%2C+C%2E%2C+Cao%2C+Z%2E%2C+Wang%2C+S%2E%2C+Zheng%2C+Z%2E%2C+Wu%2C+C%2E%2C+Long%2C+G%2E%2C+Yang%2C+J%2E%2C+Xia%2C+L%2E%2C+et+al%2E+DAP%2DPLE%3A+A+Pipelined+Data+Parallel+Approach+for+Training+Large+Models%2E+In+Proceedings+of+the+26th)
7. Griewank, A. and Walther, A. Revolve: An Implementation of Checkpointing for the Reverse or Adjoint Mode of Computational Differentiation. ACM Transactions on Mathematical Software (TOMS), 26(1):19–45, 2000. [find](https://scholar.google.com/scholar?q=Griewank%2C+A%2E+and+Walther%2C+A%2E+Revolve%3A+An+Implementation+of+Checkpointing+for+the+Reverse+or+Adjoint+Mode+of+Computational+Differentiation%2E+ACM+Transactions+on+Mathematical+Software+%28TOMS%29%2C+26%281%29%3A19%E2%80%9345)
8. Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., and Gibbons, P. [PipeDream: Fast and Efficient Pipeline Parallel DNN Training](https://md.research.elamur.ai/free/en/papers/harlap-2018-pipedream-fast-efficient-pipeline-parallel/). arXiv preprint arXiv:1806.03377, 2018.
9. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Advances in Neural Information Processing Systems, pp. 103–112, 2019. [find](https://scholar.google.com/scholar?q=Huang%2C+Y%2E%2C+Cheng%2C+Y%2E%2C+Bapna%2C+A%2E%2C+Firat%2C+O%2E%2C+Chen%2C+D%2E%2C+Chen%2C+M%2E%2C+Lee%2C+H%2E%2C+Ngiam%2C+J%2E%2C+Le%2C+Q%2E+V%2E%2C+Wu%2C+Y%2E%2C+et+al%2E+GPipe%3A+Efficient+Training+of+Giant+Neural+Networks+using+Pipeline+Parallelism%2E+In+Advances)
10. Jain, A., Phanishayee, A., Mars, J., Tang, L., and Pekhimenko, G. Gist: Efficient Data Encoding for Deep Neural Network Training. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA), pp. 776–789. [find](https://scholar.google.com/scholar?q=Jain%2C+A%2E%2C+Phanishayee%2C+A%2E%2C+Mars%2C+J%2E%2C+Tang%2C+L%2E%2C+and+Pekhimenko%2C+G%2E+Gist%3A+Efficient+Data+Encoding+for+Deep+Neural+Network+Training%2E+In+2018+ACM%2FIEEE+45th+Annual+International+Symposium+on+Computer+Archi)
11. Jain, P., Jain, A., Nrusimha, A., Gholami, A., Abbeel, P., Gonzalez, J., Keutzer, K., and Stoica, I. Breaking the Memory Wall with Optimal Tensor Rematerialization. In Proceedings of Machine Learning and Systems 2020, pp. 497–511. 2020. [find](https://scholar.google.com/scholar?q=Jain%2C+P%2E%2C+Jain%2C+A%2E%2C+Nrusimha%2C+A%2E%2C+Gholami%2C+A%2E%2C+Abbeel%2C+P%2E%2C+Gonzalez%2C+J%2E%2C+Keutzer%2C+K%2E%2C+and+Stoica%2C+I%2E+Breaking+the+Memory+Wall+with+Optimal+Tensor+Rematerialization%2E+In+Proceedings+of+Machine+Learning+)
12. Jia, Z., Zaharia, M., and Aiken, A. [Beyond Data and Model Parallelism for Deep Neural Networks](https://md.research.elamur.ai/free/en/papers/jia-2018-beyond-data-model-parallelism-deep/). In Proceedings of the 2nd Conference on Machine Learning and Systems (MLSys), 2018.
13. Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://md.research.elamur.ai/free/en/papers/jouppi-2017-datacenter-performance-analysis-tensor-processing/). In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1–12, 2017.
14. Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). arXiv preprint arXiv:1704.04683, 2017.
15. Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., and Zaharia, M. PipeDream: Generalized Pipeline Parallelism for DNN Training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pp. 1–15, 2019. [find](https://scholar.google.com/scholar?q=Narayanan%2C+D%2E%2C+Harlap%2C+A%2E%2C+Phanishayee%2C+A%2E%2C+Seshadri%2C+V%2E%2C+Devanur%2C+N%2E+R%2E%2C+Ganger%2C+G%2E+R%2E%2C+Gibbons%2C+P%2E+B%2E%2C+and+Zaharia%2C+M%2E+PipeDream%3A+Generalized+Pipeline+Parallelism+for+DNN+Training%2E+In+Proceedings+of)
16. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving Language Understanding by Generative Pre-training. URL [https://s3-us-west-2.amazonaws](https://s3-us-west-2.amazonaws). com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018. [↗](https://s3-us-west-2.amazonaws)
17. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language Models are Unsupervised Multi-task Learners. OpenAI Blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+Language+Models+are+Unsupervised+Multi%2Dtask+Learners%2E+OpenAI+Blog%2C+1%288%29%3A9%2C+2019%2E)
18. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. ZeRO: Memory Optimization Towards Training A Trillion Parameter Models. arXiv preprint arXiv:1910.02054, 2019. [find](https://scholar.google.com/scholar?q=Rajbhandari%2C+S%2E%2C+Rasley%2C+J%2E%2C+Ruwase%2C+O%2E%2C+and+He%2C+Y%2E+ZeRO%3A+Memory+Optimization+Towards+Training+A+Trillion+Parameter+Models%2E+arXiv+preprint+arXiv%3A1910%2E02054%2C+2019%2E)
19. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training Multi-Billion Parameter Language Models using GPU Model Parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Shoeybi%2C+M%2E%2C+Patwary%2C+M%2E%2C+Puri%2C+R%2E%2C+LeGresley%2C+P%2E%2C+Casper%2C+J%2E%2C+and+Catanzaro%2C+B%2E+Megatron%2DLM%3A+Training+Multi%2DBillion+Parameter+Language+Models+using+GPU+Model+Parallelism%2E+arXiv+preprint+arXiv%3A1909%2E08)
20. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). 2019. In the Proceedings of ICLR.
21. Xing, E. P., Ho, Q., Dai, W., Kim, J. K., Wei, J., Lee, S., Zheng, X., Xie, P., Kumar, A., and Yu, Y. Petuum: A New Platform for Distributed Machine Learning on Big Data. IEEE Transactions on Big Data, 1(2):49–67, 2015. [find](https://scholar.google.com/scholar?q=Xing%2C+E%2E+P%2E%2C+Ho%2C+Q%2E%2C+Dai%2C+W%2E%2C+Kim%2C+J%2E+K%2E%2C+Wei%2C+J%2E%2C+Lee%2C+S%2E%2C+Zheng%2C+X%2E%2C+Xie%2C+P%2E%2C+Kumar%2C+A%2E%2C+and+Yu%2C+Y%2E+Petuum%3A+A+New+Platform+for+Distributed+Machine+Learning+on+Big+Data%2E+IEEE+Transactions+on+Big+Data)
22. Yang, B., Zhang, J., Li, J., Ré, C., Aberger, C. R., and De Sa, C. PipeMare: Asynchronous Pipeline Parallel DNN Training. arXiv preprint arXiv:1910.05124, 2019. — [arXiv:1910.05124](https://arxiv.org/abs/1910.05124) [find](https://scholar.google.com/scholar?q=Yang%2C+B%2E%2C+Zhang%2C+J%2E%2C+Li%2C+J%2E%2C+Re%CC%81%2C+C%2E%2C+Aberger%2C+C%2E+R%2E%2C+and+De+Sa%2C+C%2E+PipeMare%3A+Asynchronous+Pipeline+Parallel+DNN+Training%2E+arXiv+preprint+arXiv%3A1910%2E05124%2C+2019%2E+%E2%80%94+%5BarXiv%3A1910%2E05124%5D%28https%3A%2F%2Farxiv%2Eorg)
23. A. Planner, Additional Details For every possible configuration of width and depth, PipeDream-2BW’s planner explores the benefit of pipelining and each space-saving optimization. For example, with activation recomputation as a target memory-savings optimization, PipeDream-2BW considers three possibl… [find](https://scholar.google.com/scholar?q=A%2E+Planner%2C+Additional+Details+For+every+possible+configuration+of+width+and+depth%2C+PipeDream%2D2BW%E2%80%99s+planner+explores+the+benefit+of+pipelining+and+each+space%2Dsaving+optimization%2E+For+example%2C+with+act)

*Updated 2 November 2026*