---
title: "ZeRO-Infinity — paper by Rajbhandari et al. (2021) · SC 2021"
description: "ZeRO-Infinity builds on ZeRO-3. 44 references, cited by 200 papers here. Among them: 54."
authors: ["Samyam Rajbhandari", "Olatunji Ruwase", "Jeff Rasley", "Shaden Smith", "Yuxiong He"]
year: 2021
venue: "SC 2021"
source: "https://arxiv.org/abs/2104.07857"
arxiv: "2104.07857"
licence: "CC BY 4.0 (arXiv OAI)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rajbhandari-2021-zero-infinity-breaking-gpu-memory/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rajbhandari-2021-zero-infinity-breaking-gpu-memory/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

# ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

**Authors:** [Samyam Rajbhandari](https://md.research.elamur.ai/free/en/authors/samyam-rajbhandari/), [Olatunji Ruwase](https://md.research.elamur.ai/free/en/authors/olatunji-ruwase/), [Jeff Rasley](https://md.research.elamur.ai/free/en/authors/jeff-rasley/), [Shaden Smith](https://md.research.elamur.ai/free/en/authors/shaden-smith/) and [Yuxiong He](https://md.research.elamur.ai/free/en/authors/yuxiong-he/) · **Year:** 2021 · **Venue:** SC 2021 · **Licence:** [CC BY 4.0 (arXiv OAI)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2104.07857) · [PDF](https://arxiv.org/pdf/2104.07857) · [arXiv 2104.07857](https://arxiv.org/abs/2104.07857) · [OpenAlex](https://openalex.org/W3156643189)

**Citations:** 54 citations (OpenAlex, 2026-10-14) · [200](https://md.research.elamur.ai/free/en/papers/rajbhandari-2021-zero-infinity-breaking-gpu-memory/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Dense model sizes have grown roughly a thousandfold in three years, while single-GPU memory has grown only about fivefold. Fitting models in aggregate GPU memory therefore demands huge clusters. Training a trillion-parameter model with 3D parallelism needs around 800 V100s, and that approach also forces heavy model code refactoring.

ZeRO-Infinity builds on ZeRO-3. It treats GPU, CPU and NVMe memory as one heterogeneous pool and combines five components:
- an infinity offload engine,
- memory-centric tiling for oversized layers,
- bandwidth-centric partitioning that draws on PCIe/memory bandwidth across all devices in parallel,
- an overlap-centric design with prefetching,
- an implementation that automates partitioning and communication, so models need no rewriting.

On 512 V100 GPUs (32 DGX-2 nodes), the system fits a 32-trillion-parameter model, about 50x beyond 3D parallelism. It sustains over 25 petaflops (roughly 40% of peak) and scales superlinearly. A single DGX-2 node can fine-tune a trillion-parameter model.

The practical message is that slow but abundant memory, accessed in parallel, can stand in for scarce HBM. This makes very large models accessible to teams without big clusters. The code is open source in DeepSpeed.

## Key points
- **Model scale:** 32T parameters on 512 V100 GPUs, which is 50x what 3D parallelism handles on the same hardware.
- **Throughput:** above 25 petaflops (about 40% of peak), with superlinear scaling shown for a 1T-parameter model.
- **Single-node scaling ladder (DGX-2):**
  - ZeRO-2 plus ZeRO-Offload reaches 13B parameters.
  - Adding CPU offload of parameters reaches nearly 100B.
  - NVMe offload reaches 1T, about 700x the data-parallel baseline.
- **Memory-centric tiling:** with GPU memory pre-fragmented into 2 GB chunks, the largest trainable hidden size rises from 8K to 64K at tiling factor 16.
- **Gradient offload:** spreading gradient offload across all GPUs' PCIe links gives nearly 2x faster backpropagation than ZeRO-Offload at 64 GPUs (8B model).
- **Activation checkpoint offload:** costs up to 1.2x throughput at small hidden sizes but is nearly free at 32K and 64K.

## Interesting ideas
- **Bandwidth-centric partitioning.** Split each parameter across all devices so every device reads its own share from slow memory at the same time, rather than having one owner fetch everything. Any pipeline bottlenecked on a single slow link, such as disk-backed embedding tables or large corpus shards, could gain from this.
- **Tiling a huge operator into sequential pieces.** Breaking an oversized layer into smaller tiles avoids tensor-slicing model parallelism altogether. The broader trick is to trade a little sequential execution for a much simpler system that needs no rewiring.
- **Fragmentation-controlled evaluation.** The authors pre-split GPU memory into fixed 2 GB blocks so that every run faces the same fragmentation conditions. This is a neat way to make memory-limit experiments reproducible and comparable.
- **Back-of-envelope bandwidth tables.** The authors estimate how much slow-memory and device-to-device bandwidth future, 10x or 100x faster accelerators would need. This sizing exercise helps decide which hardware upgrades actually matter before buying anything.
- **Prefetching and overlap where they count.** Overlapping communication with computation mattered most at small per-GPU batch sizes. That is a useful reminder to check which regime a workload is in before investing in overlap engineering.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (1): [⚡ GPU & parallel computing](https://md.research.elamur.ai/free/en/topics/gpu-parallel-computing/) · 41 works

## Cite

Cite as: Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith and Yuxiong He. 2021. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. SC 2021.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rajbhandari2021zero,
  title = {{ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning}},
  author = {Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith and Yuxiong He},
  booktitle = {SC 2021},
  year = {2021},
  eprint = {2104.07857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.07857}
}
```

- [← previous Generating Datasets with Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/schick-2021-generating-datasets-pretrained-language-models/)
- [next → ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation](https://md.research.elamur.ai/free/en/papers/malik-2021-ildc-cjpe-indian-legal-documents/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

44 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805, 2018.
2. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
3. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/), 2019.
4. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 2020.
5. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/), 2020.
6. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). arXiv preprint arXiv:1802.05365, 2018.
7. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2C+2019%2E)
8. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, and Phillip B. Gibbons. [Pipedream: Fast and efficient pipeline parallel DNN training](https://md.research.elamur.ai/free/en/papers/harlap-2018-pipedream-fast-efficient-pipeline-parallel/). CoRR, abs/1806.03377, 2018.
9. Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. ArXiv, abs/1811.06965, 2018. [find](https://scholar.google.com/scholar?q=Yanping+Huang%2C+Yonglong+Cheng%2C+Dehao+Chen%2C+HyoukJoong+Lee%2C+Jiquan+Ngiam%2C+Quoc+V%2E+Le%2C+and+Zhifeng+Chen%2E+Gpipe%3A+Efficient+training+of+giant+neural+networks+using+pipeline+parallelism%2E+ArXiv%2C+abs%2F1811%2E06)
10. Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. [Memory-efficient pipeline-parallel dnn training](https://md.research.elamur.ai/free/en/papers/narayanan-2020-memory-efficient-pipeline-parallel-dnn/). arXiv preprint arXiv:2006.09503, 2020.
11. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Optimizations toward Training Trillion Parameter Models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’20. IEEE Press, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+ZeRO%3A+Memory+Optimizations+toward+Training+Trillion+Parameter+Models%2E+In+Proceedings+of+the+International+Conference+for+High+Performa)
12. Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://md.research.elamur.ai/free/en/papers/ren-2021-zero-offload-democratizing-billion-scale/), 2021.
13. Microsoft. DeepSpeed: Extreme-scale model training for everyone. [https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/), 2020. [↗](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/)
14. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Software repository. [https://github.com/](https://github.com/) NVIDIA/Megatron-LM, 2021. [↗](https://github.com/)
15. DeepSpeed Team and Rangan Majumder. DeepSpeed: Extreme-scale model training for everyone. [https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/), 2020. [↗](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/)
16. Amir Gholami, Zhewei Yao, Sehoon Kim, Michael W. Mahoney, and Kurt Keutzer. Ai and memory wall. RiseLab Medium Post, 2021. [find](https://scholar.google.com/scholar?q=Amir+Gholami%2C+Zhewei+Yao%2C+Sehoon+Kim%2C+Michael+W%2E+Mahoney%2C+and+Kurt+Keutzer%2E+Ai+and+memory+wall%2E+RiseLab+Medium+Post%2C+2021%2E)
17. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. [Mesh-tensorflow: Deep learning for supercomputers](https://md.research.elamur.ai/free/en/papers/shazeer-2018-mesh-tensorflow-deep-learning-supercomputers/). CoRR, abs/1811.02084, 2018.
18. Minjie Wang, Chien-chin Huang, and Jinyang Li. Supporting very large models using automatic dataflow graph partitioning. In Proceedings of the Fourteenth EuroSys Conference 2019, EuroSys ’19, New York, NY, USA, 2019. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Minjie+Wang%2C+Chien%2Dchin+Huang%2C+and+Jinyang+Li%2E+Supporting+very+large+models+using+automatic+dataflow+graph+partitioning%2E+In+Proceedings+of+the+Fourteenth+EuroSys+Conference+2019%2C+EuroSys+%E2%80%9919%2C+New+York)
19. Tal Ben-Nun and Torsten Hoefler. [Demystifying parallel and distributed deep learning: An in-depth concurrency analysis](https://md.research.elamur.ai/free/en/papers/nun-2018-demystifying-parallel-distributed-deep-learning/). ACM Computing Surveys (CSUR), 52(4):1–43, 2019.
20. Mark Hildebrand, Jawad Khan, Sanjeev Trika, Jason Lowe-Power, and Venkatesh Akella. Autotm: Automatic tensor movement in heterogeneous memory systems using integer linear programming. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and O… [find](https://scholar.google.com/scholar?q=Mark+Hildebrand%2C+Jawad+Khan%2C+Sanjeev+Trika%2C+Jason+Lowe%2DPower%2C+and+Venkatesh+Akella%2E+Autotm%3A+Automatic+tensor+movement+in+heterogeneous+memory+systems+using+integer+linear+programming%2E+In+Proceedings+o)
21. Chien-Chin Huang, Gu Jin, and Jinyang Li. Swapadvisor: Pushing deep learning beyond the gpu memory limit via smart swapping. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 1341–1355, 2020. [find](https://scholar.google.com/scholar?q=Chien%2DChin+Huang%2C+Gu+Jin%2C+and+Jinyang+Li%2E+Swapadvisor%3A+Pushing+deep+learning+beyond+the+gpu+memory+limit+via+smart+swapping%2E+In+Proceedings+of+the+Twenty%2DFifth+International+Conference+on+Architectura)
22. Hai Jin, Bo Liu, Wenbin Jiang, Yang Ma, Xuanhua Shi, Bingsheng He, and Shaofeng Zhao. Layer-centric memory reuse and data migration for extreme-scale deep learning on many-core architectures. ACM Transactions on Architecture and Code Optimization (TACO), 15(3):1–26, 2018. [find](https://scholar.google.com/scholar?q=Hai+Jin%2C+Bo+Liu%2C+Wenbin+Jiang%2C+Yang+Ma%2C+Xuanhua+Shi%2C+Bingsheng+He%2C+and+Shaofeng+Zhao%2E+Layer%2Dcentric+memory+reuse+and+data+migration+for+extreme%2Dscale+deep+learning+on+many%2Dcore+architectures%2E+ACM+Tran)
23. Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. Capuchin: Tensor-based gpu memory management for deep learning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages… [find](https://scholar.google.com/scholar?q=Xuan+Peng%2C+Xuanhua+Shi%2C+Hulin+Dai%2C+Hai+Jin%2C+Weiliang+Ma%2C+Qian+Xiong%2C+Fan+Yang%2C+and+Xuehai+Qian%2E+Capuchin%3A+Tensor%2Dbased+gpu+memory+management+for+deep+learning%2E+In+Proceedings+of+the+Twenty%2DFifth+Inter)
24. Jie Ren, Jiaolin Luo, Kai Wu, Minjia Zhang, Hyeran Jeon, and Dong Li. Sentinel: Efficient tensor migration and allocation on heterogeneous memory systems for deep learning. In IEEE International Symposium on High Performance Computer Architecture, 2021. [find](https://scholar.google.com/scholar?q=Jie+Ren%2C+Jiaolin+Luo%2C+Kai+Wu%2C+Minjia+Zhang%2C+Hyeran+Jeon%2C+and+Dong+Li%2E+Sentinel%3A+Efficient+tensor+migration+and+allocation+on+heterogeneous+memory+systems+for+deep+learning%2E+In+IEEE+International+Sympo)
25. Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. [vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design](https://md.research.elamur.ai/free/en/papers/rhu-2016-vdnn-virtualized-deep-neural-networks/). In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 1–13. IEEE, 2016.
26. Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: Dynamic gpu memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN symposium on principles and practice of parallel programming, pages 41–53, 2018. [find](https://scholar.google.com/scholar?q=Linnan+Wang%2C+Jinmian+Ye%2C+Yiyang+Zhao%2C+Wei+Wu%2C+Ang+Li%2C+Shuaiwen+Leon+Song%2C+Zenglin+Xu%2C+and+Tim+Kraska%2E+Superneurons%3A+Dynamic+gpu+memory+management+for+training+deep+neural+networks%2E+In+Proceedings+of+t)
27. Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li. Distributed hierarchical gpu parameter server for massive scale deep learning ads systems, 2020. [find](https://scholar.google.com/scholar?q=Weijie+Zhao%2C+Deping+Xie%2C+Ronglai+Jia%2C+Yulei+Qian%2C+Ruiquan+Ding%2C+Mingming+Sun%2C+and+Ping+Li%2E+Distributed+hierarchical+gpu+parameter+server+for+massive+scale+deep+learning+ads+systems%2C+2020%2E)
28. Animesh Jain, Amar Phanishayee, Jason Mars, Lingjia Tang, and Gennady Pekhimenko. Gist: Efficient data encoding for deep neural network training. In International Symposium on Computer Architecture (ISCA 2018), 2018. [find](https://scholar.google.com/scholar?q=Animesh+Jain%2C+Amar+Phanishayee%2C+Jason+Mars%2C+Lingjia+Tang%2C+and+Gennady+Pekhimenko%2E+Gist%3A+Efficient+data+encoding+for+deep+neural+network+training%2E+In+International+Symposium+on+Computer+Architecture+%28I)
29. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. CoRR, abs/1604.06174, 2016. [find](https://scholar.google.com/scholar?q=Tianqi+Chen%2C+Bing+Xu%2C+Chiyuan+Zhang%2C+and+Carlos+Guestrin%2E+Training+deep+nets+with+sublinear+memory+cost%2E+CoRR%2C+abs%2F1604%2E06174%2C+2016%2E)
30. Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E. Gonzalez. Checkmate: Breaking the memory wall with optimal tensor rematerialization. ArXiv, abs/1910.02653, 2019. [find](https://scholar.google.com/scholar?q=Paras+Jain%2C+Ajay+Jain%2C+Aniruddha+Nrusimha%2C+Amir+Gholami%2C+Pieter+Abbeel%2C+Kurt+Keutzer%2C+Ion+Stoica%2C+and+Joseph+E%2E+Gonzalez%2E+Checkmate%3A+Breaking+the+memory+wall+with+optimal+tensor+rematerialization%2E+ArX)
31. Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: Dynamic GPU memory management for training deep neural networks. CoRR, abs/1801.04380, 2018. [find](https://scholar.google.com/scholar?q=Linnan+Wang%2C+Jinmian+Ye%2C+Yiyang+Zhao%2C+Wei+Wu%2C+Ang+Li%2C+Shuaiwen+Leon+Song%2C+Zenglin+Xu%2C+and+Tim+Kraska%2E+Superneurons%3A+Dynamic+GPU+memory+management+for+training+deep+neural+networks%2E+CoRR%2C+abs%2F1801%2E0438)
32. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12(null):2121–2159, July 2011. [find](https://scholar.google.com/scholar?q=John+Duchi%2C+Elad+Hazan%2C+and+Yoram+Singer%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+12%28null%29%3A2121%E2%80%932159%2C+July+2011%2E)
33. Diederik P. Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
34. Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size to 32k for imagenet training. CoRR, abs/1708.03888, 2017. — [arXiv:1708.03888](https://arxiv.org/abs/1708.03888) [find](https://scholar.google.com/scholar?q=Yang+You%2C+Igor+Gitman%2C+and+Boris+Ginsburg%2E+Scaling+SGD+batch+size+to+32k+for+imagenet+training%2E+CoRR%2C+abs%2F1708%2E03888%2C+2017%2E+%E2%80%94+%5BarXiv%3A1708%2E03888%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1708%2E03888%29)
35. Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Reducing BERT pre-training time from 3 days to 76 minutes. CoRR, abs/1904.00962, 2019. [find](https://scholar.google.com/scholar?q=Yang+You%2C+Jing+Li%2C+Jonathan+Hseu%2C+Xiaodan+Song%2C+James+Demmel%2C+and+Cho%2DJui+Hsieh%2E+Reducing+BERT+pre%2Dtraining+time+from+3+days+to+76+minutes%2E+CoRR%2C+abs%2F1904%2E00962%2C+2019%2E)
36. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training, 2017. [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Sharan+Narang%2C+Jonah+Alben%2C+Gregory+Diamos%2C+Erich+Elsen%2C+David+Garcia%2C+Boris+Ginsburg%2C+Michael+Houston%2C+Oleksii+Kuchaiev%2C+Ganesh+Venkatesh%2C+and+Hao+Wu%2E+Mixed+precision+training%2C+)
37. NVIDIA Tensor Cores. [https://www.nvidia.com/en-us/data-center/tensor-cores/](https://www.nvidia.com/en-us/data-center/tensor-cores/), 2018. [Online, accessed 5-April-2021]. [↗](https://www.nvidia.com/en-us/data-center/tensor-cores/)
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). CoRR, abs/1706.03762, 2017.
39. turing-nlg: A 17-billion-parameter language model by microsoft. [find](https://scholar.google.com/scholar?q=turing%2Dnlg%3A+A+17%2Dbillion%2Dparameter+language+model+by+microsoft%2E)
40. NVIDIA. NVIDIA DGX SuperPOD delivers world record supercomputing to any enterprise. [https://developer.nvidia.com/blog/dgx-superpod-world-record-supercomputing-enterprise/](https://developer.nvidia.com/blog/dgx-superpod-world-record-supercomputing-enterprise/), 2019. [↗](https://developer.nvidia.com/blog/dgx-superpod-world-record-supercomputing-enterprise/)
41. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
42. Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. [Pytorch distributed: Experiences on accelerating data parallel training](https://md.research.elamur.ai/free/en/papers/li-2020-pytorch-distributed-experiences-accelerating-data/). arXiv preprint arXiv:2006.15704, 2020.
43. Denis Foley and John Danskin. Ultra-performance pascal gpu and nvlink interconnect. IEEE Micro, 37(2):7–17, 2017. [find](https://scholar.google.com/scholar?q=Denis+Foley+and+John+Danskin%2E+Ultra%2Dperformance+pascal+gpu+and+nvlink+interconnect%2E+IEEE+Micro%2C+37%282%29%3A7%E2%80%9317%2C+2017%2E)
44. Oak Ridge National Laboratory. ORNL Launches Summit Supercomputer. [https://www.ornl.gov/news/ornl-launches-summit-supercomputer](https://www.ornl.gov/news/ornl-launches-summit-supercomputer), 2018. [Online; accessed 08-April-2021]. [↗](https://www.ornl.gov/news/ornl-launches-summit-supercomputer)

*Updated 2 November 2026*