---
title: "MegaScale — paper by Jiang et al. (2024) — summary and ideas"
description: "ByteDance and Peking University describe MegaScale, their production system for this setting. 69 references, cited by 134 papers here."
authors: ["Ziheng Jiang", "Haibin Lin", "Yinmin Zhong", "Qi Huang", "Yangrui Chen", "Zhi Zhang", "Yanghua Peng", "Xiang Li", "Cong Xie", "Shibiao Nong", "Yulu Jia", "Sun He", "Hongmin Chen", "Zhihao Bai", "Qi Hou", "Shipeng Yan", "Ding Zhou", "Yiyao Sheng", "Zhuo Jiang", "Haohan Xu", "Haoran Wei", "Zhang Zhang", "Pengfei Nie", "Leqi Zou", "Sida Zhao", "Liang Xiang", "Zherui Liu", "Zhe Li", "Xiaoying Jia", "Jianxi Ye", "Xin Jin", "Xin Liu"]
year: 2024
venue: "arXiv:2402.15627"
source: "https://arxiv.org/abs/2402.15627"
doi: "10.48550/arxiv.2402.15627"
arxiv: "2402.15627"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jiang-2024-megascale-scaling-large-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jiang-2024-megascale-scaling-large-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/) / MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

# MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

**Authors:** [Ziheng Jiang](https://md.research.elamur.ai/free/en/authors/ziheng-jiang/), [Haibin Lin](https://md.research.elamur.ai/free/en/authors/haibin-lin/), [Yinmin Zhong](https://md.research.elamur.ai/free/en/authors/yinmin-zhong/), Qi Huang, Yangrui Chen, [Zhi Zhang](https://md.research.elamur.ai/free/en/authors/zhi-zhang/), [Yanghua Peng](https://md.research.elamur.ai/free/en/authors/yanghua-peng/), [Xiang Li](https://md.research.elamur.ai/free/en/authors/xiang-li/), Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, [Haoran Wei](https://md.research.elamur.ai/free/en/authors/haoran-wei/), [Zhang Zhang](https://md.research.elamur.ai/free/en/authors/zhang-zhang/), Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, [Zhe Li](https://md.research.elamur.ai/free/en/authors/zhe-li/), Xiaoying Jia, Jianxi Ye, [Xin Jin](https://md.research.elamur.ai/free/en/authors/xin-jin/) and [Xin Liu](https://md.research.elamur.ai/free/en/authors/xin-liu/) · **Year:** 2024 · **Venue:** arXiv:2402.15627 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.15627) · [PDF](https://arxiv.org/pdf/2402.15627) · [DOI 10.48550/arxiv.2402.15627](https://doi.org/10.48550/arxiv.2402.15627) · [arXiv 2402.15627](https://arxiv.org/abs/2402.15627) · [OpenAlex](https://openalex.org/W4392223539)

**Citations:** 23 citations (OpenAlex, 2026-10-14) · [134](https://md.research.elamur.ai/free/en/papers/jiang-2024-megascale-scaling-large-language-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
When an LLM training job takes over a whole cluster of tens of thousands of GPUs, two engineering problems dominate. The first is keeping the hardware busy, measured as Model FLOPs Utilization (MFU). The second is keeping that efficiency steady for weeks, because hardware failures and slow nodes ("stragglers") happen constantly at this scale. ByteDance and Peking University describe MegaScale, their production system for this setting.

For efficiency, they tune the model and the infrastructure together:
- architecture and optimizer changes,
- a mix of four parallelism types, with custom overlap of communication and computation,
- a faster data pipeline and faster setup of communication groups,
- network tuning.

For stability, they rely on deep monitoring, automatic fault detection and recovery, and fine-grained tracing tools.

On a 175B model with 12,288 GPUs, MegaScale reaches 55.2% MFU, 1.34× better than Megatron-LM. A multi-week production run recovered automatically more than 100 times. The paper matters because most LLM reports leave out the systems work behind them.

## Key points
- **Headline result:** 55.2% MFU training a 175B transformer on 12,288 GPUs, a 1.34× improvement over Megatron-LM.
- **Model-level changes:** parallel transformer block, sliding window attention and the LAMB optimizer. The authors check that these changes do not harm convergence.
- **Parallelism:**
  - Data, pipeline (interleaved 1F1B), tensor and sequence parallelism are combined, with ZeRO-2-style sharding.
  - Tensor-parallel traffic stays inside a node.
  - Data-parallel groups are placed first, to limit traffic between minipods.
- **Systems tuning:**
  - Data pipeline: prefetching and tree-based loading.
  - Group setup: non-blocking initialization of collective groups without global barriers.
  - Network: custom topology, fewer ECMP hash conflicts, custom congestion control, tuned retransmit timeouts.
- **Production stability:** an in-house, closed model with several hundred billion parameters was trained for several weeks on multi-trillion tokens, recovering automatically more than 100 times. Some components are being open-sourced in veScale on GitHub.

## Interesting ideas
- **Diagnose slowdowns by elimination.** The team timed each training phase with CUDA events and saw that forward, backward and optimizer times stayed flat. They also found bandwidth was stable. That left only late-starting ranks in the gradient reduce-scatter as the cause. Splitting a slowdown into measured phases and ruling them out one by one works for any distributed pipeline, not just GPU training.
- **Place traffic where the links can carry it.** The chattiest operations (tensor parallelism) stay on the fastest links inside a node, and lighter traffic crosses node boundaries. Any multi-machine job can use the same rule: put tightly coupled work on the cheapest interconnect.
- **Plan for failure from the start.** MegaScale combines rich heartbeat messages, a suite of tests that pinpoint faulty nodes, and fast checkpoint/recovery, so recovery is routine rather than an emergency. This is useful for any long-running job, such as large corpus processing or crawling.
- **Remove global synchronization points.** Making setup steps asynchronous and dropping global barriers cut startup cost at scale. The same idea speeds up any system where many workers would otherwise wait for one another.
- **Visualize dependencies between workers.** A timeline, heat-map and 3D-parallel dependency view make it easy to spot which worker holds everyone else up. A lighter version of such tooling could help teams debug smaller multi-GPU fine-tuning runs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [⚡ GPU & parallel computing](https://md.research.elamur.ai/free/en/topics/gpu-parallel-computing/) · 41 works

## Cite

Cite as: Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin and Xin Liu. 2024. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs. arXiv:2402.15627.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jiang2024megascale,
  title = {{MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs}},
  author = {Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin and Xin Liu},
  year = {2024},
  doi = {10.48550/arxiv.2402.15627},
  eprint = {2402.15627},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.15627}
}
```

- [← previous MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models](https://md.research.elamur.ai/free/en/papers/wen-2024-mindmap-knowledge-graph-prompting-sparks/)
- [next → Moshi: a speech-text foundation model for real-time dialogue](https://md.research.elamur.ai/free/en/papers/defossez-2024-moshi-speech-text-foundation-model/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/2/)

## References

69 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “[Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/),” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
2. “Introducing chatgpt.” [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt), 2022. [↗](https://openai.com/blog/chatgpt)
3. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “[Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/),” 2020.
4. L. Floridi and M. Chiriatti, “Gpt-3: Its nature, scope, limits, and consequences,” Minds and Machines, vol. 30, pp. 681–694, 2020. [find](https://scholar.google.com/scholar?q=L%2E+Floridi+and+M%2E+Chiriatti%2C+%E2%80%9CGpt%2D3%3A+Its+nature%2C+scope%2C+limits%2C+and+consequences%2C%E2%80%9D+Minds+and+Machines%2C+vol%2E+30%2C+pp%2E+681%E2%80%93694%2C+2020%2E)
5. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al., “[Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/),” arXiv preprint arXiv:2204.02311, 2022.
6. K. He, X. Zhang, S. Ren, and J. Sun, “[Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/),” in IEEE Conference on Computer Vision and Pattern Recognition, 2016.
7. D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. A. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on gpu clusters using megatron-lm,” 2021. [find](https://scholar.google.com/scholar?q=D%2E+Narayanan%2C+M%2E+Shoeybi%2C+J%2E+Casper%2C+P%2E+LeGresley%2C+M%2E+Patwary%2C+V%2E+A%2E+Korthikanti%2C+D%2E+Vainbrand%2C+P%2E+Kashinkunti%2C+J%2E+Bernauer%2C+B%2E+Catanzaro%2C+A%2E+Phanishayee%2C+and+M%2E+Zaharia%2C+%E2%80%9CEfficient+large%2Dscale+langua)
8. I. Beltagy, M. E. Peters, and A. Cohan, “[Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/),” 2020.
9. Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh, “Large batch optimization for deep learning: Training bert in 76 minutes,” in International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=Y%2E+You%2C+J%2E+Li%2C+S%2E+Reddi%2C+J%2E+Hseu%2C+S%2E+Kumar%2C+S%2E+Bhojanapalli%2C+X%2E+Song%2C+J%2E+Demmel%2C+K%2E+Keutzer%2C+and+C%2E%2DJ%2E+Hsieh%2C+%E2%80%9CLarge+batch+optimization+for+deep+learning%3A+Training+bert+in+76+minutes%2C%E2%80%9D+in+Internationa)
10. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020. [find](https://scholar.google.com/scholar?q=M%2E+Shoeybi%2C+M%2E+Patwary%2C+R%2E+Puri%2C+P%2E+LeGresley%2C+J%2E+Casper%2C+and+B%2E+Catanzaro%2C+%E2%80%9CMegatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2C%E2%80%9D+2020%2E)
11. S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “Zero: Memory optimizations toward training trillion parameter models.” ArXiv, May 2020. [find](https://scholar.google.com/scholar?q=S%2E+Rajbhandari%2C+J%2E+Rasley%2C+O%2E+Ruwase%2C+and+Y%2E+He%2C+%E2%80%9CZero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E%E2%80%9D+ArXiv%2C+May+2020%2E)
12. Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, et al., “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism ,” in NeurIPS, 2019. [find](https://scholar.google.com/scholar?q=Y%2E+Huang%2C+Y%2E+Cheng%2C+A%2E+Bapna%2C+O%2E+Firat%2C+D%2E+Chen%2C+M%2E+Chen%2C+H%2E+Lee%2C+J%2E+Ngiam%2C+Q%2E+V%2E+Le%2C+Y%2E+Wu%2C+et+al%2E%2C+%E2%80%9CGPipe%3A+Efficient+Training+of+Giant+Neural+Networks+using+Pipeline+Parallelism+%2C%E2%80%9D+in+NeurIPS%2C+2019%2E)
13. D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: Generalized pipeline parallelism for dnn training,” in ACM SOSP, 2019. [find](https://scholar.google.com/scholar?q=D%2E+Narayanan%2C+A%2E+Harlap%2C+A%2E+Phanishayee%2C+V%2E+Seshadri%2C+N%2E+R%2E+Devanur%2C+G%2E+R%2E+Ganger%2C+P%2E+B%2E+Gibbons%2C+and+M%2E+Zaharia%2C+%E2%80%9CPipedream%3A+Generalized+pipeline+parallelism+for+dnn+training%2C%E2%80%9D+in+ACM+SOSP%2C+2019%2E)
14. B. Wang and A. Komatsuzaki, “GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.” [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), May 2021. [↗](https://github.com/kingoflolz/mesh-transformer-jax)
15. Y. Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y. Hao, A. Mathews, and S. Li, “Pytorch fsdp: Experiences on scaling fully sharded data parallel,” 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Zhao%2C+A%2E+Gu%2C+R%2E+Varma%2C+L%2E+Luo%2C+C%2E%2DC%2E+Huang%2C+M%2E+Xu%2C+L%2E+Wright%2C+H%2E+Shojanazeri%2C+M%2E+Ott%2C+S%2E+Shleifer%2C+A%2E+Desmaison%2C+C%2E+Balioglu%2C+P%2E+Damania%2C+B%2E+Nguyen%2C+G%2E+Chauhan%2C+Y%2E+Hao%2C+A%2E+Mathews%2C+and+S%2E+Li%2C+%E2%80%9CPyto)
16. T. Dao, “[Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/),” arXiv preprint arXiv:2307.08691, 2023.
17. S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, P. Damania, and S. Chintala, “[Pytorch distributed: Experiences on accelerating data parallel training](https://md.research.elamur.ai/free/en/papers/li-2020-pytorch-distributed-experiences-accelerating-data/),” 2020.
18. I. Group, “Ieee 802.1 qbb - priority-based flow control.” [https://1.ieee802.org/dcb/802-1qbb/](https://1.ieee802.org/dcb/802-1qbb/), 2009. [↗](https://1.ieee802.org/dcb/802-1qbb/)
19. Y. Zhu, H. Eran, D. Firestone, C. Guo, M. Lipshteyn, Y. Liron, J. Padhye, S. Raindel, M. H. Yahia, and M. Zhang, “Congestion Control for Large-scale RDMA Deployments,” ACM SIGCOMM Computer Communication Review, vol. 45, no. 4, pp. 523–536, 2015. [find](https://scholar.google.com/scholar?q=Y%2E+Zhu%2C+H%2E+Eran%2C+D%2E+Firestone%2C+C%2E+Guo%2C+M%2E+Lipshteyn%2C+Y%2E+Liron%2C+J%2E+Padhye%2C+S%2E+Raindel%2C+M%2E+H%2E+Yahia%2C+and+M%2E+Zhang%2C+%E2%80%9CCongestion+Control+for+Large%2Dscale+RDMA+Deployments%2C%E2%80%9D+ACM+SIGCOMM+Computer+Communicati)
20. G. Kumar, N. Dukkipati, K. Jang, H. M. Wassel, X. Wu, B. Montazeri, Y. Wang, K. Springborn, C. Alfeld, M. Ryan, et al., “Swift: Delay is Simple and Effective for Congestion Control in the Datacenter,” in SIGCOMM, pp. 514–528, 2020. [https://github.com/NVIDIA/](https://github.com/NVIDIA/) [↗](https://github.com/NVIDIA/)
21. “Megatron-LM.” Megatron-LM/tree/main, 2021. [find](https://scholar.google.com/scholar?q=%E2%80%9CMegatron%2DLM%2E%E2%80%9D+Megatron%2DLM%2Ftree%2Fmain%2C+2021%2E)
22. V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “[Reducing activation recomputation in large transformer models](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/),” Proceedings of Machine Learning and Systems, vol. 5, 2023.
23. OpenAI, “[Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/),” 2023.
24. D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” 2020. [find](https://scholar.google.com/scholar?q=D%2E+Lepikhin%2C+H%2E+Lee%2C+Y%2E+Xu%2C+D%2E+Chen%2C+O%2E+Firat%2C+Y%2E+Huang%2C+M%2E+Krikun%2C+N%2E+Shazeer%2C+and+Z%2E+Chen%2C+%E2%80%9CGshard%3A+Scaling+giant+models+with+conditional+computation+and+automatic+sharding%2C%E2%80%9D+2020%2E)
25. A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. Das-Sarma, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, J. Kernion, K. Ndousse, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, and J. Kaplan, “A general language assistant as a laborator… [find](https://scholar.google.com/scholar?q=A%2E+Askell%2C+Y%2E+Bai%2C+A%2E+Chen%2C+D%2E+Drain%2C+D%2E+Ganguli%2C+T%2E+Henighan%2C+A%2E+Jones%2C+N%2E+Joseph%2C+B%2E+Mann%2C+N%2E+Das%2DSarma%2C+N%2E+Elhage%2C+Z%2E+Hatfield%2DDodds%2C+D%2E+Hernandez%2C+J%2E+Kernion%2C+K%2E+Ndousse%2C+C%2E+Olsson%2C+D%2E+Amodei%2C+T%2E+)
26. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “[Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/),” 2022.
27. S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, E. Zhang, R. Child, R. Y. Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y. He, M. Houston, S. Tiwary, and B. Catanzaro, “Using deepspeed and megatron to train megatron-turing n… — [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://md.research.elamur.ai/free/en/papers/smith-2022-using-deepspeed-megatron-train-megatron/)
28. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training compute-op… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
29. H. Su, X. Zhou, H. Yu, X. Shen, Y. Chen, Z. Zhu, Y. Yu, and J. Zhou, “Welm: A well-read pre-trained language model for chinese,” 2023. [find](https://scholar.google.com/scholar?q=H%2E+Su%2C+X%2E+Zhou%2C+H%2E+Yu%2C+X%2E+Shen%2C+Y%2E+Chen%2C+Z%2E+Zhu%2C+Y%2E+Yu%2C+and+J%2E+Zhou%2C+%E2%80%9CWelm%3A+A+well%2Dread+pre%2Dtrained+language+model+for+chinese%2C%E2%80%9D+2023%2E)
30. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. ChenG, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “[Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/),” 2022.
31. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al., “Bloom: A 176b-parameter open-access multilingual language model,” arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=T%2E+L%2E+Scao%2C+A%2E+Fan%2C+C%2E+Akiki%2C+E%2E+Pavlick%2C+S%2E+Ilic%CC%81%2C+D%2E+Hesslow%2C+R%2E+Castagn%C3%A9%2C+A%2E+S%2E+Luccioni%2C+F%2E+Yvon%2C+M%2E+Gall%C3%A9%2C+et+al%2E%2C+%E2%80%9CBloom%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model%2C%E2%80%9D+arXiv+preprin)
32. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “[Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/),” 2023.
33. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
34. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model.” [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
35. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” 2023. [find](https://scholar.google.com/scholar?q=W%2E%2DL%2E+Chiang%2C+Z%2E+Li%2C+Z%2E+Lin%2C+Y%2E+Sheng%2C+Z%2E+Wu%2C+H%2E+Zhang%2C+L%2E+Zheng%2C+S%2E+Zhuang%2C+Y%2E+Zhuang%2C+J%2E+E%2E+Gonzalez%2C+I%2E+Stoica%2C+and+E%2E+P%2E+Xing%2C+%E2%80%9CVicuna%3A+An+open%2Dsource+chatbot+impressing+gpt%2D4+with+90%25%2A+chatgpt+qu)
36. X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song, “Koala: A dialogue model for academic research.” Blog post, April 2023. [find](https://scholar.google.com/scholar?q=X%2E+Geng%2C+A%2E+Gudibande%2C+H%2E+Liu%2C+E%2E+Wallace%2C+P%2E+Abbeel%2C+S%2E+Levine%2C+and+D%2E+Song%2C+%E2%80%9CKoala%3A+A+dialogue+model+for+academic+research%2E%E2%80%9D+Blog+post%2C+April+2023%2E)
37. Y. Ji, Y. Deng, Y. Gong, Y. Peng, Q. Niu, B. Ma, and X. Li, “Belle: Be everyone’s large language model engine.” [https://github.com/LianjiaTech/BELLE](https://github.com/LianjiaTech/BELLE), 2023. [↗](https://github.com/LianjiaTech/BELLE)
38. Z. Li, E. Wallace, S. Shen, K. Lin, K. Keutzer, D. Klein, and J. Gonzalez, “Train big, then compress: Rethinking model size for efficient training and inference of transformers,” in International Conference on Machine Learning (ICML), 2020. [find](https://scholar.google.com/scholar?q=Z%2E+Li%2C+E%2E+Wallace%2C+S%2E+Shen%2C+K%2E+Lin%2C+K%2E+Keutzer%2C+D%2E+Klein%2C+and+J%2E+Gonzalez%2C+%E2%80%9CTrain+big%2C+then+compress%3A+Rethinking+model+size+for+efficient+training+and+inference+of+transformers%2C%E2%80%9D+in+International+Conf)
39. G. Xiao, J. Lin, M. Seznec, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” arXiv, 2022. [find](https://scholar.google.com/scholar?q=G%2E+Xiao%2C+J%2E+Lin%2C+M%2E+Seznec%2C+J%2E+Demouth%2C+and+S%2E+Han%2C+%E2%80%9CSmoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2C%E2%80%9D+arXiv%2C+2022%2E)
40. E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv, 2022. [find](https://scholar.google.com/scholar?q=E%2E+Frantar%2C+S%2E+Ashkboos%2C+T%2E+Hoefler%2C+and+D%2E+Alistarh%2C+%E2%80%9CGptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2C%E2%80%9D+arXiv%2C+2022%2E)
41. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Llm. int8 (): 8-bit matrix multiplication for transformers at scale,” arXiv, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Dettmers%2C+M%2E+Lewis%2C+Y%2E+Belkada%2C+and+L%2E+Zettlemoyer%2C+%E2%80%9CLlm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2C%E2%80%9D+arXiv%2C+2022%2E)
42. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “[Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/),” 2021.
43. R. Child, S. Gray, A. Radford, and I. Sutskever, “[Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/),” 2019.
44. A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are rnns: Fast autoregressive transformers with linear attention,” 2020. [find](https://scholar.google.com/scholar?q=A%2E+Katharopoulos%2C+A%2E+Vyas%2C+N%2E+Pappas%2C+and+F%2E+Fleuret%2C+%E2%80%9CTransformers+are+rnns%3A+Fast+autoregressive+transformers+with+linear+attention%2C%E2%80%9D+2020%2E)
45. C. Zhu, W. Ping, C. Xiao, M. Shoeybi, T. Goldstein, A. Anandkumar, and B. Catanzaro, “Long-short transformer: Efficient transformers for language and vision,” 2021. [find](https://scholar.google.com/scholar?q=C%2E+Zhu%2C+W%2E+Ping%2C+C%2E+Xiao%2C+M%2E+Shoeybi%2C+T%2E+Goldstein%2C+A%2E+Anandkumar%2C+and+B%2E+Catanzaro%2C+%E2%80%9CLong%2Dshort+transformer%3A+Efficient+transformers+for+language+and+vision%2C%E2%80%9D+2021%2E)
46. B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV, X. He, H. Hou, P. Kazienko, J. Kocon, J. Kong, B. Koptyra, H. Lau, K. S. I. Mantri, F. Mom, A. Saito, X. Tang, B. Wang, J. S. Wind, S. Wozniak, R. Zhang, Z. Zhang, Q. Zhao, P. Zhou, J. Zhu, an… [find](https://scholar.google.com/scholar?q=B%2E+Peng%2C+E%2E+Alcaide%2C+Q%2E+Anthony%2C+A%2E+Albalak%2C+S%2E+Arcadinho%2C+H%2E+Cao%2C+X%2E+Cheng%2C+M%2E+Chung%2C+M%2E+Grella%2C+K%2E+K%2E+GV%2C+X%2E+He%2C+H%2E+Hou%2C+P%2E+Kazienko%2C+J%2E+Kocon%2C+J%2E+Kong%2C+B%2E+Koptyra%2C+H%2E+Lau%2C+K%2E+S%2E+I%2E+Mantri%2C+F%2E+Mom%2C+)
47. Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei, “[Retentive network: A successor to transformer for large language models](https://md.research.elamur.ai/free/en/papers/sun-2023-retentive-network-successor-transformer-large/),” 2023.
48. D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, “[QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding](https://md.research.elamur.ai/free/en/papers/alistarh-2016-qsgd-communication-efficient-sgd-gradient/),” in NeurIPS, 2017.
49. P. Micikevicius, S. Narang, J. Alben, G. F. Diamos, E. Elsen, D. García, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed Precision Training,” in ICLR, 2018. [find](https://scholar.google.com/scholar?q=P%2E+Micikevicius%2C+S%2E+Narang%2C+J%2E+Alben%2C+G%2E+F%2E+Diamos%2C+E%2E+Elsen%2C+D%2E+Garc%C3%ADa%2C+B%2E+Ginsburg%2C+M%2E+Houston%2C+O%2E+Kuchaiev%2C+G%2E+Venkatesh%2C+and+H%2E+Wu%2C+%E2%80%9CMixed+Precision+Training%2C%E2%80%9D+in+ICLR%2C+2018%2E)
50. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., “[Tensorflow: A system for large-scale machine learning](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-system-large-scale-machine/),” in OSDI, 2016.
51. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “[PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/) ,” in NeurIPS, 2019.
52. A. Jayarajan, J. Wei, G. Gibson, A. Fedorova, and G. Pekhimenko, “Priority-based Parameter Propagation for Distributed DNN Training ,” in MLSys, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Jayarajan%2C+J%2E+Wei%2C+G%2E+Gibson%2C+A%2E+Fedorova%2C+and+G%2E+Pekhimenko%2C+%E2%80%9CPriority%2Dbased+Parameter+Propagation+for+Distributed+DNN+Training+%2C%E2%80%9D+in+MLSys%2C+2019%2E)
53. S. H. Hashemi, S. Abdu Jyothi, and R. Campbell, “Tic-Tac: Accelerating Distributed Deep Learning with Communication Scheduling,” in MLSys, 2019. [find](https://scholar.google.com/scholar?q=S%2E+H%2E+Hashemi%2C+S%2E+Abdu+Jyothi%2C+and+R%2E+Campbell%2C+%E2%80%9CTic%2DTac%3A+Accelerating+Distributed+Deep+Learning+with+Communication+Scheduling%2C%E2%80%9D+in+MLSys%2C+2019%2E)
54. Y. Peng, Y. Zhu, Y. Chen, Y. Bao, B. Yi, C. Lan, C. Wu, and C. Guo, “A generic communication scheduler for distributed DNN training acceleration,” in SOSP, 2019. [find](https://scholar.google.com/scholar?q=Y%2E+Peng%2C+Y%2E+Zhu%2C+Y%2E+Chen%2C+Y%2E+Bao%2C+B%2E+Yi%2C+C%2E+Lan%2C+C%2E+Wu%2C+and+C%2E+Guo%2C+%E2%80%9CA+generic+communication+scheduler+for+distributed+DNN+training+acceleration%2C%E2%80%9D+in+SOSP%2C+2019%2E)
55. Y. Bao, Y. Peng, Y. Chen, and C. Wu, “Preemptive All-reduce Scheduling for Expediting Distributed DNN Training,” in INFOCOM, 2020. [find](https://scholar.google.com/scholar?q=Y%2E+Bao%2C+Y%2E+Peng%2C+Y%2E+Chen%2C+and+C%2E+Wu%2C+%E2%80%9CPreemptive+All%2Dreduce+Scheduling+for+Expediting+Distributed+DNN+Training%2C%E2%80%9D+in+INFOCOM%2C+2020%2E)
56. Y. Li, M. Yu, S. Li, S. Avestimehr, N. S. Kim, and A. Schwing, “Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training,” in NeurIPS, 2018. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+M%2E+Yu%2C+S%2E+Li%2C+S%2E+Avestimehr%2C+N%2E+S%2E+Kim%2C+and+A%2E+Schwing%2C+%E2%80%9CPipe%2DSGD%3A+A+Decentralized+Pipelined+SGD+Framework+for+Distributed+Deep+Net+Training%2C%E2%80%9D+in+NeurIPS%2C+2018%2E)
57. Y. Chen, C. Xie, M. Ma, J. Gu, Y. Peng, H. Lin, C. Wu, and Y. Zhu, “Sapipe: Staleness-aware pipeline for data parallel dnn training,” Advances in Neural Information Processing Systems, vol. 35, pp. 17981–17993, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Chen%2C+C%2E+Xie%2C+M%2E+Ma%2C+J%2E+Gu%2C+Y%2E+Peng%2C+H%2E+Lin%2C+C%2E+Wu%2C+and+Y%2E+Zhu%2C+%E2%80%9CSapipe%3A+Staleness%2Daware+pipeline+for+data+parallel+dnn+training%2C%E2%80%9D+Advances+in+Neural+Information+Processing+Systems%2C+vol%2E+35%2C+pp%2E+17)
58. C. Guo, L. Yuan, D. Xiang, Y. Dang, R. Huang, D. Maltz, Z. Liu, V. Wang, B. Pang, H. Chen, Z.-W. Lin, and V. Kurien, “Pingmesh: A large-scale system for data center network latency measurement and analysis,” SIG-COMM Comput. Commun. Rev., vol. 45, p. 139–152, aug 2015. [find](https://scholar.google.com/scholar?q=C%2E+Guo%2C+L%2E+Yuan%2C+D%2E+Xiang%2C+Y%2E+Dang%2C+R%2E+Huang%2C+D%2E+Maltz%2C+Z%2E+Liu%2C+V%2E+Wang%2C+B%2E+Pang%2C+H%2E+Chen%2C+Z%2E%2DW%2E+Lin%2C+and+V%2E+Kurien%2C+%E2%80%9CPingmesh%3A+A+large%2Dscale+system+for+data+center+network+latency+measurement+and+ana)
59. Y. Zhu, N. Kang, J. Cao, A. Greenberg, G. Lu, R. Mahajan, D. Maltz, L. Yuan, M. Zhang, B. Y. Zhao, and H. Zheng, “Packet-level telemetry in large datacenter networks,” SIGCOMM Comput. Commun. Rev., vol. 45, p. 479–491, aug 2015. [find](https://scholar.google.com/scholar?q=Y%2E+Zhu%2C+N%2E+Kang%2C+J%2E+Cao%2C+A%2E+Greenberg%2C+G%2E+Lu%2C+R%2E+Mahajan%2C+D%2E+Maltz%2C+L%2E+Yuan%2C+M%2E+Zhang%2C+B%2E+Y%2E+Zhao%2C+and+H%2E+Zheng%2C+%E2%80%9CPacket%2Dlevel+telemetry+in+large+datacenter+networks%2C%E2%80%9D+SIGCOMM+Comput%2E+Commun%2E+Rev%2E%2C+vo)
60. Y. Li, R. Miao, C. Kim, and M. Yu, “Lossradar: Fast detection of lost packets in data center networks,” in Proceedings of the 12th International on Conference on Emerging Networking EXperiments and Technologies, CoNEXT ’16, (New York, NY, USA), p. 481–495, Association for Computing Machinery, 2016. [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+R%2E+Miao%2C+C%2E+Kim%2C+and+M%2E+Yu%2C+%E2%80%9CLossradar%3A+Fast+detection+of+lost+packets+in+data+center+networks%2C%E2%80%9D+in+Proceedings+of+the+12th+International+on+Conference+on+Emerging+Networking+EXperiments+and+Te)
61. C. Tan, Z. Jin, C. Guo, T. Zhang, H. Wu, K. Deng, D. Bi, and D. Xiang, “Netbouncer: Active device and link failure localization in data center networks,” in Proceedings of the 16th USENIX Conference on Networked Systems Design and Implementation, NSDI’19, (USA), p. 599–613, USENIX Association, 2019. [find](https://scholar.google.com/scholar?q=C%2E+Tan%2C+Z%2E+Jin%2C+C%2E+Guo%2C+T%2E+Zhang%2C+H%2E+Wu%2C+K%2E+Deng%2C+D%2E+Bi%2C+and+D%2E+Xiang%2C+%E2%80%9CNetbouncer%3A+Active+device+and+link+failure+localization+in+data+center+networks%2C%E2%80%9D+in+Proceedings+of+the+16th+USENIX+Conference+o)
62. K. Liu, Z. Jiang, J. Zhang, H. Wei, X. Zhong, L. Tan, T. Pan, and T. Huang, “Hostping: Diagnosing intra-host network bottlenecks in RDMA servers,” in 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), (Boston, MA), pp. 15–29, USENIX Association, April 2023. [find](https://scholar.google.com/scholar?q=K%2E+Liu%2C+Z%2E+Jiang%2C+J%2E+Zhang%2C+H%2E+Wei%2C+X%2E+Zhong%2C+L%2E+Tan%2C+T%2E+Pan%2C+and+T%2E+Huang%2C+%E2%80%9CHostping%3A+Diagnosing+intra%2Dhost+network+bottlenecks+in+RDMA+servers%2C%E2%80%9D+in+20th+USENIX+Symposium+on+Networked+Systems+Design+)
63. S. Haider, N. R. Ansari, M. Akbar, and M. R. Perwez, “Fault tolerance in distributed paradigms,” 2011. [find](https://scholar.google.com/scholar?q=S%2E+Haider%2C+N%2E+R%2E+Ansari%2C+M%2E+Akbar%2C+and+M%2E+R%2E+Perwez%2C+%E2%80%9CFault+tolerance+in+distributed+paradigms%2C%E2%80%9D+2011%2E)
64. Y. Peng, Y. Bao, Y. Chen, C. Wu, and C. Guo, “Optimus: an efficient dynamic resource scheduler for deep learning clusters,” in Proceedings of the Thirteenth EuroSys Conference, pp. 1–14, 2018. [find](https://scholar.google.com/scholar?q=Y%2E+Peng%2C+Y%2E+Bao%2C+Y%2E+Chen%2C+C%2E+Wu%2C+and+C%2E+Guo%2C+%E2%80%9COptimus%3A+an+efficient+dynamic+resource+scheduler+for+deep+learning+clusters%2C%E2%80%9D+in+Proceedings+of+the+Thirteenth+EuroSys+Conference%2C+pp%2E+1%E2%80%9314%2C+2018%2E)
65. A. S. Tanenbaum, Distributed systems principles and paradigms. 2007. [find](https://scholar.google.com/scholar?q=A%2E+S%2E+Tanenbaum%2C+Distributed+systems+principles+and+paradigms%2E+2007%2E)
66. S. Chakravorty, C. L. Mendes, and L. V. Kalé, “Proactive fault tolerance in mpi applications via task migration,” in International Conference on High-Performance Computing, pp. 485–496, Springer, 2006. [find](https://scholar.google.com/scholar?q=S%2E+Chakravorty%2C+C%2E+L%2E+Mendes%2C+and+L%2E+V%2E+Kal%C3%A9%2C+%E2%80%9CProactive+fault+tolerance+in+mpi+applications+via+task+migration%2C%E2%80%9D+in+International+Conference+on+High%2DPerformance+Computing%2C+pp%2E+485%E2%80%93496%2C+Springer%2C+2006)
67. S. Chakravorty, C. Mendes, and L. V. Kale, “Proactive fault tolerance in large systems,” in HPCRI Workshop in conjunction with HPCA, vol. 2005, pp. 1–7, Citeseer, 2005. [find](https://scholar.google.com/scholar?q=S%2E+Chakravorty%2C+C%2E+Mendes%2C+and+L%2E+V%2E+Kale%2C+%E2%80%9CProactive+fault+tolerance+in+large+systems%2C%E2%80%9D+in+HPCRI+Workshop+in+conjunction+with+HPCA%2C+vol%2E+2005%2C+pp%2E+1%E2%80%937%2C+Citeseer%2C+2005%2E)
68. Y. Chen, Y. Peng, Y. Bao, C. Wu, Y. Zhu, and C. Guo, “Elastic parameter server load distribution in deep learning clusters,” in Proceedings of the 11th ACM Symposium on Cloud Computing, pp. 507–521, 2020. [find](https://scholar.google.com/scholar?q=Y%2E+Chen%2C+Y%2E+Peng%2C+Y%2E+Bao%2C+C%2E+Wu%2C+Y%2E+Zhu%2C+and+C%2E+Guo%2C+%E2%80%9CElastic+parameter+server+load+distribution+in+deep+learning+clusters%2C%E2%80%9D+in+Proceedings+of+the+11th+ACM+Symposium+on+Cloud+Computing%2C+pp%2E+507%E2%80%93521%2C+2)
69. I. Behera and C. R. Tripathy, “Performance modelling and analysis of mobile grid computing systems,” International Journal of Grid and Utility Computing, vol. 5, no. 1, pp. 11–20, 2014. [find](https://scholar.google.com/scholar?q=I%2E+Behera+and+C%2E+R%2E+Tripathy%2C+%E2%80%9CPerformance+modelling+and+analysis+of+mobile+grid+computing+systems%2C%E2%80%9D+International+Journal+of+Grid+and+Utility+Computing%2C+vol%2E+5%2C+no%2E+1%2C+pp%2E+11%E2%80%9320%2C+2014%2E)

*Updated 2 November 2026*