---
title: "Highly Scalable Deep Learning Training System with"
description: "Training large CNNs on ImageNet can take many hours even on several GPUs. 27 references, cited by 67 papers here."
authors: ["Xianyan Jia", "Shutao Song", "Wei He", "Yangzihao Wang", "Haidong Rong", "Feihu Zhou", "Liqiang Xie", "Zhenyu Guo", "Yuanzhou Yang", "Liwei Yu", "Tiegang Chen", "Guangxiao Hu", "Shaohuai Shi", "Xiaowen Chu"]
year: 2018
venue: "arXiv:1807.11205"
source: "https://arxiv.org/abs/1807.11205"
doi: "10.48550/arxiv.1807.11205"
arxiv: "1807.11205"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jia-2018-highly-scalable-deep-learning-training/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jia-2018-highly-scalable-deep-learning-training/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

# Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

**Authors:** Xianyan Jia, Shutao Song, [Wei He](https://md.research.elamur.ai/free/en/authors/wei-he/), [Yangzihao Wang](https://md.research.elamur.ai/free/en/authors/yangzihao-wang/), Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, Tiegang Chen, Guangxiao Hu, Shaohuai Shi and [Xiaowen Chu](https://md.research.elamur.ai/free/en/authors/xiaowen-chu/) · **Year:** 2018 · **Venue:** arXiv:1807.11205 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1807.11205) · [PDF](https://arxiv.org/pdf/1807.11205) · [DOI 10.48550/arxiv.1807.11205](https://doi.org/10.48550/arxiv.1807.11205) · [arXiv 1807.11205](https://arxiv.org/abs/1807.11205) · [OpenAlex](https://openalex.org/W2884711234)

**Citations:** 281 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/jia-2018-highly-scalable-deep-learning-training/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Training large CNNs on ImageNet can take many hours even on several GPUs. Scaling out with synchronous data-parallel SGD runs into two obstacles: very large mini-batches tend to reduce test accuracy, and gradient exchange becomes a bottleneck as the GPU count grows. The authors, from Tencent and Hong Kong Baptist University, built Jizhi, a training system for dense GPU clusters, to address both. It runs forward and backward passes in half precision but applies the LARS layer-wise learning-rate rule in FP32. It also uses several regularization tweaks that keep accuracy intact at 64K mini-batches, plus a hybrid all-reduce that combines their own adaptive collective with NCCL's ring all-reduce. With this setup, ResNet-50 reaches 75.8% top-1 in 6.6 minutes on 2048 Tesla P40 GPUs (90 epochs), and AlexNet reaches 58.7% in about 4 minutes on 1024 P40s (95 epochs). Both results beat earlier GPU and KNL systems in speed while matching or exceeding their accuracy. The work shows that extreme data parallelism does not require giving up generalization.

## Key points
- Mini-batch size was pushed to 64K for both AlexNet and ResNet-50 on ImageNet without dropping below baseline accuracy. Prior 64K attempts had reached only 73.2% and 73.9% for ResNet-50.
- An AlexNet ablation at 64K showed that three changes together raise top-1 accuracy from 55.8% to 58.8%: skipping weight decay on biases, skipping it on batch-norm parameters, and adding batch normalization after Pool5.
- Mixed precision raised single-node ResNet-50 throughput from 172 to 218 images/second.
- For ResNet-50 at 64 images per GPU, scaling efficiency on 1024 GPUs rose from 9.0% before optimization to 99.2%. At 32 per GPU it reached 87.9%, compared with 80.0% reported by Akiba et al.
- For AlexNet on 512 GPUs, efficiency rose from 70% with the FP32 all-reduce baseline to 91.4% once tensor fusion, FP16 all-reduce and hybrid all-reduce were combined. Overall speedups over NCCL-based training reached up to 3x for AlexNet and 11x for ResNet-50.

## Interesting ideas
- **Compute sensitive optimizer statistics in higher precision.** LARS computes norm ratios that can overflow or underflow FP16, which stalls training. The fix is to cast weights and gradients to FP32 only for the LARS step and then cast them back. This pattern of keeping bulk compute in low precision while protecting fragile scalar computations fits any low-precision pipeline with adaptive or normalized updates.
- **Break throughput into per-device speed × device count × scaling efficiency.** Framing system speed this way separates single-GPU work, such as FP16, from communication work, such as collectives and fusion. Teams can then target and measure each factor independently when planning training infrastructure.
- **Exempt biases and normalization parameters from weight decay at large batch sizes.** This is a cheap change that recovered noticeable accuracy at 64K. It is worth trying whenever large-batch training underperforms.
- **Run cumulative ablations of training tricks.** Turning strategies on one by one and reporting accuracy for each combination shows which ones actually matter. The same applies to the communication side, where FP32, fusion, FP16 and hybrid variants are measured separately. This evaluation style transfers well to any paper combining many small optimizations.
- **Mix collective algorithms instead of relying on one library.** Combining a custom adaptive all-reduce with NCCL's ring version gave large gains at scale. This suggests that communication strategies should be chosen according to tensor sizes and cluster topology.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

## Cite

Cite as: Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, Tiegang Chen, Guangxiao Hu, Shaohuai Shi and Xiaowen Chu. 2018. Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes. arXiv:1807.11205.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jia2018highly,
  title = {{Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes}},
  author = {Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, Tiegang Chen, Guangxiao Hu, Shaohuai Shi and Xiaowen Chu},
  year = {2018},
  doi = {10.48550/arxiv.1807.11205},
  eprint = {1807.11205},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1807.11205}
}
```

- [← previous T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples](https://md.research.elamur.ai/free/en/papers/elsahar-2018-t-rex-large-scale-alignment/)
- [next → TextWorld: A Learning Environment for Text-based Games](https://md.research.elamur.ai/free/en/papers/cote-2018-textworld-learning-environment-text-based/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, and others. 2016. [Tensorflow: a system for large-scale machine learning](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-system-large-scale-machine/).. In OSDI, Vol. 16. 265–283.
2. Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. 2017. Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325 (2017). [find](https://scholar.google.com/scholar?q=Takuya+Akiba%2C+Shuji+Suzuki%2C+and+Keisuke+Fukuda%2E+2017%2E+Extremely+large+minibatch+sgd%3A+Training+resnet%2D50+on+imagenet+in+15+minutes%2E+arXiv+preprint+arXiv%3A1711%2E04325+%282017%29%2E)
3. M. Barnett, L. Shuler, R. van de Geijn, S. Gupta, D. G. Payne, and J. Watts. 1994. Interprocessor collective communication library (InterCom). In Proceedings of IEEE Scalable High Performance Computing Conference. 357–364. https://doi.org/10.1109/SHPCC.1994.296665 — [doi:10.1109/shpcc.1994.296665](https://doi.org/10.1109/shpcc.1994.296665) [↗](https://doi.org/10.1109/SHPCC.1994.296665)
4. Minsik Cho, Ulrich Finkler, Sameer Kumar, David Kung, Vaibhav Saxena, and Dheeraj Sreedhar. 2017. PowerAI DDL. arXiv preprint arXiv:1708.02188 (2017). [find](https://scholar.google.com/scholar?q=Minsik+Cho%2C+Ulrich+Finkler%2C+Sameer+Kumar%2C+David+Kung%2C+Vaibhav+Saxena%2C+and+Dheeraj+Sreedhar%2E+2017%2E+PowerAI+DDL%2E+arXiv+preprint+arXiv%3A1708%2E02188+%282017%29%2E)
5. Valeriu Codreanu, Damian Podareanu, and Vikram Saletore. 2017. Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train. arXiv preprint arXiv:1711.04291 (2017). — [arXiv:1711.04291](https://arxiv.org/abs/1711.04291) [find](https://scholar.google.com/scholar?q=Valeriu+Codreanu%2C+Damian+Podareanu%2C+and+Vikram+Saletore%2E+2017%2E+Scale+out+for+large+minibatch+SGD%3A+Residual+network+training+on+ImageNet%2D1K+with+improved+accuracy+and+reduced+time+to+train%2E+arXiv+prepr)
6. Matthieu Courbariaux, Y Bengio, and Jean-Pierre David. 2015. Training deep neural networks with low precision multiplications. (12 2015). [find](https://scholar.google.com/scholar?q=Matthieu+Courbariaux%2C+Y+Bengio%2C+and+Jean%2DPierre+David%2E+2015%2E+Training+deep+neural+networks+with+low+precision+multiplications%2E+%2812+2015%29%2E)
7. Christopher De Sa, Megan Leszczynski, Jian Zhang, Alana Marzoev, Christopher R Aberger, Kunle Olukotun, and Christopher Ré. 2018. High-Accuracy Low-Precision Training. arXiv preprint arXiv:1803.03383 (2018). — [arXiv:1803.03383](https://arxiv.org/abs/1803.03383) [find](https://scholar.google.com/scholar?q=Christopher+De+Sa%2C+Megan+Leszczynski%2C+Jian+Zhang%2C+Alana+Marzoev%2C+Christopher+R+Aberger%2C+Kunle+Olukotun%2C+and+Christopher+R%C3%A9%2E+2018%2E+High%2DAccuracy+Low%2DPrecision+Training%2E+arXiv+preprint+arXiv%3A1803%2E03383+)
8. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. Ieee, 248–255. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+2009%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+Computer+Vision+and+Pattern+Recognition%2C+2009%2E+CVPR+2009%2E+IEEE+Conf)
9. Aditya Devarakonda, Maxim Naumov, and Michael Garland. 2017. AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks. arXiv preprint arXiv:1712.02029 (2017). [find](https://scholar.google.com/scholar?q=Aditya+Devarakonda%2C+Maxim+Naumov%2C+and+Michael+Garland%2E+2017%2E+AdaBatch%3A+Adaptive+Batch+Sizes+for+Training+Deep+Neural+Networks%2E+arXiv+preprint+arXiv%3A1712%2E02029+%282017%29%2E)
10. Andrew Gibiansky. 2017. Bringing HPC techniques to deep learning. [http://research.baidu.com/bringing-hpc-techniques-deep-learning](http://research.baidu.com/bringing-hpc-techniques-deep-learning) [↗](http://research.baidu.com/bringing-hpc-techniques-deep-learning)
11. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep learning. Vol. 1. MIT press Cambridge. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Yoshua+Bengio%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+2016%2E+Deep+learning%2E+Vol%2E+1%2E+MIT+press+Cambridge%2E)
12. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, large minibatch SGD: training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017). [find](https://scholar.google.com/scholar?q=Priya+Goyal%2C+Piotr+Doll%C3%A1r%2C+Ross+Girshick%2C+Pieter+Noordhuis%2C+Lukasz+Wesolowski%2C+Aapo+Kyrola%2C+Andrew+Tulloch%2C+Yangqing+Jia%2C+and+Kaiming+He%2E+2017%2E+Accurate%2C+large+minibatch+SGD%3A+training+imagenet+in+1+ho)
13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
14. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015). [find](https://scholar.google.com/scholar?q=Sergey+Ioffe+and+Christian+Szegedy%2E+2015%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+arXiv+preprint+arXiv%3A1502%2E03167+%282015%29%2E)
15. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2016. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 (2016). [find](https://scholar.google.com/scholar?q=Nitish+Shirish+Keskar%2C+Dheevatsa+Mudigere%2C+Jorge+Nocedal%2C+Mikhail+Smelyanskiy%2C+and+Ping+Tak+Peter+Tang%2E+2016%2E+On+large%2Dbatch+training+for+deep+learning%3A+Generalization+gap+and+sharp+minima%2E+arXiv+prep)
16. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In Advances in neural information processing systems. 1097–1105.
17. Anders Krogh and John A. Hertz. 1992. A Simple Weight Decay Can Improve Generalization. In Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann (Eds.). Morgan-Kaufmann, 950–957. [http://papers.nips.cc/paper/563-a-simple-weight-decay-can-improve-generaliza…](http://papers.nips.cc/paper/563-a-simple-weight-decay-can-improve-generaliza…) [↗](http://papers.nips.cc/paper/563-a-simple-weight-decay-can-improve-generaliza…)
18. Jianmin Chen Zhifeng Chen Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Geoffrey Irving Michael Isard Manjunath Kudlur Josh Levenberg Rajat Monga Sherry Moore Derek G. Murray Benoit Steiner Paul Tucker Vijay Vasudevan Pete Warden Martin Wicke Yuan Yu Xiaoqiang Zheng Martin Abadi, Paul Barha… — [TensorFlow: A system for large-scale machine learning](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-system-large-scale-machine/)
19. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaev, Ganesh Venkatesh, and others. 2017. Mixed precision training. arXiv preprint arXiv:1710.03740 (2017). [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Sharan+Narang%2C+Jonah+Alben%2C+Gregory+Diamos%2C+Erich+Elsen%2C+David+Garcia%2C+Boris+Ginsburg%2C+Michael+Houston%2C+Oleksii+Kuchaev%2C+Ganesh+Venkatesh%2C+and+others%2E+2017%2E+Mixed+precision+train)
20. Alexander Sergeev and Mike Del Balso. 2018. [Horovod: fast and easy distributed deep learning in TensorFlow](https://md.research.elamur.ai/free/en/papers/sergeev-2018-horovod-fast-easy-distributed-deep/). arXiv preprint arXiv:1802.05799 (2018).
21. Shaohuai Shi and Xiaowen Chu. 2017. Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs. arXiv preprint arXiv:1711.05979 (2017). [find](https://scholar.google.com/scholar?q=Shaohuai+Shi+and+Xiaowen+Chu%2E+2017%2E+Performance+Modeling+and+Evaluation+of+Distributed+Deep+Learning+Frameworks+on+GPUs%2E+arXiv+preprint+arXiv%3A1711%2E05979+%282017%29%2E)
22. Shaohuai Shi, Qiang Wang, Xiaowen Chu, and Bo Li. 2018. Modeling and Evaluation of Synchronous Stochastic Gradient Descent in Distributed Deep Learning on Multiple GPUs. arXiv preprint arXiv:1805.03812 (2018). [find](https://scholar.google.com/scholar?q=Shaohuai+Shi%2C+Qiang+Wang%2C+Xiaowen+Chu%2C+and+Bo+Li%2E+2018%2E+Modeling+and+Evaluation+of+Synchronous+Stochastic+Gradient+Descent+in+Distributed+Deep+Learning+on+Multiple+GPUs%2E+arXiv+preprint+arXiv%3A1805%2E0381)
23. Samuel L Smith, Pieter-Jan Kindermans, and Quoc V Le. 2017. Don’t Decay the Learning Rate, Increase the Batch Size. arXiv preprint arXiv:1711.00489 (2017). [find](https://scholar.google.com/scholar?q=Samuel+L+Smith%2C+Pieter%2DJan+Kindermans%2C+and+Quoc+V+Le%2E+2017%2E+Don%E2%80%99t+Decay+the+Learning+Rate%2C+Increase+the+Batch+Size%2E+arXiv+preprint+arXiv%3A1711%2E00489+%282017%29%2E)
24. Rajeev Thakur, Rolf Rabenseifner, and William Gropp. 2005. Optimization of collective communication operations in MPICH. The International Journal of High Performance Computing Applications 19, 1 (2005), 49–66. [find](https://scholar.google.com/scholar?q=Rajeev+Thakur%2C+Rolf+Rabenseifner%2C+and+William+Gropp%2E+2005%2E+Optimization+of+collective+communication+operations+in+MPICH%2E+The+International+Journal+of+High+Performance+Computing+Applications+19%2C+1+%28200)
25. Pijika Watcharapichat, Victoria Lopez Morales, Raul Castro Fernandez, and Peter Pietzuch. 2016. Ako: Decentralised deep learning with partial gradient exchange. In Proceedings of the Seventh ACM Symposium on Cloud Computing. ACM, 84–97. [find](https://scholar.google.com/scholar?q=Pijika+Watcharapichat%2C+Victoria+Lopez+Morales%2C+Raul+Castro+Fernandez%2C+and+Peter+Pietzuch%2E+2016%2E+Ako%3A+Decentralised+deep+learning+with+partial+gradient+exchange%2E+In+Proceedings+of+the+Seventh+ACM+Sympo)
26. Yang You, Igor Gitman, and Boris Ginsburg. 2017. Scaling SGD Batch Size to 32K for ImageNet Training. CoRR abs/1708.03888 (2017). arXiv:1708.03888 http://arxiv.org/abs/1708.03888 — [arXiv:1708.03888](https://arxiv.org/abs/1708.03888) [↗](http://arxiv.org/abs/1708.03888)
27. Yang You, Zhao Zhang, C Hsieh, James Demmel, and Kurt Keutzer. 2017. [ImageNet training in minutes](https://md.research.elamur.ai/free/en/papers/you-2017-imagenet-training-minutes/). CoRR, abs/1709.05011 (2017).

*Updated 2 November 2026*