---
title: "MXNet: A Flexible and Efficient Machine Learning — Chen 2015"
description: "Deep learning models keep growing in size and cost, and most frameworks commit to one programming style."
authors: ["Tianqi Chen", "Mu Li", "Yutian Li", "Min Lin", "Naiyan Wang", "Minjie Wang", "Tianjun Xiao", "Bing Xu", "Chiyuan Zhang", "Zheng Zhang"]
year: 2015
venue: "arXiv:1512.01274"
source: "https://arxiv.org/abs/1512.01274"
doi: "10.48550/arxiv.1512.01274"
arxiv: "1512.01274"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2015-mxnet-flexible-efficient-machine-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2015-mxnet-flexible-efficient-machine-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/) / MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

# MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

**Authors:** [Tianqi Chen](https://md.research.elamur.ai/free/en/authors/tianqi-chen/), [Mu Li](https://md.research.elamur.ai/free/en/authors/mu-li/), Yutian Li, [Min Lin](https://md.research.elamur.ai/free/en/authors/min-lin/), Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, [Zheng Zhang](https://md.research.elamur.ai/free/en/authors/zheng-zhang/) · **Year:** 2015 · **Venue:** arXiv:1512.01274 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1512.01274) · [PDF](https://arxiv.org/pdf/1512.01274) · [DOI 10.48550/arxiv.1512.01274](https://doi.org/10.48550/arxiv.1512.01274) · [arXiv 1512.01274](https://arxiv.org/abs/1512.01274) · [OpenAlex](https://openalex.org/W2186615578)

**Citations:** 1787 citations (openalex-snapshot · 2026-10-14) · [194](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Deep learning models keep growing in size and cost, and most frameworks commit to one programming style. Imperative code (numpy, Torch7) is flexible. Declarative graph definitions (Caffe, Theano, TensorFlow) allow global optimization. MXNet, built by researchers from several universities and companies, offers both styles in one library.

Users describe network structure with declarative Symbol graphs, which support automatic differentiation. Imperative NDArray operations handle data and parameter updates. KVStore, a two-level parameter server, keeps GPUs and machines in sync. All three run lazily through a single shared dependency engine that tracks both reads and writes of resources, so mixed code runs as fast as a purely symbolic program. Before execution, the graph is pruned, operators are fused, and memory is assigned by two linear-time reuse heuristics.

On standard convnet benchmarks, MXNet matches Torch7 and Caffe in speed. It also needs much less memory and scales super-linearly on a 10-machine GPU cluster. The work matters as a lightweight, portable toolkit with bindings for many languages, usable on hardware from phones up to distributed clusters.

## Key points
- Three core abstractions:
  - **Symbol**: declarative graphs with forward and backward passes.
  - **NDArray**: imperative tensors.
  - **KVStore**: push/pull synchronization with sequential or eventual consistency.
  
  Bindings exist for C++, Python, R, Go and Julia. The prediction code fits in one dependency-free C++ file of about 50K lines.
- Memory planning uses the "inplace" (reference counting) and "co-share" (sharing between nodes that cannot run in parallel) heuristics. Together they cut internal memory about 2x for training and 4x for prediction. VGG training needs under 16MB of extra memory.
- On convnet-benchmarks (GTX 980, batch size 32), MXNet's speed is comparable to Torch7 and Caffe. TensorFlow was about 2x slower, likely because it used an older CUDNN.
- GoogLeNet with batch normalization was trained on ILSVRC12 (1.3M images, 1,000 classes) using Amazon EC2 g2.8x instances with 4 GK104 GPUs each.
  - One data pass took about 14K s on 1 machine versus 1.4K s on 10 machines.
  - The distributed run overtook single-machine accuracy after about 10 passes, which the authors describe as super-linear speedup.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

194 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 188 not yet.

- [TVM: An Automated End-to-End Optimizing Compiler for Deep Learning](https://md.research.elamur.ai/free/en/papers/chen-2018-tvm-automated-end-end-optimizing/)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://md.research.elamur.ai/free/en/papers/gardner-2018-allennlp-deep-semantic-natural-language/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [DyNet: The Dynamic Neural Network Toolkit](https://md.research.elamur.ai/free/en/papers/neubig-2017-dynet-dynamic-neural-network-toolkit/)
- [Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2015mxnet,
  title = {{MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems}},
  author = {Chen, Tianqi and Li, Mu and Li, Yutian and Lin, Min and Wang, Naiyan and Wang, Minjie and Xiao, Tianjun and Xu, Bing and Zhang, Chiyuan and Zhang, Zheng},
  year = {2015},
  doi = {10.48550/arxiv.1512.01274},
  eprint = {1512.01274},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1512.01274}
}
```

- [← previous Multi-task Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
- [next → Named Entity Recognition with Bidirectional LSTM-CNNs](https://md.research.elamur.ai/free/en/papers/chiu-2015-named-entity-recognition-bidirectional-lstm/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/)

## References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley, and Yoshua Bengio. Theano: new features and speed improvements. arXiv preprint arXiv:1211.5590, 2012. [find](https://scholar.google.com/scholar?q=Fre%CC%81de%CC%81ric+Bastien%2C+Pascal+Lamblin%2C+Razvan+Pascanu%2C+James+Bergstra%2C+Ian+Goodfellow%2C+Arnaud+Bergeron%2C+Nicolas+Bouchard%2C+David+Warde%2DFarley%2C+and+Yoshua+Bengio%2E+Theano%3A+new+features+and+speed+improvement)
2. Soumith Chintala. Easy benchmarking of all public open-source implementations of convnets, 2015. [https://github.com/soumith/convnet-benchmarks](https://github.com/soumith/convnet-benchmarks). [↗](https://github.com/soumith/convnet-benchmarks)
3. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, number EPFL-CONF-192376, 2011. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Koray+Kavukcuoglu%2C+and+Cle%CC%81ment+Farabet%2E+Torch7%3A+A+matlab%2Dlike+environment+for+machine+learning%2E+In+BigLearn%2C+NIPS+Workshop%2C+number+EPFL%2DCONF%2D192376%2C+2011%2E)
4. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng. Large scale distributed deep networks. In Neural Information Processing Systems, 2012. [find](https://scholar.google.com/scholar?q=J%2E+Dean%2C+G%2E+Corrado%2C+R%2E+Monga%2C+K%2E+Chen%2C+M%2E+Devin%2C+Q%2E+Le%2C+M%2E+Mao%2C+M%2E+Ranzato%2C+A%2E+Senior%2C+P%2E+Tucker%2C+K%2E+Yang%2C+and+A%2E+Ng%2E+Large+scale+distributed+deep+networks%2E+In+Neural+Information+Processing+Systems%2C+)
5. Chainer Developers. Chainer: A powerful, flexible, and intuitive framework of neural networks, 2015. [http://chainer.org/](http://chainer.org/). [↗](http://chainer.org/)
6. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. [find](https://scholar.google.com/scholar?q=Sergey+Ioffe+and+Christian+Szegedy%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+arXiv+preprint+arXiv%3A1502%2E03167%2C+2015%2E)
7. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia, pages 675– 678. ACM, 2014. [find](https://scholar.google.com/scholar?q=Yangqing+Jia%2C+Evan+Shelhamer%2C+Jeff+Donahue%2C+Sergey+Karayev%2C+Jonathan+Long%2C+Ross+Girshick%2C+Sergio+Guadarrama%2C+and+Trevor+Darrell%2E+Caffe%3A+Convolutional+architecture+for+fast+feature+embedding%2E+In+Procee) [our card](https://md.research.elamur.ai/free/en/papers/jia-2014-caffe-convolutional-architecture-fast-feature/)
8. M. Li, D. G. Andersen, J. Park, A. J. Smola, A. Amhed, V. Josifovski, J. Long, E. Shekita, and B. Y. Su. Scaling distributed machine learning with the parameter server. In OSDI, 2014. [find](https://scholar.google.com/scholar?q=M%2E+Li%2C+D%2E+G%2E+Andersen%2C+J%2E+Park%2C+A%2E+J%2E+Smola%2C+A%2E+Amhed%2C+V%2E+Josifovski%2C+J%2E+Long%2C+E%2E+Shekita%2C+and+B%2E+Y%2E+Su%2E+Scaling+distributed+machine+learning+with+the+parameter+server%2E+In+OSDI%2C+2014%2E)
9. M. Li, D. G. Andersen, A. J. Smola, and K. Yu. Communication efficient distributed machine learning with the parameter server. In Neural Information Processing Systems, 2014. [find](https://scholar.google.com/scholar?q=M%2E+Li%2C+D%2E+G%2E+Andersen%2C+A%2E+J%2E+Smola%2C+and+K%2E+Yu%2E+Communication+efficient+distributed+machine+learning+with+the+parameter+server%2E+In+Neural+Information+Processing+Systems%2C+2014%2E)
10. Min Lin, Shuo Li, Xuan Luo, and Shuicheng Yan. Purine: A bi-graph based deep learning framework. arXiv preprint arXiv:1412.6249, 2014. — [arXiv:1412.6249](https://arxiv.org/abs/1412.6249) [find](https://scholar.google.com/scholar?q=Min+Lin%2C+Shuo+Li%2C+Xuan+Luo%2C+and+Shuicheng+Yan%2E+Purine%3A+A+bi%2Dgraph+based+deep+learning+framework%2E+arXiv+preprint+arXiv%3A1412%2E6249%2C+2014%2E+%E2%80%94+%5BarXiv%3A1412%2E6249%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1412%2E6249%29)
11. Abadi Martn, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, D… [find](https://scholar.google.com/scholar?q=Abadi+Martn%2C+Ashish+Agarwal%2C+Paul+Barham%2C+Eugene+Brevdo%2C+Zhifeng+Chen%2C+Craig+Citro%2C+Greg+Corrado%2C+Andy+Davis%2C+Jeffrey+Dean%2C+Matthieu+Devin%2C+Sanjay+Ghemawat%2C+Ian+Goodfellow%2C+Andrew+Harp%2C+Geoffrey+Irvin)
12. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 201… [find](https://scholar.google.com/scholar?q=Olga+Russakovsky%2C+Jia+Deng%2C+Hao+Su%2C+Jonathan+Krause%2C+Sanjeev+Satheesh%2C+Sean+Ma%2C+Zhiheng+Huang%2C+Andrej+Karpathy%2C+Aditya+Khosla%2C+Michael+Bernstein%2C+Alexander+C%2E+Berg%2C+and+Li+Fei%2DFei%2E+ImageNet+Large+Scal) [our card](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
13. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, pages 1–42, 2014. [find](https://scholar.google.com/scholar?q=Olga+Russakovsky%2C+Jia+Deng%2C+Hao+Su%2C+Jonathan+Krause%2C+Sanjeev+Satheesh%2C+Sean+Ma%2C+Zhiheng+Huang%2C+Andrej+Karpathy%2C+Aditya+Khosla%2C+Michael+Bernstein%2C+et+al%2E+Imagenet+large+scale+visual+recognition+challen) [our card](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
14. Minjie Wang, Tianjun Xiao, Jianpeng Li, Jiaxing Zhang, Chuntao Hong, and Zheng Zhang. Minerva: A scalable and highly efficient training platform for deep learning, 2014. [find](https://scholar.google.com/scholar?q=Minjie+Wang%2C+Tianjun+Xiao%2C+Jianpeng+Li%2C+Jiaxing+Zhang%2C+Chuntao+Hong%2C+and+Zheng+Zhang%2E+Minerva%3A+A+scalable+and+highly+efficient+training+platform+for+deep+learning%2C+2014%2E)