---
title: "Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN"
description: "The authors describe Philly, Microsoft's internal training service built on Apache YARN. 56 references, cited by 112 papers here."
authors: ["Myeongjae Jeon", "Shivaram Venkataraman", "Amar Phanishayee", "Junjie Qian", "Wencong Xiao", "Fan Yang"]
year: 2019
venue: "arXiv:1901.05758"
source: "https://arxiv.org/abs/1901.05758"
doi: "10.48550/arxiv.1901.05758"
arxiv: "1901.05758"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jeon-2019-analysis-large-scale-multi-tenant/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jeon-2019-analysis-large-scale-multi-tenant/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads

# Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads

**Authors:** Myeongjae Jeon, [Shivaram Venkataraman](https://md.research.elamur.ai/free/en/authors/shivaram-venkataraman/), [Amar Phanishayee](https://md.research.elamur.ai/free/en/authors/amar-phanishayee/), Junjie Qian, [Wencong Xiao](https://md.research.elamur.ai/free/en/authors/wencong-xiao/) and [Fan Yang](https://md.research.elamur.ai/free/en/authors/fan-yang/) · **Year:** 2019 · **Venue:** arXiv:1901.05758 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1901.05758) · [PDF](https://arxiv.org/pdf/1901.05758) · [DOI 10.48550/arxiv.1901.05758](https://doi.org/10.48550/arxiv.1901.05758) · [arXiv 1901.05758](https://arxiv.org/abs/1901.05758) · [OpenAlex](https://openalex.org/W2910100551)

**Citations:** 67 citations (OpenAlex, 2026-10-14) · [112](https://md.research.elamur.ai/free/en/papers/jeon-2019-analysis-large-scale-multi-tenant/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper examines how deep learning training behaves on a shared, production GPU cluster at Microsoft. Such clusters differ from classic big-data clusters in two ways: a GPU cannot easily be split among users, and multi-GPU jobs must have all their GPUs allocated at once (gang scheduling). The authors describe Philly, Microsoft's internal training service built on Apache YARN. They analyze a two-month trace of about 100,000 jobs from hundreds of users by combining scheduler logs, Ganglia utilization metrics and per-job logs. They focus on three issues:
- how gang scheduling and locality requirements create queueing delay,
- how job placement affects actual GPU utilization,
- why jobs fail.

GPUs that are allocated are busy only about half the time on average. Roughly 30% of jobs end unsuccessfully. From these findings the authors derive design guidelines for future DNN cluster schedulers and publish the scheduler trace, which matters because public data on real ML training workloads is scarce.

## Key points
- **Cluster setup:** Philly runs on YARN with HDFS storage and Docker containers. Each production group gets a virtual cluster with a GPU quota, managed by the Fair Scheduler. Preemption starts only once about 90% of all GPUs are in use.
- **Locality-aware placement:** The scheduler tries to put a job's GPUs on as few servers as possible, inside one RDMA domain (100-Gbps InfiniBand within a rack, Ethernet across racks). It waits a timeout of 2–3 minutes before relaxing these constraints.
- **Cluster growth:** The cluster grew fivefold in one year, and servers moved from 2-GPU to 8-GPU configurations.
- **Low real utilization:** Average hardware utilization of allocated GPUs is only around 52%. The authors attribute this to two causes:
  - jobs spread across servers pay synchronization costs,
  - jobs packed onto the same server interfere with each other.
- **Failures:** About 30% of jobs are killed or fail.
  - Programming errors dominate and appear early in training.
  - Infrastructure failures, such as HDFS problems, tend to surface much later.

## Interesting ideas
- **Look past allocation numbers.** A cluster where nearly every GPU is assigned can still waste much of its hardware. Tracking actual device utilization alongside allocation is a cheap check that applies to any shared compute pool.
- **Prefer locality over a faster start.** For long-running jobs, it can pay to wait longer in the queue in exchange for good placement, since poor placement slows the whole run. The same reasoning fits any batch system where startup delay is small relative to runtime.
- **Smoke-test jobs before scaling.** Run each new job briefly on a small dedicated pool, or even a single GPU, before giving it many shared GPUs. This catches code and configuration bugs cheaply and fits any expensive experiment pipeline, including NLP training runs.
- **Retry based on failure type.** Classify failures at runtime and adjust retries accordingly: stop retrying on bad inputs, keep retrying on network timeouts. This avoids wasting resources on reruns that are certain to fail.
- **Join several log sources and release the trace.** Linking scheduler, monitoring and per-job logs gives a full view of each job's lifecycle. Publishing the resulting anonymized trace lets others test scheduling ideas on real workloads.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations
- [Beihang University](https://md.research.elamur.ai/free/en/universities/beihang-university/) — University, Beijing, China · 86 works · 9,097 citations
- [Microsoft Research (United Kingdom)](https://md.research.elamur.ai/free/en/companies/microsoft-research-united-kingdom/) — Company, Cambridge, United Kingdom · 11 works · 3,215 citations
- [University of Wisconsin System](https://md.research.elamur.ai/free/en/universities/university-of-wisconsin-system/) — University, Madison, United States · 2 works · 179 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 100 works

🧭 Topics (1): [⚡ GPU & parallel computing](https://md.research.elamur.ai/free/en/topics/gpu-parallel-computing/) · 41 works

## Cite

Cite as: Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao and Fan Yang. 2019. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. arXiv:1901.05758.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jeon2019analysis,
  title = {{Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads}},
  author = {Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao and Fan Yang},
  year = {2019},
  doi = {10.48550/arxiv.1901.05758},
  eprint = {1901.05758},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1901.05758}
}
```

- [← previous Representation Degeneration Problem in Training Natural Language Generation Models](https://md.research.elamur.ai/free/en/papers/gao-2019-representation-degeneration-problem-training-natural/)
- [next → Generalized Data Augmentation for Low-Resource Translation](https://md.research.elamur.ai/free/en/papers/xia-2019-generalized-data-augmentation-low-resource/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

56 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Deep Learning for Siri’s Voice. [https://machinelearning.apple.com/2017/08/06/siri-voices.html](https://machinelearning.apple.com/2017/08/06/siri-voices.html). [↗](https://machinelearning.apple.com/2017/08/06/siri-voices.html)
2. Hadoop: Fair Scheduler. [https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/FairScheduler.html](https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/FairScheduler.html). [↗](https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/FairScheduler.html)
3. NVIDIA Management Library. [https://developer.nvidia.com/nvidia-management-library-nvml](https://developer.nvidia.com/nvidia-management-library-nvml). [↗](https://developer.nvidia.com/nvidia-management-library-nvml)
4. Using Deep Learning to Create Professional-Level Photographs. [https://research.googleblog.com/2017/07/using-deep-learning-to-create.html](https://research.googleblog.com/2017/07/using-deep-learning-to-create.html). [↗](https://research.googleblog.com/2017/07/using-deep-learning-to-create.html)
5. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. [TensorFlow: A System for Large-Scale Machine Learning](https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-system-large-scale-machine/). In OSDI, 2016.
6. Sameer Agarwal, Barzan Mozafari, Aurojit Panda, Henry Milner, Samuel Madden, and Ion Stoica. BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data. In EuroSys, 2013. [find](https://scholar.google.com/scholar?q=Sameer+Agarwal%2C+Barzan+Mozafari%2C+Aurojit+Panda%2C+Henry+Milner%2C+Samuel+Madden%2C+and+Ion+Stoica%2E+BlinkDB%3A+Queries+with+Bounded+Errors+and+Bounded+Response+Times+on+Very+Large+Data%2E+In+EuroSys%2C+2013%2E)
7. Ganesh Ananthanarayanan, Sameer Agarwal, Srikanth Kandula, Albert Greenberg, Ion Stoica, Duke Harlan, and Ed Harris. Scarlett: Coping with Skewed Content Popularity in Mapreduce Clusters. In EuroSys, 2011. [find](https://scholar.google.com/scholar?q=Ganesh+Ananthanarayanan%2C+Sameer+Agarwal%2C+Srikanth+Kandula%2C+Albert+Greenberg%2C+Ion+Stoica%2C+Duke+Harlan%2C+and+Ed+Harris%2E+Scarlett%3A+Coping+with+Skewed+Content+Popularity+in+Mapreduce+Clusters%2E+In+EuroSys%2C+)
8. Ganesh Ananthanarayanan, Michael Chien-Chun Hung, Xiaoqi Ren, Ion Stoica, Adam Wierman, and Minlan Yu. GRASS: Trimming Stragglers in Approximation Analytics. In NSDI, 2014. [find](https://scholar.google.com/scholar?q=Ganesh+Ananthanarayanan%2C+Michael+Chien%2DChun+Hung%2C+Xiaoqi+Ren%2C+Ion+Stoica%2C+Adam+Wierman%2C+and+Minlan+Yu%2E+GRASS%3A+Trimming+Stragglers+in+Approximation+Analytics%2E+In+NSDI%2C+2014%2E)
9. Brian Babcock, Surajit Chaudhuri, and Gautam Das. Dynamic Sample Selection for Approximate Query Processing. In SIGMOD, 2003. [find](https://scholar.google.com/scholar?q=Brian+Babcock%2C+Surajit+Chaudhuri%2C+and+Gautam+Das%2E+Dynamic+Sample+Selection+for+Approximate+Query+Processing%2E+In+SIGMOD%2C+2003%2E)
10. Luiz André Barroso, Jeffrey Dean, and Urs Hölzle. Web Search for a Planet: The Google Cluster Architecture. IEEE Micro, 23(2):22–28, March 2003. [find](https://scholar.google.com/scholar?q=Luiz+Andr%C3%A9+Barroso%2C+Jeffrey+Dean%2C+and+Urs+H%C3%B6lzle%2E+Web+Search+for+a+Planet%3A+The+Google+Cluster+Architecture%2E+IEEE+Micro%2C+23%282%29%3A22%E2%80%9328%2C+March+2003%2E)
11. Eric Boutin, Jaliya Ekanayake, Wei Lin, Bing Shi, Jingren Zhou, Zhengping Qian, Ming Wu, and Lidong Zhou. Apollo: Scalable and Coordinated Scheduling for Cloud-scale Computing. In OSDI, 2014. [find](https://scholar.google.com/scholar?q=Eric+Boutin%2C+Jaliya+Ekanayake%2C+Wei+Lin%2C+Bing+Shi%2C+Jingren+Zhou%2C+Zhengping+Qian%2C+Ming+Wu%2C+and+Lidong+Zhou%2E+Apollo%3A+Scalable+and+Coordinated+Scheduling+for+Cloud%2Dscale+Computing%2E+In+OSDI%2C+2014%2E)
12. Ronnie Chaiken, Bob Jenkins, Per-Åke Larson, Bill Ramsey, Darren Shakib, Simon Weaver, and Jingren Zhou. SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. VLDB, 2008. [find](https://scholar.google.com/scholar?q=Ronnie+Chaiken%2C+Bob+Jenkins%2C+Per%2D%C3%85ke+Larson%2C+Bill+Ramsey%2C+Darren+Shakib%2C+Simon+Weaver%2C+and+Jingren+Zhou%2E+SCOPE%3A+Easy+and+Efficient+Parallel+Processing+of+Massive+Data+Sets%2E+VLDB%2C+2008%2E)
13. Quan Chen, Hailong Yang, Minyi Guo, Ram Srivatsa Kannan, Jason Mars, and Lingjia Tang. Prophet: Precise QoS Prediction on Non-Preemptive Accelerators to Improve Utilization in Warehouse-Scale Computers. In ASPLOS, 2017. [find](https://scholar.google.com/scholar?q=Quan+Chen%2C+Hailong+Yang%2C+Minyi+Guo%2C+Ram+Srivatsa+Kannan%2C+Jason+Mars%2C+and+Lingjia+Tang%2E+Prophet%3A+Precise+QoS+Prediction+on+Non%2DPreemptive+Accelerators+to+Improve+Utilization+in+Warehouse%2DScale+Computer)
14. Quan Chen, Hailong Yang, Jason Mars, and Lingjia Tang. Baymax: QoS Awareness and Increased Utilization for Non-Preemptive Accelerators in Warehouse Scale Computers. In ASPLOS, 2016. [find](https://scholar.google.com/scholar?q=Quan+Chen%2C+Hailong+Yang%2C+Jason+Mars%2C+and+Lingjia+Tang%2E+Baymax%3A+QoS+Awareness+and+Increased+Utilization+for+Non%2DPreemptive+Accelerators+in+Warehouse+Scale+Computers%2E+In+ASPLOS%2C+2016%2E)
15. Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. [MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://md.research.elamur.ai/free/en/papers/chen-2015-mxnet-flexible-efficient-machine-learning/). arXiv preprint arXiv:1512.01274, 2015.
16. Tyson Condie, Neil Conway, Peter Alvaro, Joseph M. Hellerstein, John Gerth, Justin Talbot, Khaled Elmeleegy, and Russell Sears. Online Aggregation and Continuous Query Support in MapReduce. In SIGMOD, 2010. [find](https://scholar.google.com/scholar?q=Tyson+Condie%2C+Neil+Conway%2C+Peter+Alvaro%2C+Joseph+M%2E+Hellerstein%2C+John+Gerth%2C+Justin+Talbot%2C+Khaled+Elmeleegy%2C+and+Russell+Sears%2E+Online+Aggregation+and+Continuous+Query+Support+in+MapReduce%2E+In+SIGMOD%2C)
17. Jeffrey Dean and Luiz André Barroso. The Tail at Scale. Commun. ACM, 56(2):74–80, February 2013. [find](https://scholar.google.com/scholar?q=Jeffrey+Dean+and+Luiz+Andr%C3%A9+Barroso%2E+The+Tail+at+Scale%2E+Commun%2E+ACM%2C+56%282%29%3A74%E2%80%9380%2C+February+2013%2E)
18. Dror G Feitelson. Packing schemes for gang scheduling. In Workshop on Job Scheduling Strategies for Parallel Processing, pages 89–110. Springer, 1996. [find](https://scholar.google.com/scholar?q=Dror+G+Feitelson%2E+Packing+schemes+for+gang+scheduling%2E+In+Workshop+on+Job+Scheduling+Strategies+for+Parallel+Processing%2C+pages+89%E2%80%93110%2E+Springer%2C+1996%2E)
19. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep Learning, volume 1. MIT press Cambridge, 2016. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Yoshua+Bengio%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+Deep+Learning%2C+volume+1%2E+MIT+press+Cambridge%2C+2016%2E)
20. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint arXiv:1706.02677, 2017. [find](https://scholar.google.com/scholar?q=Priya+Goyal%2C+Piotr+Doll%C3%A1r%2C+Ross+Girshick%2C+Pieter+Noordhuis%2C+Lukasz+Wesolowski%2C+Aapo+Kyrola%2C+Andrew+Tulloch%2C+Yangqing+Jia%2C+and+Kaiming+He%2E+Accurate%2C+Large+Minibatch+SGD%3A+Training+ImageNet+in+1+Hour%2E+ar)
21. Juncheng Gu, Kang G. Chowdhury, Mosharaf abd Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. Tiresias: A GPU Cluster Manager for Distributed Deep Learning. In NSDI, 2019. [find](https://scholar.google.com/scholar?q=Juncheng+Gu%2C+Kang+G%2E+Chowdhury%2C+Mosharaf+abd+Shin%2C+Yibo+Zhu%2C+Myeongjae+Jeon%2C+Junjie+Qian%2C+Hongqiang+Liu%2C+and+Chuanxiong+Guo%2E+Tiresias%3A+A+GPU+Cluster+Manager+for+Distributed+Deep+Learning%2E+In+NSDI%2C+201)
22. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep Residual Learning for Image Recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In CVPR, 2016.
23. Joseph M. Hellerstein, Peter J. Haas, and Helen J. Wang. Online Aggregation. In SIGMOD, 1997. [find](https://scholar.google.com/scholar?q=Joseph+M%2E+Hellerstein%2C+Peter+J%2E+Haas%2C+and+Helen+J%2E+Wang%2E+Online+Aggregation%2E+In+SIGMOD%2C+1997%2E)
24. ImageNet, 2016. [http://image-net.org](http://image-net.org). [↗](http://image-net.org)
25. Michael Isard, Vijayan Prabhakaran, Jon Currey, Udi Wieder, Kunal Talwar, and Andrew Goldberg. Quincy: Fair Scheduling for Distributed Computing Clusters. In SOSP, 2009. [find](https://scholar.google.com/scholar?q=Michael+Isard%2C+Vijayan+Prabhakaran%2C+Jon+Currey%2C+Udi+Wieder%2C+Kunal+Talwar%2C+and+Andrew+Goldberg%2E+Quincy%3A+Fair+Scheduling+for+Distributed+Computing+Clusters%2E+In+SOSP%2C+2009%2E)
26. Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. Multi-tenant GPU Clusters for Deep Learning Workloads: Analysis and Implications. Technical Report MSR-TR-2018-13, 2018. [find](https://scholar.google.com/scholar?q=Myeongjae+Jeon%2C+Shivaram+Venkataraman%2C+Amar+Phanishayee%2C+Junjie+Qian%2C+Wencong+Xiao%2C+and+Fan+Yang%2E+Multi%2Dtenant+GPU+Clusters+for+Deep+Learning+Workloads%3A+Analysis+and+Implications%2E+Technical+Report+MSR)
27. Chris Jermaine, Subramanian Arumugam, Abhijit Pol, and Alin Dobra. Scalable Approximate Query Processing with the DBO Engine. ACM Trans. Database Syst., 33(4):23:1–23:54, December 2008. [find](https://scholar.google.com/scholar?q=Chris+Jermaine%2C+Subramanian+Arumugam%2C+Abhijit+Pol%2C+and+Alin+Dobra%2E+Scalable+Approximate+Query+Processing+with+the+DBO+Engine%2E+ACM+Trans%2E+Database+Syst%2E%2C+33%284%29%3A23%3A1%E2%80%9323%3A54%2C+December+2008%2E)
28. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. [Caffe: Convolutional Architecture for Fast Feature Embedding](https://md.research.elamur.ai/free/en/papers/jia-2014-caffe-convolutional-architecture-fast-feature/). In MM, 2014.
29. Soila Kavulya, Jiaqi Tan, Rajeev Gandhi, and Priya Narasimhan. An Analysis of Traces from a Production MapReduce Cluster. In CCGRID ’10, 2010. [find](https://scholar.google.com/scholar?q=Soila+Kavulya%2C+Jiaqi+Tan%2C+Rajeev+Gandhi%2C+and+Priya+Narasimhan%2E+An+Analysis+of+Traces+from+a+Production+MapReduce+Cluster%2E+In+CCGRID+%E2%80%9910%2C+2010%2E)
30. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. [ImageNet Classification with Deep Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In NIPS, 2012.
31. Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. Scaling Distributed Machine Learning with the Parameter Server. In OSDI, 2014. [find](https://scholar.google.com/scholar?q=Mu+Li%2C+David+G+Andersen%2C+Jun+Woo+Park%2C+Alexander+J+Smola%2C+Amr+Ahmed%2C+Vanja+Josifovski%2C+James+Long%2C+Eugene+J+Shekita%2C+and+Bor%2DYiing+Su%2E+Scaling+Distributed+Machine+Learning+with+the+Parameter+Server%2E+I)
32. Matthew L Massie, Brent N Chun, and David E Culler. The Ganglia Distributed Monitoring System: Design, Implementation And Experience. Parallel Computing, 30(7):817–840, 2004. [find](https://scholar.google.com/scholar?q=Matthew+L+Massie%2C+Brent+N+Chun%2C+and+David+E+Culler%2E+The+Ganglia+Distributed+Monitoring+System%3A+Design%2C+Implementation+And+Experience%2E+Parallel+Computing%2C+30%287%29%3A817%E2%80%93840%2C+2004%2E)
33. Avner May, Alireza Bagheri Garakani, Zhiyun Lu, Dong Guo, Kuan Liu, Aurélien Bellet, Linxi Fan, Michael Collins, Daniel Hsu, Brian Kingsbury, et al. Kernel Approximation Methods for Speech Recognition. arXiv preprint arXiv:1701.03577, 2017. — [arXiv:1701.03577](https://arxiv.org/abs/1701.03577) [find](https://scholar.google.com/scholar?q=Avner+May%2C+Alireza+Bagheri+Garakani%2C+Zhiyun+Lu%2C+Dong+Guo%2C+Kuan+Liu%2C+Aur%C3%A9lien+Bellet%2C+Linxi+Fan%2C+Michael+Collins%2C+Daniel+Hsu%2C+Brian+Kingsbury%2C+et+al%2E+Kernel+Approximation+Methods+for+Speech+Recognition)
34. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. Recurrent Neural Network Based Language Model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. [find](https://scholar.google.com/scholar?q=Tom%C3%A1%C5%A1+Mikolov%2C+Martin+Karafi%C3%A1t%2C+Luk%C3%A1%C5%A1+Burget%2C+Jan+C%CC%8Cernocky%CC%80%2C+and+Sanjeev+Khudanpur%2E+Recurrent+Neural+Network+Based+Language+Model%2E+In+Eleventh+Annual+Conference+of+the+International+Speech+Communicat)
35. Open Platform for AI, 2018. [https://github.com/microsoft/pai](https://github.com/microsoft/pai). [↗](https://github.com/microsoft/pai)
36. Kay Ousterhout, Patrick Wendell, Matei Zaharia, and Ion Stoica. Sparrow: Distributed, Low Latency Scheduling. In SOSP, 2013. [find](https://scholar.google.com/scholar?q=Kay+Ousterhout%2C+Patrick+Wendell%2C+Matei+Zaharia%2C+and+Ion+Stoica%2E+Sparrow%3A+Distributed%2C+Low+Latency+Scheduling%2E+In+SOSP%2C+2013%2E)
37. Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, and Chuanxiong Guo. Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters. In EuroSys, 2018. [find](https://scholar.google.com/scholar?q=Yanghua+Peng%2C+Yixin+Bao%2C+Yangrui+Chen%2C+Chuan+Wu%2C+and+Chuanxiong+Guo%2E+Optimus%3A+An+Efficient+Dynamic+Resource+Scheduler+for+Deep+Learning+Clusters%2E+In+EuroSys%2C+2018%2E)
38. DNN training workloads on Microsoft’s internal Philly clusters, 2019. [https://github.com/msr-fiddle/philly-traces](https://github.com/msr-fiddle/philly-traces). [↗](https://github.com/msr-fiddle/philly-traces)
39. PyTorch, 2018. [https://pytorch.org/](https://pytorch.org/). [↗](https://pytorch.org/)
40. Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. [vDNN: Virtualized Deep Neural Networks for Scalable, Memory-efficient Neural Network Design](https://md.research.elamur.ai/free/en/papers/rhu-2016-vdnn-virtualized-deep-neural-networks/). In MICRO, 2016.
41. Malte Schwarzkopf, Andy Konwinski, Michael Abd-El-Malek, and John Wilkes. Omega: Flexible, Scalable Schedulers for Large Compute Clusters. In EuroSys, 2013. [find](https://scholar.google.com/scholar?q=Malte+Schwarzkopf%2C+Andy+Konwinski%2C+Michael+Abd%2DEl%2DMalek%2C+and+John+Wilkes%2E+Omega%3A+Flexible%2C+Scalable+Schedulers+for+Large+Compute+Clusters%2E+In+EuroSys%2C+2013%2E)
42. Frank Seide and Amit Agarwal. CNTK: Microsoft’s Open-Source Deep-Learning Toolkit. In KDD, 2016. [find](https://scholar.google.com/scholar?q=Frank+Seide+and+Amit+Agarwal%2E+CNTK%3A+Microsoft%E2%80%99s+Open%2DSource+Deep%2DLearning+Toolkit%2E+In+KDD%2C+2016%2E)
43. Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. The Hadoop Distributed File System. In MSST, 2010. [find](https://scholar.google.com/scholar?q=Konstantin+Shvachko%2C+Hairong+Kuang%2C+Sanjay+Radia%2C+and+Robert+Chansler%2E+The+Hadoop+Distributed+File+System%2E+In+MSST%2C+2010%2E)
44. Apache Hadoop 3.2.0 Submarine, 2019. [https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/](https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/) hadoop-yarn-submarine/. [↗](https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/)
45. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. LSTM Neural Networks for Language Modeling. In Thirteenth Annual Conference of the International Speech Communication Association, 2012. [find](https://scholar.google.com/scholar?q=Martin+Sundermeyer%2C+Ralf+Schl%C3%BCter%2C+and+Hermann+Ney%2E+LSTM+Neural+Networks+for+Language+Modeling%2E+In+Thirteenth+Annual+Conference+of+the+International+Speech+Communication+Association%2C+2012%2E)
46. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going Deeper With Convolutions. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wei+Liu%2C+Yangqing+Jia%2C+Pierre+Sermanet%2C+Scott+Reed%2C+Dragomir+Anguelov%2C+Dumitru+Erhan%2C+Vincent+Vanhoucke%2C+and+Andrew+Rabinovich%2E+Going+Deeper+With+Convolutions%2E+In+CVPR%2C+2015%2E)
47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention Is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS, 2017.
48. Vinod Kumar Vavilapalli, Arun C Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, Jason Lowe, Hitesh Shah, Siddharth Seth, et al. Apache Hadoop YARN: Yet Another Resource Negotiator. In SoCC, 2013. [find](https://scholar.google.com/scholar?q=Vinod+Kumar+Vavilapalli%2C+Arun+C+Murthy%2C+Chris+Douglas%2C+Sharad+Agarwal%2C+Mahadev+Konar%2C+Robert+Evans%2C+Thomas+Graves%2C+Jason+Lowe%2C+Hitesh+Shah%2C+Siddharth+Seth%2C+et+al%2E+Apache+Hadoop+YARN%3A+Yet+Another+Resou)
49. Shivaram Venkataraman, Aurojit Panda, Ganesh Ananthanarayanan, Michael J. Franklin, and Ion Stoica. The Power of Choice in Data-aware Cluster Scheduling. In OSDI, 2014. [find](https://scholar.google.com/scholar?q=Shivaram+Venkataraman%2C+Aurojit+Panda%2C+Ganesh+Ananthanarayanan%2C+Michael+J%2E+Franklin%2C+and+Ion+Stoica%2E+The+Power+of+Choice+in+Data%2Daware+Cluster+Scheduling%2E+In+OSDI%2C+2014%2E)
50. Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. Large-scale Cluster Management at Google with Borg. In EuroSys, 2015. [find](https://scholar.google.com/scholar?q=Abhishek+Verma%2C+Luis+Pedrosa%2C+Madhukar+Korupolu%2C+David+Oppenheimer%2C+Eric+Tune%2C+and+John+Wilkes%2E+Large%2Dscale+Cluster+Management+at+Google+with+Borg%2E+In+EuroSys%2C+2015%2E)
51. Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: Dynamic GPU Memory Management for Training Deep Neural Networks. In PPoPP, 2018. [find](https://scholar.google.com/scholar?q=Linnan+Wang%2C+Jinmian+Ye%2C+Yiyang+Zhao%2C+Wei+Wu%2C+Ang+Li%2C+Shuaiwen+Leon+Song%2C+Zenglin+Xu%2C+and+Tim+Kraska%2E+Superneurons%3A+Dynamic+GPU+Memory+Management+for+Training+Deep+Neural+Networks%2E+In+PPoPP%2C+2018%2E)
52. Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, Lidong Zhou. Gandiva: Introspective Cluster Scheduling for Deep Learning. In OSDI, 2018. [find](https://scholar.google.com/scholar?q=Wencong+Xiao%2C+Romil+Bhardwaj%2C+Ramachandran+Ramjee%2C+Muthian+Sivathanu%2C+Nipun+Kwatra%2C+Zhenhua+Han%2C+Pratyush+Patel%2C+Xuan+Peng%2C+Hanyu+Zhao%2C+Quanlu+Zhang%2C+Fan+Yang%2C+Lidong+Zhou%2E+Gandiva%3A+Introspective+Clus)
53. Hangchen Yu and Christopher J. Rossbach. Full Virtualization for GPUs Reconsidered. In Workshop on Duplicating, Deconstructing, and Debunking (WDDD), 2017. [find](https://scholar.google.com/scholar?q=Hangchen+Yu+and+Christopher+J%2E+Rossbach%2E+Full+Virtualization+for+GPUs+Reconsidered%2E+In+Workshop+on+Duplicating%2C+Deconstructing%2C+and+Debunking+%28WDDD%29%2C+2017%2E)
54. Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, and Ion Stoica. Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling. In EuroSys, 2010. [find](https://scholar.google.com/scholar?q=Matei+Zaharia%2C+Dhruba+Borthakur%2C+Joydeep+Sen+Sarma%2C+Khaled+Elmeleegy%2C+Scott+Shenker%2C+and+Ion+Stoica%2E+Delay+Scheduling%3A+A+Simple+Technique+for+Achieving+Locality+and+Fairness+in+Cluster+Scheduling%2E+In+)
55. Kai Zeng, Sameer Agarwal, and Ion Stoica. iOLAP: Managing Uncertainty for Efficient Incremental OLAP. In SIGMOD, 2016. [find](https://scholar.google.com/scholar?q=Kai+Zeng%2C+Sameer+Agarwal%2C+and+Ion+Stoica%2E+iOLAP%3A+Managing+Uncertainty+for+Efficient+Incremental+OLAP%2E+In+SIGMOD%2C+2016%2E)
56. Haoyu Zhang, Logan Stafman, Andrew Or, and Michael J. Freedman. SLAQ: Quality-driven Scheduling for Distributed Machine Learning. In SoCC, 2017. [find](https://scholar.google.com/scholar?q=Haoyu+Zhang%2C+Logan+Stafman%2C+Andrew+Or%2C+and+Michael+J%2E+Freedman%2E+SLAQ%3A+Quality%2Ddriven+Scheduling+for+Distributed+Machine+Learning%2E+In+SoCC%2C+2017%2E)

*Updated 2 November 2026*