---
title: "Once-for-All — paper by Cai et al. (2019) — summary and ideas"
description: "Design cost per scenario becomes roughly constant, which saves GPU hours and CO2. 44 references, cited by 155 papers here."
authors: ["Han Cai", "Chuang Gan", "Tianzhe Wang", "Zhekai Zhang", "Song Han"]
year: 2019
venue: "arXiv:1908.09791"
source: "https://arxiv.org/abs/1908.09791"
doi: "10.48550/arxiv.1908.09791"
arxiv: "1908.09791"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cai-2019-once-all-train-one-network/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cai-2019-once-all-train-one-network/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Once-for-All: Train One Network and Specialize it for Efficient Deployment

# Once-for-All: Train One Network and Specialize it for Efficient Deployment

**Authors:** [Han Cai](https://md.research.elamur.ai/free/en/authors/han-cai/), [Chuang Gan](https://md.research.elamur.ai/free/en/authors/chuang-gan/), Tianzhe Wang, [Zhekai Zhang](https://md.research.elamur.ai/free/en/authors/zhekai-zhang/) and [Song Han](https://md.research.elamur.ai/free/en/authors/song-han/) · **Year:** 2019 · **Venue:** arXiv:1908.09791 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1908.09791) · [PDF](https://arxiv.org/pdf/1908.09791) · [DOI 10.48550/arxiv.1908.09791](https://doi.org/10.48550/arxiv.1908.09791) · [arXiv 1908.09791](https://arxiv.org/abs/1908.09791) · [OpenAlex](https://openalex.org/W2969797940)

**Citations:** 97 citations (OpenAlex, 2026-10-14) · [155](https://md.research.elamur.ai/free/en/papers/cai-2019-once-all-train-one-network/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Deploying neural networks across many devices (phones, CPUs, GPUs, FPGAs, microcontrollers) and latency budgets usually means designing or searching a new architecture and training it from scratch for each target, so the cost grows with every new scenario. The authors separate training from architecture search. They train one large "once-for-all" (OFA) network whose weights are shared by a huge family of sub-networks that differ in depth, width, kernel size and input resolution. For a new device, a sub-network is picked using cheap accuracy and latency predictors, with no retraining. To keep so many weight-sharing sub-networks accurate, they propose progressive shrinking: train the biggest model first, then fine-tune step by step so that ever smaller variants are supported, distilling from the large model. On ImageNet, the chosen sub-networks match or beat individually trained, NAS-designed models such as MobileNetV3 and EfficientNet. Design cost per scenario becomes roughly constant, which saves GPU hours and CO2.

## Key points
- The OFA space contains more than 10^19 sub-networks. Slimmable Nets, by comparison, support only 4 width settings.
- Results on ImageNet:
  - Up to 4.0% higher top-1 accuracy than MobileNetV3.
  - At equal accuracy, 1.5× lower measured latency than MobileNetV3 and 2.6× lower than EfficientNet.
  - 80.0% top-1 with 595M MACs, the first result at 80% in the mobile (\<600M MACs) setting.
- The progressive shrinking schedule:
  - Kernel size {7, 5, 3}, sampling one sub-network per step.
  - Depth {4, 3, 2} per stage, sampling two sub-networks per step.
  - Width expand ratio {6, 4, 3}, sampling four sub-networks per step.
- The accuracy predictor is a 3-layer MLP with 400 hidden units. Its input is one-hot encodings of each layer's kernel size and expand ratio (zeros for skipped layers) plus the resolution. It reaches 0.21% RMSE.
- OFA took first place in the DSP classification track of the 3rd LPCVC, and in both the classification and detection tracks of the 4th edition of that competition. Code and 50 pretrained models are available at mit-han-lab/once-for-all.

## Interesting ideas
- **Train once, specialize many times.** Separating expensive shared training from cheap per-target selection turns a per-scenario cost into a one-time cost. This fits any setting with many deployment targets, for example NLP models for servers and mobile apps.
- **Shrinking as generalized pruning.** Each new stage starts from the larger network and keeps the most important channels, ranked by weight importance, as the shared core. This gives small variants a good starting point instead of letting them compete from random weights.
- **Increase sub-network sampling as flexibility grows.** The number of sub-networks whose gradients are aggregated per step rises (1, then 2, then 4) as more elastic dimensions are added. This is a simple way to control interference among weight-sharing models.
- **Use a cheap surrogate for search.** A small MLP over a one-hot architecture encoding replaces expensive evaluation of candidates. The authors also show that lower predictor error tends to give better final picks, so predictor quality is worth checking explicitly.
- **Report measured latency and carbon cost.** Comparing methods by latency measured on real devices, and accounting for GPU hours and CO2, gives a more honest picture of efficiency than MAC counts alone.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 243 works · 76,506 citations
- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 154 works · 72,606 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang and Song Han. 2019. Once-for-All: Train One Network and Specialize it for Efficient Deployment. arXiv:1908.09791.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{cai2019once,
  title = {{Once-for-All: Train One Network and Specialize it for Efficient Deployment}},
  author = {Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang and Song Han},
  year = {2019},
  doi = {10.48550/arxiv.1908.09791},
  eprint = {1908.09791},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.09791}
}
```

- [← previous NULI at SemEval-2019 Task 6: Transfer Learning for Offensive Language Detection using Bidirectional Transformers](https://md.research.elamur.ai/free/en/papers/liu-2019-nuli-semeval-2019-task-6/)
- [next → Sentence Centrality Revisited for Unsupervised Summarization](https://md.research.elamur.ai/free/en/papers/zheng-2019-sentence-centrality-revisited-unsupervised-summarization/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

44 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, and Kris M Kitani. N2n learning: Network to network compression via policy gradient reinforcement learning. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Anubhav+Ashok%2C+Nicholas+Rhinehart%2C+Fares+Beainy%2C+and+Kris+M+Kitani%2E+N2n+learning%3A+Network+to+network+compression+via+policy+gradient+reinforcement+learning%2E+In+ICLR%2C+2018%2E)
2. Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. [Efficient architecture search by network transformation](https://md.research.elamur.ai/free/en/papers/cai-2017-efficient-architecture-search-network-transformation/). In AAAI, 2018a.
3. Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformation for efficient architecture search. In ICML, 2018b. [find](https://scholar.google.com/scholar?q=Han+Cai%2C+Jiacheng+Yang%2C+Weinan+Zhang%2C+Song+Han%2C+and+Yong+Yu%2E+Path%2Dlevel+network+transformation+for+efficient+architecture+search%2E+In+ICML%2C+2018b%2E)
4. Han Cai, Ligeng Zhu, and Song Han. [ProxylessNAS: Direct neural architecture search on target task and hardware](https://md.research.elamur.ai/free/en/papers/cai-2018-proxylessnas-direct-neural-architecture-search/). In ICLR, 2019. URL [https://arxiv.org/pdf/1812.00332.pdf](https://arxiv.org/pdf/1812.00332.pdf). [↗](https://arxiv.org/pdf/1812.00332.pdf)
5. Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. In NeurIPS, 2019. [find](https://scholar.google.com/scholar?q=Brian+Cheung%2C+Alex+Terekhov%2C+Yubei+Chen%2C+Pulkit+Agrawal%2C+and+Bruno+Olshausen%2E+Superposition+of+many+models+into+one%2E+In+NeurIPS%2C+2019%2E)
6. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. [Binaryconnect: Training deep neural networks with binary weights during propagations](https://md.research.elamur.ai/free/en/papers/courbariaux-2015-binaryconnect-training-deep-neural-networks/). In NeurIPS, 2015.
7. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+CVPR%2C+2009%2E)
8. Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420, 2019. — [arXiv:1904.00420](https://arxiv.org/abs/1904.00420) [find](https://scholar.google.com/scholar?q=Zichao+Guo%2C+Xiangyu+Zhang%2C+Haoyuan+Mu%2C+Wen+Heng%2C+Zechun+Liu%2C+Yichen+Wei%2C+and+Jian+Sun%2E+Single+path+one%2Dshot+neural+architecture+search+with+uniform+sampling%2E+arXiv+preprint+arXiv%3A1904%2E00420%2C+2019%2E+%E2%80%94+%5B)
9. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurIPS, 2015. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Jeff+Pool%2C+John+Tran%2C+and+William+Dally%2E+Learning+both+weights+and+connections+for+efficient+neural+network%2E+In+NeurIPS%2C+2015%2E)
10. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J+Dally%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+huffman+coding%2E+In+ICLR%2C+2016%2E)
11. Cong Hao, Xiaofan Zhang, Yuhong Li, Sitao Huang, Jinjun Xiong, Kyle Rupnow, Wen-mei Hwu, and Deming Chen. Fpga/dnn co-design: An efficient design methodology for 1ot intelligence on the edge. In 2019 56th ACM/IEEE Design Automation Conference (DAC), pp. 1–6. [find](https://scholar.google.com/scholar?q=Cong+Hao%2C+Xiaofan+Zhang%2C+Yuhong+Li%2C+Sitao+Huang%2C+Jinjun+Xiong%2C+Kyle+Rupnow%2C+Wen%2Dmei+Hwu%2C+and+Deming+Chen%2E+Fpga%2Fdnn+co%2Ddesign%3A+An+efficient+design+methodology+for+1ot+intelligence+on+the+edge%2E+In+2019+)
12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In CVPR, 2016.
13. Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. [Amc: Automl for model compression and acceleration on mobile devices](https://md.research.elamur.ai/free/en/papers/he-2018-amc-automl-model-compression-acceleration/). In ECCV, 2018.
14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531, 2015.
15. Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV 2019, 2019. [find](https://scholar.google.com/scholar?q=Andrew+Howard%2C+Mark+Sandler%2C+Grace+Chu%2C+Liang%2DChieh+Chen%2C+Bo+Chen%2C+Mingxing+Tan%2C+Weijun+Wang%2C+Yukun+Zhu%2C+Ruoming+Pang%2C+Vijay+Vasudevan%2C+et+al%2E+Searching+for+mobilenetv3%2E+In+ICCV+2019%2C+2019%2E)
16. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. [Mobilenets: Efficient convolutional neural networks for mobile vision applications](https://md.research.elamur.ai/free/en/papers/howard-2017-mobilenets-efficient-convolutional-neural-networks/). arXiv preprint arXiv:1704.04861, 2017.
17. Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017. [find](https://scholar.google.com/scholar?q=Gao+Huang%2C+Zhuang+Liu%2C+Laurens+Van+Der+Maaten%2C+and+Kilian+Q+Weinberger%2E+Densely+connected+convolutional+networks%2E+In+CVPR%2C+2017%2E)
18. Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Gao+Huang%2C+Danlu+Chen%2C+Tianhong+Li%2C+Felix+Wu%2C+Laurens+van+der+Maaten%2C+and+Kilian+Q+Weinberger%2E+Multi%2Dscale+dense+networks+for+resource+efficient+image+classification%2E+In+ICLR%2C+2018%2E)
19. Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016. [find](https://scholar.google.com/scholar?q=Forrest+N+Iandola%2C+Song+Han%2C+Matthew+W+Moskewicz%2C+Khalid+Ashraf%2C+William+J+Dally%2C+and+Kurt+Keutzer%2E+Squeezenet%3A+Alexnet%2Dlevel+accuracy+with+50x+fewer+parameters+and%C2%A1+0%2E5+mb+model+size%2E+arXiv+preprint+)
20. Weiwen Jiang, Lei Yang, Edwin Sha, Qingfeng Zhuge, Shouzhen Gu, Yiyu Shi, and Jingtong Hu. Hardware/software co-exploration of neural architectures. arXiv preprint arXiv:1907.04650, 2019a. — [arXiv:1907.04650](https://arxiv.org/abs/1907.04650) [find](https://scholar.google.com/scholar?q=Weiwen+Jiang%2C+Lei+Yang%2C+Edwin+Sha%2C+Qingfeng+Zhuge%2C+Shouzhen+Gu%2C+Yiyu+Shi%2C+and+Jingtong+Hu%2E+Hardware%2Fsoftware+co%2Dexploration+of+neural+architectures%2E+arXiv+preprint+arXiv%3A1907%2E04650%2C+2019a%2E+%E2%80%94+%5BarXiv%3A19)
21. Weiwen Jiang, Xinyi Zhang, Edwin H-M Sha, Lei Yang, Qingfeng Zhuge, Yiyu Shi, and Jingtong Hu. Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search. In Proceedings of the 56th Annual Design Automation Conference 2019, pp. 1–6, 2019b. [find](https://scholar.google.com/scholar?q=Weiwen+Jiang%2C+Xinyi+Zhang%2C+Edwin+H%2DM+Sha%2C+Lei+Yang%2C+Qingfeng+Zhuge%2C+Yiyu+Shi%2C+and+Jingtong+Hu%2E+Accuracy+vs%2E+efficiency%3A+Achieving+both+through+fpga%2Dimplementation+aware+neural+architecture+search%2E+In+)
22. Jason Kuen, Xiangfei Kong, Zhe Lin, Gang Wang, Jianxiong Yin, Simon See, and Yap-Peng Tan. Stochastic downsampling for cost-adjustable inference and improved regularization in convolutional networks. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Jason+Kuen%2C+Xiangfei+Kong%2C+Zhe+Lin%2C+Gang+Wang%2C+Jianxiong+Yin%2C+Simon+See%2C+and+Yap%2DPeng+Tan%2E+Stochastic+downsampling+for+cost%2Dadjustable+inference+and+improved+regularization+in+convolutional+networks%2E+)
23. Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NeurIPS, 2017. [find](https://scholar.google.com/scholar?q=Ji+Lin%2C+Yongming+Rao%2C+Jiwen+Lu%2C+and+Jie+Zhou%2E+Runtime+neural+pruning%2E+In+NeurIPS%2C+2017%2E)
24. Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018. [find](https://scholar.google.com/scholar?q=Chenxi+Liu%2C+Barret+Zoph%2C+Maxim+Neumann%2C+Jonathon+Shlens%2C+Wei+Hua%2C+Li%2DJia+Li%2C+Li+Fei%2DFei%2C+Alan+Yuille%2C+Jonathan+Huang%2C+and+Kevin+Murphy%2E+Progressive+neural+architecture+search%2E+In+ECCV%2C+2018%2E)
25. Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2019. [find](https://scholar.google.com/scholar?q=Hanxiao+Liu%2C+Karen+Simonyan%2C+and+Yiming+Yang%2E+Darts%3A+Differentiable+architecture+search%2E+In+ICLR%2C+2019%2E)
26. Lanlan Liu and Jia Deng. Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution. In AAAI, 2018. [find](https://scholar.google.com/scholar?q=Lanlan+Liu+and+Jia+Deng%2E+Dynamic+deep+neural+networks%3A+Optimizing+accuracy%2Defficiency+trade%2Doffs+by+selective+execution%2E+In+AAAI%2C+2018%2E)
27. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017. [find](https://scholar.google.com/scholar?q=Zhuang+Liu%2C+Jianguo+Li%2C+Zhiqiang+Shen%2C+Gao+Huang%2C+Shoumeng+Yan%2C+and+Changshui+Zhang%2E+Learning+efficient+convolutional+networks+through+network+slimming%2E+In+ICCV%2C+2017%2E)
28. Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Sgdr%3A+Stochastic+gradient+descent+with+warm+restarts%2E+arXiv+preprint+arXiv%3A1608%2E03983%2C+2016%2E)
29. Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018. [find](https://scholar.google.com/scholar?q=Ningning+Ma%2C+Xiangyu+Zhang%2C+Hai%2DTao+Zheng%2C+and+Jian+Sun%2E+Shufflenet+v2%3A+Practical+guidelines+for+efficient+cnn+architecture+design%2E+In+ECCV%2C+2018%2E)
30. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In AAAI, 2019. [find](https://scholar.google.com/scholar?q=Esteban+Real%2C+Alok+Aggarwal%2C+Yanping+Huang%2C+and+Quoc+V+Le%2E+Regularized+evolution+for+image+classifier+architecture+search%2E+In+AAAI%2C+2019%2E)
31. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Mark+Sandler%2C+Andrew+Howard%2C+Menglong+Zhu%2C+Andrey+Zhmoginov%2C+and+Liang%2DChieh+Chen%2E+Mobilenetv2%3A+Inverted+residuals+and+linear+bottlenecks%2E+In+CVPR%2C+2018%2E)
32. Emma Strubell, Ananya Ganesh, and Andrew McCallum. [Energy and policy considerations for deep learning in nlp](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). In ACL, 2019.
33. Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820–2828, 2019. [find](https://scholar.google.com/scholar?q=Mingxing+Tan%2C+Bo+Chen%2C+Ruoming+Pang%2C+Vijay+Vasudevan%2C+Mark+Sandler%2C+Andrew+Howard%2C+and+Quoc+V+Le%2E+Mnasnet%3A+Platform%2Daware+neural+architecture+search+for+mobile%2E+In+Proceedings+of+the+IEEE+Conference+o)
34. Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In ECCV, 2018. [find](https://scholar.google.com/scholar?q=Xin+Wang%2C+Fisher+Yu%2C+Zi%2DYi+Dou%2C+Trevor+Darrell%2C+and+Joseph+E+Gonzalez%2E+Skipnet%3A+Learning+dynamic+routing+in+convolutional+networks%2E+In+ECCV%2C+2018%2E)
35. Samuel Williams, Andrew Waterman, and David Patterson. Roofline: An insightful visual performance model for floating-point programs and multicore architectures. Technical report, Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States), 2009. [find](https://scholar.google.com/scholar?q=Samuel+Williams%2C+Andrew+Waterman%2C+and+David+Patterson%2E+Roofline%3A+An+insightful+visual+performance+model+for+floating%2Dpoint+programs+and+multicore+architectures%2E+Technical+report%2C+Lawrence+Berkeley+Nat)
36. Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. [Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search](https://md.research.elamur.ai/free/en/papers/wu-2018-fbnet-hardware-aware-efficient-convnet/). In CVPR, 2019.
37. Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Zuxuan+Wu%2C+Tushar+Nagarajan%2C+Abhishek+Kumar%2C+Steven+Rennie%2C+Larry+S+Davis%2C+Kristen+Grauman%2C+and+Rogerio+Feris%2E+Blockdrop%3A+Dynamic+inference+paths+in+residual+networks%2E+In+CVPR%2C+2018%2E)
38. Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11728, 2019a. — [arXiv:1903.11728](https://arxiv.org/abs/1903.11728) [find](https://scholar.google.com/scholar?q=Jiahui+Yu+and+Thomas+Huang%2E+Autoslim%3A+Towards+one%2Dshot+architecture+search+for+channel+numbers%2E+arXiv+preprint+arXiv%3A1903%2E11728%2C+2019a%2E+%E2%80%94+%5BarXiv%3A1903%2E11728%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1903%2E11728%29)
39. Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In ICCV, 2019b. [find](https://scholar.google.com/scholar?q=Jiahui+Yu+and+Thomas+Huang%2E+Universally+slimmable+networks+and+improved+training+techniques%2E+In+ICCV%2C+2019b%2E)
40. Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In ICLR, 2019. [find](https://scholar.google.com/scholar?q=Jiahui+Yu%2C+Linjie+Yang%2C+Ning+Xu%2C+Jianchao+Yang%2C+and+Thomas+Huang%2E+Slimmable+neural+networks%2E+In+ICLR%2C+2019%2E)
41. Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018. [find](https://scholar.google.com/scholar?q=Xiangyu+Zhang%2C+Xinyu+Zhou%2C+Mengxiao+Lin%2C+and+Jian+Sun%2E+Shufflenet%3A+An+extremely+efficient+convolutional+neural+network+for+mobile+devices%2E+In+CVPR%2C+2018%2E)
42. Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In ICLR, 2017. [find](https://scholar.google.com/scholar?q=Chenzhuo+Zhu%2C+Song+Han%2C+Huizi+Mao%2C+and+William+J+Dally%2E+Trained+ternary+quantization%2E+In+ICLR%2C+2017%2E)
43. Barret Zoph and Quoc V Le. [Neural architecture search with reinforcement learning](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/). In ICLR, 2017.
44. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. [Learning transferable architectures for scalable image recognition](https://md.research.elamur.ai/free/en/papers/zoph-2017-learning-transferable-architectures-scalable-image/). In CVPR, 2018.

*Updated 2 November 2026*