---
title: "Designing Network Design Spaces — Radosavovic 2020"
description: "The authors instead design whole design spaces, meaning parametrized families of architectures. 35 references, cited by 111 papers here."
authors: ["Ilija Radosavovic", "Raj Prateek Kosaraju", "Ross Girshick", "Kaiming He", "Piotr Dollar"]
year: 2020
venue: "arXiv:2003.13678"
source: "https://arxiv.org/abs/2003.13678"
doi: "10.48550/arxiv.2003.13678"
arxiv: "2003.13678"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/radosavovic-2020-designing-network-design-spaces/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/radosavovic-2020-designing-network-design-spaces/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / Designing Network Design Spaces

# Designing Network Design Spaces

**Authors:** Ilija Radosavovic, Raj Prateek Kosaraju, [Ross Girshick](https://md.research.elamur.ai/free/en/authors/ross-girshick/), [Kaiming He](https://md.research.elamur.ai/free/en/authors/kaiming-he/) and [Piotr Dollar](https://md.research.elamur.ai/free/en/authors/piotr-dollar/) · **Year:** 2020 · **Venue:** [arXiv:2003.13678](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/) ([IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2020](https://md.research.elamur.ai/free/en/venues/ieee-computer-society-conference-on-computer-vision-and-pattern-recognition/2020/)) · **Publisher:** [Institute of Electrical and Electronics Engineers](https://md.research.elamur.ai/free/en/publishers/institute-of-electrical-and-electronics-engineers/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2003.13678) · [PDF](https://arxiv.org/pdf/2003.13678) · [DOI 10.48550/arxiv.2003.13678](https://doi.org/10.48550/arxiv.2003.13678) · [arXiv 2003.13678](https://arxiv.org/abs/2003.13678) · [OpenAlex](https://openalex.org/W3016265891)

**Citations:** 1,929 citations (OpenAlex, 2026-10-14) · [111](https://md.research.elamur.ai/free/en/papers/radosavovic-2020-designing-network-design-spaces/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Neural architecture search usually returns one network tuned to one setting, such as a specific hardware target. That output says little about why the network works or how to adapt it elsewhere. The authors instead design whole design spaces, meaning parametrized families of architectures. They judge each space by sampling and training many models and examining the resulting error distribution, building on the methodology of Radosavovic et al. [21].

They start from a loosely constrained space called AnyNet, where stage widths and depths vary freely. A sequence of human-guided refinement steps then produces simpler spaces that keep or improve population quality. The result is RegNet, a low-dimensional space where stage widths and depths follow a quantized linear rule. The space is designed cheaply on ImageNet (400MF models, 10 epochs) and then shown to hold for larger compute, longer schedules and other block types.

RegNet models clearly beat ResNe(X)t and do well in the mobile regime. They also outperform EfficientNet at comparable flops and training setups while running up to 5× faster on GPUs. The work matters because it yields transferable design rules rather than one isolated model.

## Key points
- The design process goes from AnyNet to RegNet. Space quality is measured with the error empirical distribution function (EDF) over sampled models, for example n = 500 models in the 400MF, 10-epoch regime.
- RegNet's main finding is that good networks have stage widths and depths explained by a quantized linear function of the stage/block index.
- Some findings go against common practice. The best models have roughly 20 blocks regardless of compute regime, and they use neither a bottleneck nor an inverted bottleneck.
- Training settings were checked on populations of models. lr = 0.1 and wd = 5·10⁻⁵ proved stable across flop regimes and schedule lengths. Among the block types tested (X, R, V, VR), the X block performed best, and residual connections did not help the V block.
- Training tricks strongly affect EfficientNet-B0 results: without them, error is about 2% worse. Generalization was also tested on ImageNetV2, and code and models are released in pycls.

## Interesting ideas
- **Evaluate distributions, not single winners.** Sample and train many cheap models from a configuration space and compare error distributions, which removes much of the luck of a single run. The same idea could be used to compare tokenizers, data mixes or model variants in NLP.
- **Refine the space step by step.** Each step adds a constraint, such as shared parameters or monotone widths, and keeps it only if the population does not get worse. The result is an interpretable rule rather than a black-box search outcome.
- **Explore in a low-cost regime, then confirm at scale.** Run the exploration at small compute and few epochs, then check that the conclusions hold for bigger models and longer training. This makes broad sweeps affordable on limited budgets.
- **Bootstrap confidence ranges for good settings.** An empirical bootstrap over sampled models gives a likely range for the best value of each parameter, such as depth, width, learning rate or weight decay. This approach can be applied to hyperparameter studies in any domain.
- **Control the training setup in comparisons.** Retrain baselines under the same recipe before comparing. The EfficientNet-B0 gap shows that training enhancements alone can account for differences that might otherwise be credited to the architecture.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 214 works · 241,492 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He and Piotr Dollar. 2020. Designing Network Design Spaces. arXiv:2003.13678.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{radosavovic2020designing,
  title = {{Designing Network Design Spaces}},
  author = {Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He and Piotr Dollar},
  year = {2020},
  doi = {10.48550/arxiv.2003.13678},
  eprint = {2003.13678},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.13678}
}
```

- [← previous A Primer in BERTology: What We Know About How BERT Works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/)
- [next → BLEURT: Learning Robust Metrics for Text Generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. F. Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. 7 [find](https://scholar.google.com/scholar?q=F%2E+Chollet%2E+Xception%3A+Deep+learning+with+depthwise+separable+convolutions%2E+In+CVPR%2C+2017%2E+7)
2. E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. AutoAugment: Learning augmentation policies from data. arXiv:1805.09501, 2018. 8, 11 [find](https://scholar.google.com/scholar?q=E%2E+D%2E+Cubuk%2C+B%2E+Zoph%2C+D%2E+Mane%2C+V%2E+Vasudevan%2C+and+Q%2E+V%2E+Le%2E+AutoAugment%3A+Learning+augmentation+policies+from+data%2E+arXiv%3A1805%2E09501%2C+2018%2E+8%2C+11)
3. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 2, 3, 8, 10 [find](https://scholar.google.com/scholar?q=J%2E+Deng%2C+W%2E+Dong%2C+R%2E+Socher%2C+L%2E%2DJ%2E+Li%2C+K%2E+Li%2C+and+L%2E+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+CVPR%2C+2009%2E+2%2C+3%2C+8%2C+10)
4. T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv:1708.04552, 2017. 8 — [arXiv:1708.04552](https://arxiv.org/abs/1708.04552) [find](https://scholar.google.com/scholar?q=T%2E+DeVries+and+G%2E+W%2E+Taylor%2E+Improved+regularization+of+convolutional+neural+networks+with+cutout%2E+arXiv%3A1708%2E04552%2C+2017%2E+8+%E2%80%94+%5BarXiv%3A1708%2E04552%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1708%2E04552%29)
5. B. Efron and R. J. Tibshirani. An introduction to the bootstrap. CRC press, 1994. 3 [find](https://scholar.google.com/scholar?q=B%2E+Efron+and+R%2E+J%2E+Tibshirani%2E+An+introduction+to+the+bootstrap%2E+CRC+press%2C+1994%2E+3)
6. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv:1706.02677, 2017. 11 [find](https://scholar.google.com/scholar?q=P%2E+Goyal%2C+P%2E+Dolla%CC%81r%2C+R%2E+Girshick%2C+P%2E+Noordhuis%2C+L%2E+Wesolowski%2C+A%2E+Kyrola%2C+A%2E+Tulloch%2C+Y%2E+Jia%2C+and+K%2E+He%2E+Accurate%2C+large+minibatch+sgd%3A+Training+imagenet+in+1+hour%2E+arXiv%3A1706%2E02677%2C+2017%2E+11)
7. K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015. 2 [find](https://scholar.google.com/scholar?q=K%2E+He%2C+X%2E+Zhang%2C+S%2E+Ren%2C+and+J%2E+Sun%2E+Delving+deep+into+rectifiers%3A+Surpassing+human%2Dlevel+performance+on+imagenet+classification%2E+In+ICCV%2C+2015%2E+2)
8. K. He, X. Zhang, S. Ren, and J. Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In CVPR, 2016. 1, 2, 3, 9
9. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. [Mobilenets: Efficient convolutional neural networks for mobile vision applications](https://md.research.elamur.ai/free/en/papers/howard-2017-mobilenets-efficient-convolutional-neural-networks/). arXiv:1704.04861, 2017. 2, 8
10. J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In CVPR, 2018. 7 [find](https://scholar.google.com/scholar?q=J%2E+Hu%2C+L%2E+Shen%2C+and+G%2E+Sun%2E+Squeeze%2Dand%2Dexcitation+networks%2E+In+CVPR%2C+2018%2E+7)
11. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017. 2 [find](https://scholar.google.com/scholar?q=G%2E+Huang%2C+Z%2E+Liu%2C+L%2E+Van+Der+Maaten%2C+and+K%2E+Q%2E+Weinberger%2E+Densely+connected+convolutional+networks%2E+In+CVPR%2C+2017%2E+2)
12. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. 4 [find](https://scholar.google.com/scholar?q=S%2E+Ioffe+and+C%2E+Szegedy%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+In+ICML%2C+2015%2E+4)
13. A. Krizhevsky, I. Sutskever, and G. E. Hinton. [Imagenet classification with deep convolutional neural networks](https://md.research.elamur.ai/free/en/papers/krizhevsky-2012-imagenet-classification-deep-convolutional/). In NIPS, 2012. 1, 2
14. G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017. 8, 11 [find](https://scholar.google.com/scholar?q=G%2E+Larsson%2C+M%2E+Maire%2C+and+G%2E+Shakhnarovich%2E+Fractalnet%3A+Ultra%2Ddeep+neural+networks+without+residuals%2E+In+ICLR%2C+2017%2E+8%2C+11)
15. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1989. 1 [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+B%2E+Boser%2C+J%2E+S%2E+Denker%2C+D%2E+Henderson%2C+R%2E+E%2E+Howard%2C+W%2E+Hubbard%2C+and+L%2E+D%2E+Jackel%2E+Backpropagation+applied+to+handwritten+zip+code+recognition%2E+Neural+computation%2C+1989%2E+1)
16. C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. In AISTATS, 2015. 8 [find](https://scholar.google.com/scholar?q=C%2E%2DY%2E+Lee%2C+S%2E+Xie%2C+P%2E+Gallagher%2C+Z%2E+Zhang%2C+and+Z%2E+Tu%2E+Deeply%2Dsupervised+nets%2E+In+AISTATS%2C+2015%2E+8)
17. C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy. Progressive neural architecture search. In ECCV, 2018. 2, 8 [find](https://scholar.google.com/scholar?q=C%2E+Liu%2C+B%2E+Zoph%2C+M%2E+Neumann%2C+J%2E+Shlens%2C+W%2E+Hua%2C+L%2E%2DJ%2E+Li%2C+L%2E+Fei%2DFei%2C+A%2E+Yuille%2C+J%2E+Huang%2C+and+K%2E+Murphy%2E+Progressive+neural+architecture+search%2E+In+ECCV%2C+2018%2E+2%2C+8)
18. H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. In ICLR, 2019. 1, 2, 8 [find](https://scholar.google.com/scholar?q=H%2E+Liu%2C+K%2E+Simonyan%2C+and+Y%2E+Yang%2E+Darts%3A+Differentiable+architecture+search%2E+In+ICLR%2C+2019%2E+1%2C+2%2C+8)
19. N. Ma, X. Zhang, H.-T. Zheng, and J. Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018. 8 [find](https://scholar.google.com/scholar?q=N%2E+Ma%2C+X%2E+Zhang%2C+H%2E%2DT%2E+Zheng%2C+and+J%2E+Sun%2E+Shufflenet+v2%3A+Practical+guidelines+for+efficient+cnn+architecture+design%2E+In+ECCV%2C+2018%2E+8)
20. H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean. [Efficient neural architecture search via parameter sharing](https://md.research.elamur.ai/free/en/papers/pham-2018-efficient-neural-architecture-search-parameter/). In ICML, 2018. 2
21. I. Radosavovic, J. Johnson, S. Xie, W.-Y. Lo, and P. Dollár. On network design spaces for visual recognition. In ICCV, 2019. 1, 2, 3, 4, 11 [find](https://scholar.google.com/scholar?q=I%2E+Radosavovic%2C+J%2E+Johnson%2C+S%2E+Xie%2C+W%2E%2DY%2E+Lo%2C+and+P%2E+Dolla%CC%81r%2E+On+network+design+spaces+for+visual+recognition%2E+In+ICCV%2C+2019%2E+1%2C+2%2C+3%2C+4%2C+11)
22. P. Ramachandran, B. Zoph, and Q. V. Le. Searching for activation functions. arXiv:1710.05941, 2017. 10 [find](https://scholar.google.com/scholar?q=P%2E+Ramachandran%2C+B%2E+Zoph%2C+and+Q%2E+V%2E+Le%2E+Searching+for+activation+functions%2E+arXiv%3A1710%2E05941%2C+2017%2E+10)
23. E. Real, A. Aggarwal, Y. Huang, and Q. V. Le. Regularized evolution for image classifier architecture search. In AAAI, 2019. 2, 8 [find](https://scholar.google.com/scholar?q=E%2E+Real%2C+A%2E+Aggarwal%2C+Y%2E+Huang%2C+and+Q%2E+V%2E+Le%2E+Regularized+evolution+for+image+classifier+architecture+search%2E+In+AAAI%2C+2019%2E+2%2C+8)
24. B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? arXiv:1902.10811, 2019. 2, 10 [find](https://scholar.google.com/scholar?q=B%2E+Recht%2C+R%2E+Roelofs%2C+L%2E+Schmidt%2C+and+V%2E+Shankar%2E+Do+imagenet+classifiers+generalize+to+imagenet%3F+arXiv%3A1902%2E10811%2C+2019%2E+2%2C+10)
25. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 2, 7, 8 [find](https://scholar.google.com/scholar?q=M%2E+Sandler%2C+A%2E+Howard%2C+M%2E+Zhu%2C+A%2E+Zhmoginov%2C+and+L%2E%2DC%2E+Chen%2E+Mobilenetv2%3A+Inverted+residuals+and+linear+bottlenecks%2E+In+CVPR%2C+2018%2E+2%2C+7%2C+8)
26. K. Simonyan and A. Zisserman. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). In ICLR, 2015. 1, 2
27. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015. 2 [find](https://scholar.google.com/scholar?q=C%2E+Szegedy%2C+W%2E+Liu%2C+Y%2E+Jia%2C+P%2E+Sermanet%2C+S%2E+Reed%2C+D%2E+Anguelov%2C+D%2E+Erhan%2C+V%2E+Vanhoucke%2C+and+A%2E+Rabinovich%2E+Going+deeper+with+convolutions%2E+In+CVPR%2C+2015%2E+2)
28. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016. 2 [find](https://scholar.google.com/scholar?q=C%2E+Szegedy%2C+V%2E+Vanhoucke%2C+S%2E+Ioffe%2C+J%2E+Shlens%2C+and+Z%2E+Wojna%2E+Rethinking+the+inception+architecture+for+computer+vision%2E+In+CVPR%2C+2016%2E+2)
29. M. Tan and Q. V. Le. [Efficientnet: Rethinking model scaling for convolutional neural networks](https://md.research.elamur.ai/free/en/papers/tan-2019-efficientnet-rethinking-model-scaling-convolutional/). ICML, 2019. 1, 2, 7, 9, 10, 11
30. T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. Coursera: Neural networks for machine learning, 2012. 11 [find](https://scholar.google.com/scholar?q=T%2E+Tieleman+and+G%2E+Hinton%2E+Lecture+6%2E5%2Drmsprop%3A+Divide+the+gradient+by+a+running+average+of+its+recent+magnitude%2E+Coursera%3A+Neural+networks+for+machine+learning%2C+2012%2E+11)
31. S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In CVPR, 2017. 2, 4, 9 [find](https://scholar.google.com/scholar?q=S%2E+Xie%2C+R%2E+Girshick%2C+P%2E+Dolla%CC%81r%2C+Z%2E+Tu%2C+and+K%2E+He%2E+Aggregated+residual+transformations+for+deep+neural+networks%2E+In+CVPR%2C+2017%2E+2%2C+4%2C+9)
32. S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC, 2016. 2 [find](https://scholar.google.com/scholar?q=S%2E+Zagoruyko+and+N%2E+Komodakis%2E+Wide+residual+networks%2E+In+BMVC%2C+2016%2E+2)
33. X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018. 8 [find](https://scholar.google.com/scholar?q=X%2E+Zhang%2C+X%2E+Zhou%2C+M%2E+Lin%2C+and+J%2E+Sun%2E+Shufflenet%3A+An+extremely+efficient+convolutional+neural+network+for+mobile+devices%2E+In+CVPR%2C+2018%2E+8)
34. B. Zoph and Q. V. Le. [Neural architecture search with reinforcement learning](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/). In ICLR, 2017. 1
35. B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. [Learning transferable architectures for scalable image recognition](https://md.research.elamur.ai/free/en/papers/zoph-2017-learning-transferable-architectures-scalable-image/). In CVPR, 2018. 2, 8

*Updated 2 November 2026*