---
title: "Measuring the Intrinsic Dimension of Objective Landscapes (2018)"
description: "They then train only a small d-dimensional vector that is mapped into the full… 34.34% accuracy. 36 references, cited by 84 papers here."
authors: ["Chunyuan Li", "Heerad Farkhoor", "Rosanne Liu", "Jason Yosinski"]
year: 2018
venue: "arXiv:1804.08838"
source: "https://arxiv.org/abs/1804.08838"
doi: "10.48550/arxiv.1804.08838"
arxiv: "1804.08838"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2018-measuring-intrinsic-dimension-objective-landscapes/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2018-measuring-intrinsic-dimension-objective-landscapes/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Measuring the Intrinsic Dimension of Objective Landscapes

# Measuring the Intrinsic Dimension of Objective Landscapes

**Authors:** [Chunyuan Li](https://md.research.elamur.ai/free/en/authors/chunyuan-li/), Heerad Farkhoor, [Rosanne Liu](https://md.research.elamur.ai/free/en/authors/rosanne-liu/) and [Jason Yosinski](https://md.research.elamur.ai/free/en/authors/jason-yosinski/) · **Year:** 2018 · **Venue:** arXiv:1804.08838 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1804.08838) · [PDF](https://arxiv.org/pdf/1804.08838) · [DOI 10.48550/arxiv.1804.08838](https://doi.org/10.48550/arxiv.1804.08838) · [arXiv 1804.08838](https://arxiv.org/abs/1804.08838) · [OpenAlex](https://openalex.org/W2785533664)

**Citations:** 59 citations (OpenAlex, 2026-10-14) · [84](https://md.research.elamur.ai/free/en/papers/li-2018-measuring-intrinsic-dimension-objective-landscapes/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks how many trainable degrees of freedom a neural network problem actually needs, as opposed to how many parameters a model has. Instead of optimizing all D weights directly, the authors freeze a random initialization and a random D×d projection matrix. They then train only a small d-dimensional vector that is mapped into the full weight space. They increase d step by step and record the smallest value at which the model reaches 90% of a directly trained baseline. Performance is measured as validation accuracy, or as shifted total reward for RL. This threshold value is called the intrinsic dimension, d_int90. Experiments cover fully connected and convolutional networks on MNIST, CIFAR-10 and ImageNet, plus RL tasks such as CartPole, Inverted Pendulum, Humanoid and Atari Pong. Many tasks need far fewer dimensions than expected. The measured value also stays fairly stable across models of very different sizes, which suggests that extra parameters mostly enlarge the set of solutions. The measure allows difficulty comparisons across learning paradigms. It also gives a constructive upper bound on a solution's description length and can be used to compress networks.

## Key points
- **Sanity check:** On a toy problem with D = 1000 and 10 sum constraints, the method recovers the analytically expected intrinsic dimension of 10.
- **Supervised vision tasks:**
  - MNIST: FC (D = 199,210) needs d_int90 = 750, and LeNet (D = 44,426) needs 275.
  - CIFAR-10: FC needs 9,000, and LeNet needs 2,900.
  - ImageNet: SqueezeNet needs more than 500,000; at d = 500k it reached only 34.34% accuracy, still below the threshold.
- **RL tasks:** Inverted Pendulum needs 4, CartPole 25, Humanoid 700, and Atari Pong (ConvNet) 6,000. By this measure the pendulum is roughly 100× easier than MNIST, and Pong is about as hard as CIFAR-10.
- **Memorization:** MNIST with fully shuffled labels requires d_int90 = 190,000 with an FC-5 network. Shuffling pixels raises LeNet's value (650 vs. 275) but leaves the FC network unchanged (750).
- **Convolution ablation and compression:**
  - Both local connectivity and weight tying lower intrinsic dimension. On MNIST the values are 290 for standard LeNet, 600 for untied, 425 for FC-tied and 2,000 for FC-LeNet.
  - Subspace training compresses some networks by more than 100×.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 134 works · 65,159 citations
- [Duke University](https://md.research.elamur.ai/free/en/universities/duke-university/) — University, Durham, United States · 37 works · 15,861 citations
- [Uber AI](https://md.research.elamur.ai/free/en/companies/uber-ai/) — Company, San Francisco, United States · 2 works · 721 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

## Cite

Cite as: Chunyuan Li, Heerad Farkhoor, Rosanne Liu and Jason Yosinski. 2018. Measuring the Intrinsic Dimension of Objective Landscapes. arXiv:1804.08838.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2018measuring,
  title = {{Measuring the Intrinsic Dimension of Objective Landscapes}},
  author = {Chunyuan Li, Heerad Farkhoor, Rosanne Liu and Jason Yosinski},
  year = {2018},
  doi = {10.48550/arxiv.1804.08838},
  eprint = {1804.08838},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1804.08838}
}
```

- [← previous Mapping to Declarative Knowledge for Word Problem Solving](https://md.research.elamur.ai/free/en/papers/roy-2018-mapping-declarative-knowledge-word-problem/)
- [next → Deep contextualized word representations for detecting sarcasm and irony](https://md.research.elamur.ai/free/en/papers/ilic-2018-deep-contextualized-word-representations-detecting/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

36 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. [find](https://scholar.google.com/scholar?q=Greg+Brockman%2C+Vicki+Cheung%2C+Ludwig+Pettersson%2C+Jonas+Schneider%2C+John+Schulman%2C+Jie+Tang%2C+and+Wojciech+Zaremba%2E+Openai+gym%2C+2016%2E)
2. Francesco Camastra and Alessandro Vinciarelli. Estimating the intrinsic dimension of data with a fractal-based method. IEEE Transactions on pattern analysis and machine intelligence, 24(10): 1404–1407, 2002. [find](https://scholar.google.com/scholar?q=Francesco+Camastra+and+Alessandro+Vinciarelli%2E+Estimating+the+intrinsic+dimension+of+data+with+a+fractal%2Dbased+method%2E+IEEE+Transactions+on+pattern+analysis+and+machine+intelligence%2C+24%2810%29%3A+1404%E2%80%931407)
3. Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. CoRR, abs/1504.04788, 2015. URL http://arxiv.org/abs/1504.04788. — [arXiv:1504.04788](https://arxiv.org/abs/1504.04788) [↗](http://arxiv.org/abs/1504.04788)
4. Yann Dauphin and Yoshua Bengio. Big neural networks waste capacity. CoRR, abs/1301.3583, 2013. URL http://arxiv.org/abs/1301.3583. — [arXiv:1301.3583](https://arxiv.org/abs/1301.3583) [↗](http://arxiv.org/abs/1301.3583)
5. Yann Dauphin, Razvan Pascanu, Çaglar Gülçehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. CoRR, abs/1406.2572, 2014. URL http://arxiv.org/abs/1406.2572. [↗](http://arxiv.org/abs/1406.2572)
6. Misha Denil, Babak Shakibi, Laurent Dinh, Nando de Freitas, et al. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, pp. 2148–2156, 2013. [find](https://scholar.google.com/scholar?q=Misha+Denil%2C+Babak+Shakibi%2C+Laurent+Dinh%2C+Nando+de+Freitas%2C+et+al%2E+Predicting+parameters+in+deep+learning%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+2148%E2%80%932156%2C+2013%2E)
7. Keinosuke Fukunaga and David R Olsen. An algorithm for finding intrinsic dimensionality of data. IEEE Transactions on Computers, 100(2):176–183, 1971. [find](https://scholar.google.com/scholar?q=Keinosuke+Fukunaga+and+David+R+Olsen%2E+An+algorithm+for+finding+intrinsic+dimensionality+of+data%2E+IEEE+Transactions+on+Computers%2C+100%282%29%3A176%E2%80%93183%2C+1971%2E)
8. Raja Giryes, Guillermo Sapiro, and Alexander M Bronstein. Deep neural networks with random gaussian weights: a universal classification strategy? IEEE Trans. Signal Processing, 2016. [find](https://scholar.google.com/scholar?q=Raja+Giryes%2C+Guillermo+Sapiro%2C+and+Alexander+M+Bronstein%2E+Deep+neural+networks+with+random+gaussian+weights%3A+a+universal+classification+strategy%3F+IEEE+Trans%2E+Signal+Processing%2C+2016%2E)
9. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Artificial Intelligence and Statistics, 2010. [find](https://scholar.google.com/scholar?q=Xavier+Glorot+and+Yoshua+Bengio%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+Artificial+Intelligence+and+Statistics%2C+2010%2E)
10. Ian J. Goodfellow, Oriol Vinyals, and Andrew M. Saxe. Qualitatively characterizing neural network optimization problems. In International Conference on Learning Representations (ICLR 2015), 2015. URL http://arxiv.org/abs/1412.6544. [↗](http://arxiv.org/abs/1412.6544)
11. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J+Dally%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+huffman+coding%2E+ICLR%2C+2016%2E)
12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+Delving+deep+into+rectifiers%3A+Surpassing+human%2Dlevel+performance+on+imagenet+classification%2E+In+CVPR%2C+2015%2E)
13. Geoffrey E Hinton and Drew Van Camp. Keeping neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5–13. ACM, 1993. [find](https://scholar.google.com/scholar?q=Geoffrey+E+Hinton+and+Drew+Van+Camp%2E+Keeping+neural+networks+simple+by+minimizing+the+description+length+of+the+weights%2E+In+Proceedings+of+the+sixth+annual+conference+on+Computational+learning+theory%2C)
14. Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016. [find](https://scholar.google.com/scholar?q=Forrest+N+Iandola%2C+Song+Han%2C+Matthew+W+Moskewicz%2C+Khalid+Ashraf%2C+William+J+Dally%2C+and+Kurt+Keutzer%2E+Squeezenet%3A+Alexnet%2Dlevel+accuracy+with+50x+fewer+parameters+and%C2%A1+0%2E5+mb+model+size%2E+arXiv+preprint+)
15. Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. CoRR, abs/1706.05137, 2017. URL http://arxiv.org/abs/1706.05137. [↗](http://arxiv.org/abs/1706.05137)
16. Balázs Kégl. Intrinsic dimension estimation using packing numbers. In Advances in neural information processing systems, pp. 697–704, 2003. [find](https://scholar.google.com/scholar?q=Bala%CC%81zs+Ke%CC%81gl%2E+Intrinsic+dimension+estimation+using+packing+numbers%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+697%E2%80%93704%2C+2003%2E)
17. Diederik Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014.
18. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proc… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A%2E+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+Demis+Hassabis%2C+Claudia+Clop)
19. Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky+and+Geoffrey+Hinton%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+Technical+report%2C+2009%2E)
20. Quoc Le, Tamás Sarlós, and Alex Smola. Fastfood-approximating kernel expansions in loglinear time. In ICML, 2013. [find](https://scholar.google.com/scholar?q=Quoc+Le%2C+Tama%CC%81s+Sarlo%CC%81s%2C+and+Alex+Smola%2E+Fastfood%2Dapproximating+kernel+expansions+in+loglinear+time%2E+In+ICML%2C+2013%2E)
21. Elizaveta Levina and Peter J Bickel. Maximum likelihood estimation of intrinsic dimension. In Advances in neural information processing systems, pp. 777–784, 2005. [find](https://scholar.google.com/scholar?q=Elizaveta+Levina+and+Peter+J+Bickel%2E+Maximum+likelihood+estimation+of+intrinsic+dimension%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+777%E2%80%93784%2C+2005%2E)
22. Ping Li, T. Hastie, and K. W. Church. Very sparse random projections. In KDD, 2006. [find](https://scholar.google.com/scholar?q=Ping+Li%2C+T%2E+Hastie%2C+and+K%2E+W%2E+Church%2E+Very+sparse+random+projections%2E+In+KDD%2C+2006%2E)
23. C. Louizos, K. Ullrich, and M. Welling. Bayesian Compression for Deep Learning. ArXiv e-prints, May 2017. [find](https://scholar.google.com/scholar?q=C%2E+Louizos%2C+K%2E+Ullrich%2C+and+M%2E+Welling%2E+Bayesian+Compression+for+Deep+Learning%2E+ArXiv+e%2Dprints%2C+May+2017%2E)
24. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with Deep Reinforcement Learning. ArXiv e-prints, December 2013. [find](https://scholar.google.com/scholar?q=V%2E+Mnih%2C+K%2E+Kavukcuoglu%2C+D%2E+Silver%2C+A%2E+Graves%2C+I%2E+Antonoglou%2C+D%2E+Wierstra%2C+and+M%2E+Riedmiller%2E+Playing+Atari+with+Deep+Reinforcement+Learning%2E+ArXiv+e%2Dprints%2C+December+2013%2E)
25. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. [Faster r-cnn: Towards real-time object detection with region proposal networks](https://md.research.elamur.ai/free/en/papers/ren-2015-faster-r-cnn-towards-real/). In Advances in neural information processing systems, pp. 91–99, 2015.
26. Jorma Rissanen. Modeling by shortest data description. Automatica, 14(5):465–471, 1978. [find](https://scholar.google.com/scholar?q=Jorma+Rissanen%2E+Modeling+by+shortest+data+description%2E+Automatica%2C+14%285%29%3A465%E2%80%93471%2C+1978%2E)
27. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. [Imagenet large scale visual recognition challenge](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/). International Journal of Computer Vision, 2015.
28. T. Salimans, J. Ho, X. Chen, and I. Sutskever. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. ArXiv e-prints, March 2017. [find](https://scholar.google.com/scholar?q=T%2E+Salimans%2C+J%2E+Ho%2C+X%2E+Chen%2C+and+I%2E+Sutskever%2E+Evolution+Strategies+as+a+Scalable+Alternative+to+Reinforcement+Learning%2E+ArXiv+e%2Dprints%2C+March+2017%2E)
29. Kenneth Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99–127, 2002. [find](https://scholar.google.com/scholar?q=Kenneth+Stanley+and+Risto+Miikkulainen%2E+Evolving+neural+networks+through+augmenting+topologies%2E+Evolutionary+computation%2C+10%282%29%3A99%E2%80%93127%2C+2002%2E)
30. Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319–2323, 2000. [find](https://scholar.google.com/scholar?q=Joshua+B+Tenenbaum%2C+Vin+De+Silva%2C+and+John+C+Langford%2E+A+global+geometric+framework+for+nonlinear+dimensionality+reduction%2E+science%2C+290%285500%29%3A2319%E2%80%932323%2C+2000%2E)
31. Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26– 31, 2012. [find](https://scholar.google.com/scholar?q=Tijmen+Tieleman+and+Geoffrey+Hinton%2E+Lecture+6%2E5%2Drmsprop%3A+Divide+the+gradient+by+a+running+average+of+its+recent+magnitude%2E+COURSERA%3A+Neural+networks+for+machine+learning%2C+4%282%29%3A26%E2%80%93+31%2C+2012%2E)
32. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026– 5033. [find](https://scholar.google.com/scholar?q=Emanuel+Todorov%2C+Tom+Erez%2C+and+Yuval+Tassa%2E+Mujoco%3A+A+physics+engine+for+model%2Dbased+control%2E+In+Intelligent+Robots+and+Systems+%28IROS%29%2C+2012+IEEE%2FRSJ+International+Conference+on%2C+pp%2E+5026%E2%80%93+5033%2E)
33. Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. [Learning structured sparsity in deep neural networks](https://md.research.elamur.ai/free/en/papers/wen-2016-learning-structured-sparsity-deep-neural/). In NIPS, 2016.
34. Zichao Yang, Marcin Moczulski, Misha Denil, Nando de Freitas, Alex Smola, Le Song, and Ziyu Wang. Deep fried convnets. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1476–1483, 2015. [find](https://scholar.google.com/scholar?q=Zichao+Yang%2C+Marcin+Moczulski%2C+Misha+Denil%2C+Nando+de+Freitas%2C+Alex+Smola%2C+Le+Song%2C+and+Ziyu+Wang%2E+Deep+fried+convnets%2E+In+Proceedings+of+the+IEEE+International+Conference+on+Computer+Vision%2C+pp%2E+1476%E2%80%93)
35. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017. [find](https://scholar.google.com/scholar?q=Chiyuan+Zhang%2C+Samy+Bengio%2C+Moritz+Hardt%2C+Benjamin+Recht%2C+and+Oriol+Vinyals%2E+Understanding+deep+learning+requires+rethinking+generalization%2E+ICLR%2C+2017%2E)
36. S UPPLEMENTARY I NFORMATION FOR : M EASURING THE I NTRINSIC D IMENSION OF O BJECTIVE L ANDSCAPES S5 [find](https://scholar.google.com/scholar?q=S+UPPLEMENTARY+I+NFORMATION+FOR+%3A+M+EASURING+THE+I+NTRINSIC+D+IMENSION+OF+O+BJECTIVE+L+ANDSCAPES+S5)

*Updated 2 November 2026*