---
title: "Efficient Neural Architecture Search via Parameter Sharing (2018)"
description: "This makes automated model design practical on modest hardware. 49 references, cited by 182 papers here."
authors: ["Hieu Pham", "Melody Y. Guan", "Barret Zoph", "Quoc V. Le", "Jeff Dean"]
year: 2018
venue: "arXiv:1802.03268"
source: "https://arxiv.org/abs/1802.03268"
doi: "10.48550/arxiv.1802.03268"
arxiv: "1802.03268"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pham-2018-efficient-neural-architecture-search-parameter/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pham-2018-efficient-neural-architecture-search-parameter/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / Efficient Neural Architecture Search via Parameter Sharing

# Efficient Neural Architecture Search via Parameter Sharing

**Authors:** [Hieu Pham](https://md.research.elamur.ai/free/en/authors/hieu-pham/), [Melody Y. Guan](https://md.research.elamur.ai/free/en/authors/melody-y-guan/), [Barret Zoph](https://md.research.elamur.ai/free/en/authors/barret-zoph/), [Quoc V. Le](https://md.research.elamur.ai/free/en/authors/quoc-viet-le/) and [Jeff Dean](https://md.research.elamur.ai/free/en/authors/jeff-dean/) · **Year:** 2018 · **Venue:** arXiv:1802.03268 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1802.03268) · [PDF](https://arxiv.org/pdf/1802.03268) · [DOI 10.48550/arxiv.1802.03268](https://doi.org/10.48550/arxiv.1802.03268) · [arXiv 1802.03268](https://arxiv.org/abs/1802.03268) · [OpenAlex](https://openalex.org/W2785366763)

**Citations:** 517 citations (OpenAlex, 2026-10-14) · [182](https://md.research.elamur.ai/free/en/papers/pham-2018-efficient-neural-architecture-search-parameter/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Neural Architecture Search (NAS) can find strong models, but it is very costly: each candidate network is trained to convergence, scored, and then its weights are thrown away. Earlier work used hundreds of GPUs for several days. ENAS cuts this cost by treating every candidate architecture as a subgraph of one large directed acyclic graph whose weights all candidates share. An LSTM controller picks subgraphs, meaning which connections to use and which operation or activation each node applies. Training alternates between two steps. First, the shared weights are updated with cross-entropy on training data. Second, the controller is updated with REINFORCE, using a reward measured on validation data. The authors test ENAS on language modelling (Penn Treebank) and image classification (CIFAR-10). On Penn Treebank the discovered recurrent cell reaches a test perplexity of 55.8, the best result among methods that skip post-training processing. On CIFAR-10 the error is 2.89%, close to NASNet's 2.65%. Each search ran on one GPU in under 16 hours, more than 1000x fewer GPU-hours than standard NAS. This makes automated model design practical on modest hardware.

## Key points
- **Cost:** a full search ran on a single Nvidia GTX 1080Ti in less than 16 hours. The NAS baseline it is compared against used 450 GPUs for 3–4 days, or about 32,400–43,200 GPU-hours.
- **Results:**
  - Penn Treebank: test perplexity 55.8, versus 62.4 for the original NAS.
  - CIFAR-10: 2.89% test error, versus 2.65% for NASNet.
- **Recurrent search space:** each of the N nodes chooses an earlier node as its input and one of four activations (tanh, ReLU, identity, sigmoid). That gives 4^N × N! cells, roughly 10^15 when N = 12. Nodes that feed no other node are averaged to form the cell output.
- **Controller and training details:**
  - The controller is an LSTM with 100 hidden units that makes its choices one after another.
  - It is trained with Adam and REINFORCE, with a moving-average baseline.
  - The reward is validation accuracy for images and c / validation perplexity for text.
  - The shared weights get a gradient from a single sampled child model per step (M = 1).
- **Ablations:**
  - Random architectures from the same spaces do much worse: perplexity 81.2 on Penn Treebank, and 5.86% (whole network) or 6.77% (cells) error on CIFAR-10. ENAS reaches 4.23% and 3.54% in the same comparison.
  - Training the shared weights without updating the controller gives 8.92% error. An ensemble of 250 such sampled configurations still only reaches 5.49%.

## Interesting ideas
- **One supergraph for all candidates.** If every option you want to compare can be written as a path through a single shared graph, you train shared weights once instead of training every option from scratch. This could cheaply compare encoder variants or feature pipelines for NLP tasks with limited compute.
- **Reward on held-out data, not training loss.** The controller is pushed toward models that generalize, not ones that fit training data well. Any automated tuning loop, such as hyperparameters or data-mixing choices, can use the same separation.
- **Cheap final selection.** Instead of fully retraining many sampled candidates, ENAS scores them on one validation minibatch with the shared weights and retrains only the best one. This is a useful budget trick whenever full evaluation is the expensive step.
- **Ablations against the search space itself.** The authors separate the value of the search method from the quality of the search space. They compare against random sampling from the same space and against an untrained controller. This is a good habit for any paper claiming a clever search or selection procedure.
- **Watch for hidden constraints in weight-generation shortcuts.** The authors argue that SMASH's hypernetwork limits child weights to a low-rank space, so it may favour architectures that only look good under that limit. Keeping shared weights unconstrained avoids this, which is worth checking in any proxy-evaluation setup.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le and Jeff Dean. 2018. Efficient Neural Architecture Search via Parameter Sharing. arXiv:1802.03268.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{pham2018efficient,
  title = {{Efficient Neural Architecture Search via Parameter Sharing}},
  author = {Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le and Jeff Dean},
  year = {2018},
  doi = {10.48550/arxiv.1802.03268},
  eprint = {1802.03268},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.03268}
}
```

- [← previous Rumor Detection on Twitter with Tree-structured Recursive Neural Networks](https://md.research.elamur.ai/free/en/papers/ma-2018-rumor-detection-twitter-tree-structured/)
- [next → Horovod: fast and easy distributed deep learning in TensorFlow](https://md.research.elamur.ai/free/en/papers/sergeev-2018-horovod-fast-easy-distributed-deep/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/)

## References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Baker, Bowen, Gupta, Otkrist, Naik, Nikhil, and Raskar, Ramesh. Designing neural network architectures using reinforcement learning. In ICLR, 2017a. [find](https://scholar.google.com/scholar?q=Baker%2C+Bowen%2C+Gupta%2C+Otkrist%2C+Naik%2C+Nikhil%2C+and+Raskar%2C+Ramesh%2E+Designing+neural+network+architectures+using+reinforcement+learning%2E+In+ICLR%2C+2017a%2E)
2. Baker, Bowen, Otkrist, Gupta, Raskar, Ramesh, and Naik, Nikhil. Accelerating neural architecture search using performance prediction. Arxiv, 1705.10823, 2017b. [find](https://scholar.google.com/scholar?q=Baker%2C+Bowen%2C+Otkrist%2C+Gupta%2C+Raskar%2C+Ramesh%2C+and+Naik%2C+Nikhil%2E+Accelerating+neural+architecture+search+using+performance+prediction%2E+Arxiv%2C+1705%2E10823%2C+2017b%2E)
3. Bello, Irwan, Pham, Hieu, Le, Quoc V., Norouzi, Mohammad, and Bengio, Samy. Neural combinatorial optimization with reinforcement learning. In ICLR Workshop, 2017a. [find](https://scholar.google.com/scholar?q=Bello%2C+Irwan%2C+Pham%2C+Hieu%2C+Le%2C+Quoc+V%2E%2C+Norouzi%2C+Mohammad%2C+and+Bengio%2C+Samy%2E+Neural+combinatorial+optimization+with+reinforcement+learning%2E+In+ICLR+Workshop%2C+2017a%2E)
4. Bello, Irwan, Zoph, Barret, Vasudevan, Vijay, and Le, Quoc V. Neural optimizer search with reinforcement learning. In ICML, 2017b. [find](https://scholar.google.com/scholar?q=Bello%2C+Irwan%2C+Zoph%2C+Barret%2C+Vasudevan%2C+Vijay%2C+and+Le%2C+Quoc+V%2E+Neural+optimizer+search+with+reinforcement+learning%2E+In+ICML%2C+2017b%2E)
5. Brock, Andrew, Lim, Theodore, Ritchie, James M., and Weston, Nick. SMASH: one-shot model architecture search through hypernetworks. ICLR, 2018. [find](https://scholar.google.com/scholar?q=Brock%2C+Andrew%2C+Lim%2C+Theodore%2C+Ritchie%2C+James+M%2E%2C+and+Weston%2C+Nick%2E+SMASH%3A+one%2Dshot+model+architecture+search+through+hypernetworks%2E+ICLR%2C+2018%2E)
6. Cai, Han, Chen, Tianyao, Zhang, Weinan, Yu, Yong., and Wang, Jun. [Efficient architecture search by network transformation](https://md.research.elamur.ai/free/en/papers/cai-2017-efficient-architecture-search-network-transformation/). In AAAI, 2018.
7. Chollet, Francois. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. [find](https://scholar.google.com/scholar?q=Chollet%2C+Francois%2E+Xception%3A+Deep+learning+with+depthwise+separable+convolutions%2E+In+CVPR%2C+2017%2E)
8. Collins, Jasmine, Sohl-Dickstein, Jascha, and Sussillo, David. Capacity and trainability in recurrent neural networks. In ICLR, 2017. [find](https://scholar.google.com/scholar?q=Collins%2C+Jasmine%2C+Sohl%2DDickstein%2C+Jascha%2C+and+Sussillo%2C+David%2E+Capacity+and+trainability+in+recurrent+neural+networks%2E+In+ICLR%2C+2017%2E)
9. Deng, Boyang, Yan, Junjie, and Lin, Dahua. Peephole: Predicting network performance before training. Arxiv, 1705.10823, 2017. [find](https://scholar.google.com/scholar?q=Deng%2C+Boyang%2C+Yan%2C+Junjie%2C+and+Lin%2C+Dahua%2E+Peephole%3A+Predicting+network+performance+before+training%2E+Arxiv%2C+1705%2E10823%2C+2017%2E)
10. DeVries, Terrance and Taylor, Graham W. Improved regularization of convolutional neural networks with cutout. Arxiv, 1708.04552, 2017. [find](https://scholar.google.com/scholar?q=DeVries%2C+Terrance+and+Taylor%2C+Graham+W%2E+Improved+regularization+of+convolutional+neural+networks+with+cutout%2E+Arxiv%2C+1708%2E04552%2C+2017%2E)
11. Gal, Yarin and Ghahramani, Zoubin. [A theoretically grounded application of dropout in recurrent neural networks](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/). In NIPS, 2016.
12. Gastaldi, Xavier. Shake-shake regularization of 3-branch residual networks. In ICLR Workshop Track, 2016. [find](https://scholar.google.com/scholar?q=Gastaldi%2C+Xavier%2E+Shake%2Dshake+regularization+of+3%2Dbranch+residual+networks%2E+In+ICLR+Workshop+Track%2C+2016%2E)
13. Grave, Edouard, Joulin, Armand, and Usunier, Nicolas. [Improving neural language models with a continuous cache](https://md.research.elamur.ai/free/en/papers/grave-2016-improving-neural-language-models-continuous/). In ICLR, 2017.
14. Ha, David, Dai, Andrew, and Le, Quoc V. Hypernetworks. In ICLR, 2017. [find](https://scholar.google.com/scholar?q=Ha%2C+David%2C+Dai%2C+Andrew%2C+and+Le%2C+Quoc+V%2E+Hypernetworks%2E+In+ICLR%2C+2017%2E)
15. He, Kaiming, Zhang, Xiangyu, Rein, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In CVPR, 2015. [find](https://scholar.google.com/scholar?q=He%2C+Kaiming%2C+Zhang%2C+Xiangyu%2C+Rein%2C+Shaoqing%2C+and+Sun%2C+Jian%2E+Delving+deep+into+rectifiers%3A+Surpassing+human%2Dlevel+performance+on+imagenet+classification%2E+In+CVPR%2C+2015%2E)
16. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In CPVR, 2016a.
17. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Identity mappings in deep residual networks. In CPVR, 2016b. [find](https://scholar.google.com/scholar?q=He%2C+Kaiming%2C+Zhang%2C+Xiangyu%2C+Ren%2C+Shaoqing%2C+and+Sun%2C+Jian%2E+Identity+mappings+in+deep+residual+networks%2E+In+CPVR%2C+2016b%2E)
18. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. In Neural Computations, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+Sepp+and+Schmidhuber%2C+Ju%CC%88rgen%2E+Long+short%2Dterm+memory%2E+In+Neural+Computations%2C+1997%2E)
19. Huang, Gao, Liu, Zhuang, van der Maaten, Laurens, and Weinberger, Kilian Q. Densely connected convolutional networks. In CVPR, 2016. [find](https://scholar.google.com/scholar?q=Huang%2C+Gao%2C+Liu%2C+Zhuang%2C+van+der+Maaten%2C+Laurens%2C+and+Weinberger%2C+Kilian+Q%2E+Densely+connected+convolutional+networks%2E+In+CVPR%2C+2016%2E)
20. Inan, Hakan, Khosravi, Khashayar, and Socher, Richard. [Tying word vectors and word classifiers: a loss framework for language modeling](https://md.research.elamur.ai/free/en/papers/inan-2016-tying-word-vectors-word-classifiers/). In ICLR, 2017.
21. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. [find](https://scholar.google.com/scholar?q=Ioffe%2C+Sergey+and+Szegedy%2C+Christian%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+In+ICML%2C+2015%2E)
22. Kingma, Diederik P. and Ba, Jimmy Lei. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In ICLR, 2015.
23. Krause, Ben, Kahembwe, Emmanuel, Murray, Iain, and Renals, Steve. Dynamic evaluation of neural sequence models. Arxiv, 1709.07432, 2017. [find](https://scholar.google.com/scholar?q=Krause%2C+Ben%2C+Kahembwe%2C+Emmanuel%2C+Murray%2C+Iain%2C+and+Renals%2C+Steve%2E+Dynamic+evaluation+of+neural+sequence+models%2E+Arxiv%2C+1709%2E07432%2C+2017%2E)
24. Krizhevsky, Alex. Learning multiple layers of features from tiny images. Technical report, 2009. [find](https://scholar.google.com/scholar?q=Krizhevsky%2C+Alex%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+Technical+report%2C+2009%2E)
25. Larsson, Gustav, Maire, Michael, and Shakhnarovich, Gregory. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017. [find](https://scholar.google.com/scholar?q=Larsson%2C+Gustav%2C+Maire%2C+Michael%2C+and+Shakhnarovich%2C+Gregory%2E+Fractalnet%3A+Ultra%2Ddeep+neural+networks+without+residuals%2E+In+ICLR%2C+2017%2E)
26. Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in network. Arxiv, 1312.4400, 2013. [find](https://scholar.google.com/scholar?q=Lin%2C+Min%2C+Chen%2C+Qiang%2C+and+Yan%2C+Shuicheng%2E+Network+in+network%2E+Arxiv%2C+1312%2E4400%2C+2013%2E)
27. Liu, Chenxi, Zoph, Barret, Shlens, Jonathon, Hua, Wei, Li, Li-Jia, Fei-Fei, Li, Yuille, Alan, Huang, Jonathan, and Murphy, Kevin. Progressive neural architecture search. Arxiv, 1712.00559, 2017. [find](https://scholar.google.com/scholar?q=Liu%2C+Chenxi%2C+Zoph%2C+Barret%2C+Shlens%2C+Jonathon%2C+Hua%2C+Wei%2C+Li%2C+Li%2DJia%2C+Fei%2DFei%2C+Li%2C+Yuille%2C+Alan%2C+Huang%2C+Jonathan%2C+and+Murphy%2C+Kevin%2E+Progressive+neural+architecture+search%2E+Arxiv%2C+1712%2E00559%2C+2017%2E)
28. Liu, Hanxiao, Simonyan, Karen, Vinyals, Oriol, Fernando, Chrisantha, and Kavukcuoglu, Koray. Hierarchical representations for efficient architecture search. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Liu%2C+Hanxiao%2C+Simonyan%2C+Karen%2C+Vinyals%2C+Oriol%2C+Fernando%2C+Chrisantha%2C+and+Kavukcuoglu%2C+Koray%2E+Hierarchical+representations+for+efficient+architecture+search%2E+In+ICLR%2C+2018%2E)
29. Loshchilov, Ilya and Hutter, Frank. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+Ilya+and+Hutter%2C+Frank%2E+Sgdr%3A+Stochastic+gradient+descent+with+warm+restarts%2E+In+ICLR%2C+2017%2E)
30. Luong, Minh-Thang, Le, Quoc V., Sutskever, Ilya, Vinyals, Oriol, and Kaiser, Lukasz. [Multi-task sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/). In ICLR, 2016.
31. Marcus, Mitchell, Kim, Grace, Marcinkiewicz, Mary Ann, MacIntyre, Robert, Bies, Ann, Ferguson, Mark, Katz, Karen, and Schasberger, Britta. [The penn treebank: Annotating predicate argument structure](https://md.research.elamur.ai/free/en/papers/marcus-1994-penn-treebank-annotating-predicate-argument/). In Proceedings of the Workshop on Human Language Technology, 1994.
32. Melis, Gábor, Dyer, Chris, and Blunsom, Phil. On the state of the art of evaluation in neural language models. Arxiv, 1707.05589, 2017. [find](https://scholar.google.com/scholar?q=Melis%2C+Ga%CC%81bor%2C+Dyer%2C+Chris%2C+and+Blunsom%2C+Phil%2E+On+the+state+of+the+art+of+evaluation+in+neural+language+models%2E+Arxiv%2C+1707%2E05589%2C+2017%2E)
33. Merity, Stephen, Keskar, Nitish Shirish, and Socher, Richard. [Regularizing and optimizing LSTM language models](https://md.research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/). Arxiv, 1708.02182, 2017.
34. Negrinho, Renato and Gordon, Geoff. Deeparchitect: Automatically designing and training deep architectures. In CPVR, 2017. [find](https://scholar.google.com/scholar?q=Negrinho%2C+Renato+and+Gordon%2C+Geoff%2E+Deeparchitect%3A+Automatically+designing+and+training+deep+architectures%2E+In+CPVR%2C+2017%2E)
35. Nesterov, Yurii E. A method for solving the convex programming problem with convergence rate o(1/k 2 ). Soviet Mathematics Doklady, 1983. [find](https://scholar.google.com/scholar?q=Nesterov%2C+Yurii+E%2E+A+method+for+solving+the+convex+programming+problem+with+convergence+rate+o%281%2Fk+2+%29%2E+Soviet+Mathematics+Doklady%2C+1983%2E)
36. Razavian, Ali Sharif, Azizpour, Hossein, Josephine, Sullivan, and Carlsson, Stefan. Cnn features off-the-shelf: an astounding baseline for recognition. In CVPR, 2014. [find](https://scholar.google.com/scholar?q=Razavian%2C+Ali+Sharif%2C+Azizpour%2C+Hossein%2C+Josephine%2C+Sullivan%2C+and+Carlsson%2C+Stefan%2E+Cnn+features+off%2Dthe%2Dshelf%3A+an+astounding+baseline+for+recognition%2E+In+CVPR%2C+2014%2E)
37. Real, Esteban, Moore, Sherry, Selle, Andrew, Saxena, Saurabh, Leon, Yutaka Suematsu, Tan, Jie, Le, Quoc, and Kurakin, Alex. Large-scale evolution of image classifiers. In ICML, 2017. [find](https://scholar.google.com/scholar?q=Real%2C+Esteban%2C+Moore%2C+Sherry%2C+Selle%2C+Andrew%2C+Saxena%2C+Saurabh%2C+Leon%2C+Yutaka+Suematsu%2C+Tan%2C+Jie%2C+Le%2C+Quoc%2C+and+Kurakin%2C+Alex%2E+Large%2Dscale+evolution+of+image+classifiers%2E+In+ICML%2C+2017%2E)
38. Real, Esteban, Aggarwal, Alok, Huang, Yanping, and Le, Quoc V. Peephole: Predicting network performance before training. Arxiv, 1802.01548, 2018. [find](https://scholar.google.com/scholar?q=Real%2C+Esteban%2C+Aggarwal%2C+Alok%2C+Huang%2C+Yanping%2C+and+Le%2C+Quoc+V%2E+Peephole%3A+Predicting+network+performance+before+training%2E+Arxiv%2C+1802%2E01548%2C+2018%2E)
39. Saxena, Shreyas and Verbeek, Jakob. Convolutional neural fabrics. In NIPS, 2016. [find](https://scholar.google.com/scholar?q=Saxena%2C+Shreyas+and+Verbeek%2C+Jakob%2E+Convolutional+neural+fabrics%2E+In+NIPS%2C+2016%2E)
40. Schulman, John, Heess, Nicolas, Weber, Theophane, and Abbeel, Pieter. Gradient estimation using stochastic computation graphs. In NIPS, 2015. [find](https://scholar.google.com/scholar?q=Schulman%2C+John%2C+Heess%2C+Nicolas%2C+Weber%2C+Theophane%2C+and+Abbeel%2C+Pieter%2E+Gradient+estimation+using+stochastic+computation+graphs%2E+In+NIPS%2C+2015%2E)
41. Veniat, Tom and Denoyer, Ludovic. Learning time-efficient deep architectures with budgeted super networks. Arxiv, 1706.00046, 2017. [find](https://scholar.google.com/scholar?q=Veniat%2C+Tom+and+Denoyer%2C+Ludovic%2E+Learning+time%2Defficient+deep+architectures+with+budgeted+super+networks%2E+Arxiv%2C+1706%2E00046%2C+2017%2E)
42. Williams, Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 1992. [find](https://scholar.google.com/scholar?q=Williams%2C+Ronald+J%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+Machine+Learning%2C+1992%2E)
43. Yang, Zhilin, Dai, Zihang, Salakhutdinov, Ruslan, and Cohen, William. Breaking the softmax bottleneck: A high-rank rnn language model. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Yang%2C+Zhilin%2C+Dai%2C+Zihang%2C+Salakhutdinov%2C+Ruslan%2C+and+Cohen%2C+William%2E+Breaking+the+softmax+bottleneck%3A+A+high%2Drank+rnn+language+model%2E+In+ICLR%2C+2018%2E)
44. Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. [Recurrent neural network regularization](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/). Arxiv, 1409.2329, 2014.
45. Zhong, Zhao, Yan, Junjie, and Liu, Cheng-Lin. Practical network blocks design with q-learning. AAAI, 2018. [find](https://scholar.google.com/scholar?q=Zhong%2C+Zhao%2C+Yan%2C+Junjie%2C+and+Liu%2C+Cheng%2DLin%2E+Practical+network+blocks+design+with+q%2Dlearning%2E+AAAI%2C+2018%2E)
46. Zilly, Julian Georg, Srivastava, Rupesh Kumar, Koutnı́k, Jan, and Schmidhuber, Jürgen. [Recurrent highway networks](https://md.research.elamur.ai/free/en/papers/zilly-2016-recurrent-highway-networks/). In ICML, 2017.
47. Zoph, Barret and Le, Quoc V. [Neural architecture search with reinforcement learning](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/). In ICLR, 2017.
48. Zoph, Barret, Yuret, Deniz, May, Jonathan, and Knight, Kevin. Transfer learning for low-resource neural machine translation. In EMNLP, 2016. [find](https://scholar.google.com/scholar?q=Zoph%2C+Barret%2C+Yuret%2C+Deniz%2C+May%2C+Jonathan%2C+and+Knight%2C+Kevin%2E+Transfer+learning+for+low%2Dresource+neural+machine+translation%2E+In+EMNLP%2C+2016%2E)
49. Zoph, Barret, Vasudevan, Vijay, Shlens, Jonathon, and Le, Quoc V. [Learning transferable architectures for scalable image recognition](https://md.research.elamur.ai/free/en/papers/zoph-2017-learning-transferable-architectures-scalable-image/). In CVPR, 2018.

*Updated 2 November 2026*